feat(core): cross-document entity resolution for batch extraction (#431) - #438
Merged
Merged
Conversation
After a batch of extractions is applied, collapse proposed nodes that refer to
the same entity across documents into one proposal — the per-chunk resolver only
matches exact canonical names, so variant names ("OrderService" vs "Order
Service") in different docs stack duplicate proposals.
- core.resolveBatchDuplicates: over the identity scopes a poll touched, union-find
proposed nodes sharing a NORMALIZED name/alias (same normExpr the review-time
merge proposer uses — lowercase, strip non-alphanumeric), merge into the oldest
survivor (fold aliases, repoint proposed edges, retire the dup to historical).
Conservative: exact normalized-string identity only (no fuzzy/embedding
similarity), proposed nodes only (nothing live changes; a human still reviews),
scope-keyed so cross-project entities never merge.
- Wired into PollExtractionBatches (runs once per poll over touched scopes).
- store.ListProposedNodesInScopes + store.RepointProposedEdges (proposed edges
carry no current-only unique index, so a plain repoint; merge self-loops retired).
- DB tests: cross-doc variant names merge (order-independent); distinct entities
never merge. SYSTEM.md decision log updated.
Embedding/semantic cross-document clustering (near-synonyms/typos the normalizer
misses) needs proposal-summary embeddings + threshold tuning — filed as follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The self-loop cleanup statement referenced $2 but the call binds only newID (as $1), so $1 was declared-but-unused (undeterminable type → SQLSTATE 42P18) and $2 unbound — failing every batch merge. Use $1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #431.
Problem
Per-chunk extraction resolves entities by exact canonical name (
resolveOrProposeNode), so the same real entity written under variant names across a batch's documents — "OrderService" in one doc, "Order Service" in another — lands as separate proposals. A batch gives us the whole document set at once, so we can do better.What
After a batch's items are all applied,
PollExtractionBatchesrunsresolveBatchDuplicatesover the identity scopes it touched:normExprthe review-time merge proposer uses (lowercase, strip non-alphanumeric, so "Order Service" ≡ "order-service" ≡ "OrderService");RepointProposedEdges), retire the duplicate tohistorical.Conservative by construction
Consolidate.Tests (DB-gated, run in CI)
Follow-up
🤖 Generated with Claude Code