Skip to content

feat(core): cross-document entity resolution for batch extraction (#431) - #438

Merged
CheeryProgrammer merged 2 commits into
mainfrom
feat/431-cross-doc-entity-resolution
Jul 20, 2026
Merged

feat(core): cross-document entity resolution for batch extraction (#431)#438
CheeryProgrammer merged 2 commits into
mainfrom
feat/431-cross-doc-entity-resolution

Conversation

@CheeryProgrammer

Copy link
Copy Markdown
Contributor

Closes #431.

Problem

Per-chunk extraction resolves entities by exact canonical name (resolveOrProposeNode), so the same real entity written under variant names across a batch's documents — "OrderService" in one doc, "Order Service" in another — lands as separate proposals. A batch gives us the whole document set at once, so we can do better.

What

After a batch's items are all applied, PollExtractionBatches runs resolveBatchDuplicates over the identity scopes it touched:

  • union-find proposed nodes that share a normalized name or alias — the same normExpr the review-time merge proposer uses (lowercase, strip non-alphanumeric, so "Order Service" ≡ "order-service" ≡ "OrderService");
  • merge each cluster into the oldest survivor: fold aliases, repoint proposed edges (RepointProposedEdges), retire the duplicate to historical.

Conservative by construction

Tests (DB-gated, run in CI)

  • Two docs in one batch mention the same entity under variant names → collapse to one proposal (asserted order-independently, since a batch applies items in map order), alias folded, edge repointed.
  • Distinct entities (OrderService / PaymentService / Kafka) are never merged.

Follow-up

🤖 Generated with Claude Code

CheeryProgrammer and others added 2 commits July 20, 2026 05:10
After a batch of extractions is applied, collapse proposed nodes that refer to
the same entity across documents into one proposal — the per-chunk resolver only
matches exact canonical names, so variant names ("OrderService" vs "Order
Service") in different docs stack duplicate proposals.

- core.resolveBatchDuplicates: over the identity scopes a poll touched, union-find
  proposed nodes sharing a NORMALIZED name/alias (same normExpr the review-time
  merge proposer uses — lowercase, strip non-alphanumeric), merge into the oldest
  survivor (fold aliases, repoint proposed edges, retire the dup to historical).
  Conservative: exact normalized-string identity only (no fuzzy/embedding
  similarity), proposed nodes only (nothing live changes; a human still reviews),
  scope-keyed so cross-project entities never merge.
- Wired into PollExtractionBatches (runs once per poll over touched scopes).
- store.ListProposedNodesInScopes + store.RepointProposedEdges (proposed edges
  carry no current-only unique index, so a plain repoint; merge self-loops retired).
- DB tests: cross-doc variant names merge (order-independent); distinct entities
  never merge. SYSTEM.md decision log updated.

Embedding/semantic cross-document clustering (near-synonyms/typos the normalizer
misses) needs proposal-summary embeddings + threshold tuning — filed as follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The self-loop cleanup statement referenced $2 but the call binds only newID
(as $1), so $1 was declared-but-unused (undeterminable type → SQLSTATE 42P18)
and $2 unbound — failing every batch merge. Use $1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@CheeryProgrammer
CheeryProgrammer merged commit 0f647f7 into main Jul 20, 2026
5 checks passed
@CheeryProgrammer
CheeryProgrammer deleted the feat/431-cross-doc-entity-resolution branch July 20, 2026 05:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cross-document entity resolution for batch extraction (follow-up to #420)

1 participant