feat(core): async batch-extraction submit/poll/apply + kb poll-batches (#420 part 1) - #432
Merged
Conversation
#420 part 1) Turns #383's batch infrastructure into a working submit->apply pipeline. - applyExtraction is factored out of extractChunk, so the synchronous path and the batch poller share one graph-apply. - BatchExtractor interface (satisfied by the anthropic Extractor; wired via WithBatchExtractor when EXTRACTION_BATCH + Claude). - SubmitExtractionBatch(items) calls CreateBatch, records the job (#383) + a new extraction_batch_items mapping (migration 0024: custom_id -> source_uri / discriminators / force_review). - kb poll-batches (PollExtractionBatches) checks each submitted job's FetchBatchResults; once ended it applies each succeeded result to the graph via applyExtraction keyed by custom_id (honoring scope + trust), then marks the job applied. Best-effort per item — a missing result is a failed extraction. Opt-in, nothing runs by default. Wiring the submit into the ingest loop (batch instead of sync extract) is follow-up #430; cross-doc entity resolution is #431. Tests: TestBatchExtractionSubmitAndPoll (DB + mock BatchExtractor) — submit records job+items; poll applies nothing while processing, applies nodes/edges once ended and marks the job applied. TestMigrate covers the new table. SYSTEM.md + .env.example updated. Part of #420 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nces it Migration 0024 added extraction_batch_items with a FK to extraction_batches, so TestExtractionBatchJobStore's plain TRUNCATE fails (SQLSTATE 0A000). Add CASCADE. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Part of #420 (ingestion). Turns #383's batch infrastructure into a working submit → apply pipeline. Opt-in — nothing runs by default.
How
applyExtractionis factored out ofextractChunk, so the synchronous path and the batch poller share one graph-apply.BatchExtractorinterface (satisfied by the anthropicExtractor; wired viaWithBatchExtractorwhenEXTRACTION_BATCH+ Claude).SubmitExtractionBatch(items)callsCreateBatch, records the job (Wire BatchExtractor into an async ingest job + cross-doc entity resolution (follow-up to #326) #383) + a newextraction_batch_itemsmapping (migration 0024: custom_id → source_uri / discriminators / force_review).kb poll-batches(PollExtractionBatches) checks each submitted job'sFetchBatchResults; once ended it applies each succeeded result to the graph viaapplyExtractionkeyed by custom_id (honoring scope + trust), then marks the job applied. Best-effort per item — a missing result is a failed extraction.Tests (DB-gated, CI)
TestBatchExtractionSubmitAndPoll(mockBatchExtractor): submit records job + items; poll applies nothing while processing, then applies nodes/edges once ended and marks the job applied.TestMigratecovers the new table.gofmt/vet/golangci-lintclean. SYSTEM.md (§10) +.env.exampleupdated.Not in this PR
Wiring the submit into the ingest loop (batch instead of sync extract) is #430; cross-doc entity resolution is #431.
🤖 Generated with Claude Code