feat(crawler): populate phase 6-d entity and edge tables#205
Merged
YusukeHirao merged 1 commit intoJul 13, 2026
Merged
Conversation
Adds the Phase 6-D migration helpers that populate content_items, page_meta, resource_items, anchor_edges, resource_ref_edges, and image_items from the legacy write model, plus a standalone migration script `scripts/migrate-to-phase6.mjs` that runs the whole 6-A → 6-D sequence on an existing archive. Every populate helper is chunked keyset-paginated and idempotent via `INSERT OR IGNORE` on the natural key. `anchor_edges` collapses duplicate `(pageId, hrefId)` rows via a single-pass JS scan whose "first row wins" contract is guarded by a sort-order assertion. `image_items.dom_path_text_id` is derived through an injected callback so the crawler runtime stays jsdom-free; the migration script wires the jsdom-backed implementation. Correctness safeguards applied in response to review findings: - `resolve-blob-refs` maps `hash → [raw values]` so two distinct data-URI strings that decode to the same payload both get the shared blob_refs id. - `resolve-header-sets` falls back from `raw_json_hash` to `raw_hash` so a JSON variant with different key ordering still resolves to the shared header_sets row. - Populate helpers throw when a non-null source value has no matching ref, instead of silently writing NULL — the count-based acceptance check would otherwise pass while individual rows lost data. - `matchImagesToDomPaths` overflow returns `unknown/<id>` instead of mapping every extra row to the last DOM candidate. - `populateImageItems` iterates by pageId, resolving each page's HTML and dom paths exactly once so a page whose images straddle a chunk cannot re-parse HTML or reset the ordinal cursor. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
content_items/page_meta/resource_items/anchor_edges/resource_ref_edges/image_items) plus a standalone migration scriptscripts/migrate-to-phase6.mjsthat runs 6-A → 6-D end-to-end against an existing.nitpickerarchive.INSERT OR IGNORE. Correctness safeguards (blob-refs hash-list, header-setsraw_hashfallback, throw-on-missing-ref, dom-path overflow →unknown, page-by-page image iteration) applied in response to code-review / QA / PdM findings.anchor-edge-normalization(collapse-anchor-rows.spec.ts),dom-path-derivationall-3-cases (match-images-to-dom-paths.spec.ts+derive-dom-path.spec.ts), andurl-refs-population(resolve-url-refs.spec.ts). Fullyarn test(3154) passes;yarn lint/lint:checkclean.Notes for reviewers
docs/write-model-refactor-plan.md) currently lives oncodex/111-write-model-refactor-planand is not yet onfeature/impl-new-db; referenced here but out of scope for this PR..baksemantics differ from the plan verbiage: the migration script uses out-of-place migration (untar → workDir → new tar; input untouched), which is strictly safer than the in-place.bakthe plan describes. Phase 6-B and 6-D run in separatedb.transactionblocks.verifyAcceptanceCountsis a 4-check subset of Phase 6-E: migration verification (8 invariant checks + .bak rollback) #194's 8-check + rollback spec — extract intorunPhase6EVerificationin Phase 6-E: migration verification (8 invariant checks + .bak rollback) #194.anchor-edge-normalization→collapse-anchor-rows.spec.ts;dom-path-derivation→derive-dom-path.spec.ts+match-images-to-dom-paths.spec.ts(all 3 cases in the latter);url-refs-population→resolve-url-refs.spec.ts.Closes #193
Test plan
yarn buildyarn testyarn lint:checknode scripts/migrate-to-phase6.mjs <fixture>.nitpicker→ verify acceptance count log🤖 Generated with Claude Code