feat(crawler): phase 6-e — migration verification#206
Merged
YusukeHirao merged 1 commit intoJul 14, 2026
Merged
Conversation
Split the acceptance invariants for scripts/migrate-to-phase6.mjs into eight standalone check functions under packages/@nitpicker/crawler/src/archive/phase6e/, orchestrated by verifyPhase6Migration. The orchestrator now runs inside the same knex.transaction() block as populatePhase6DEntities so a thrown Phase6VerificationError rolls back every 6-D INSERT; ref tables from 6-B stay committed but are additive and idempotent on re-run. On success the orchestrator returns a Phase6VerificationSummary that the script logs to stdout so operators can audit content_items / page_meta / anchor_edges / image_items / resource_items counts from CI pipeline logs without re-opening the archive. Notable design decisions (details live in the per-file JSDoc): - Check #3 (anchor_edges vs anchors) enforces anchor_edges <= anchors rather than the strict `< count(anchors)` spelled in the issue text — legitimate small crawls whose (pageId, hrefId) pairs are all unique produce anchor_edges == anchors and are not a failure. Check #4 (SUM(count) == count(anchors)) already covers the "no rows lost" invariant when counts happen to match. - Check #8 (URL round-trip) uses deterministic stride sampling (id % stride == 0) instead of ORDER BY RANDOM() so retries return the same verdict for the same archive, and LEFT JOINs so FK gaps surface as null-side rows rather than silently trimming the sample. - verifyPhase6Migration wraps non-Phase6VerificationError exceptions in a Phase6VerificationError with check='runtime' so operator log greps for "Phase 6 verification failed" catch driver-side failures uniformly. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #194. Phase 6-E of the write-model refactor (#103) — replaces the four ad-hoc row-count asserts in
scripts/migrate-to-phase6.mjswith the full set of eight issue #194 acceptance invariants, factored into standalone check functions underpackages/@nitpicker/crawler/src/archive/phase6e/. The orchestrator now runs inside the same 6-D populate transaction so a failing invariant rolls back every 6-D INSERT; ref tables from 6-B stay committed but are additive.Notable design decisions (details in per-file JSDoc):
<to<=— legitimate small crawls whose (pageId, hrefId) pairs are all unique are not a failure. Check analyze コマンドに主要プラグインオプションの CLI フラグを追加する #4 (SUM(count) == count(anchors)) already covers the no-rows-lost invariant when counts happen to match.verifyPhase6Migrationreturns a summary — the script logs content_items / page_meta / anchor_edges / image_items / resource_items counts on stdout so CI batch pipelines can audit outcomes without re-opening the archive.Phase6VerificationErrorwithcheck='runtime'so operator log greps forPhase 6 verification failedcatch driver-side failures uniformly.Scope explicitly excluded (deferred to #197 Phase 6-H): the acceptance's third bullet (
assert-compatible-version-phase6.spec.ts/ pre-Phase-6 CLI rejects a Phase-6 archive) — the plan doc placesinfo.phase6_completed_atand the reject check in 6-H; 6-E archives keep the legacy tables alongside the new entities so functional rejection is not yet needed.Test plan
yarn buildyarn lintyarn test— 3204 passed / 5 skippedphase6e/check-*.spec.ts) plusPhase6VerificationErrormessage formatting (types.spec.ts).migrate-to-phase6-script.spec.ts) runs the actual .mjs against anArchive.create-built fixture — happy path completes and writes output, phantom-row injection aborts with#1in stderr and leaves the input tar bit-identical.🤖 Generated with Claude Code