Reconcile the two snippet gates: fix audit cache handling, add RENDERING bucket - #265
Merged
Merged
Conversation
…ING bucket (#257) evidence_snippet_audit.py and `just validate-references` disagreed 1-vs-14 on hCom2_Complex_Gut_Microbiome.yaml. Root cause was two bugs in the audit, both making it report false MISMATCHes, plus one genuine difference in matching strictness that is now surfaced instead of hidden. Bug 1 - a real full text was discarded as a stub. A .md cache was trusted only with `content_type:` frontmatter or a `## Content` heading. Several caches were fetched with neither (they start "Full text (re-fetched ... via NCBI BioC PMC)"), including a 200 KB, a 138 KB and an 88 KB full text -- 55 files, 642 KB total. The audit fell back to the short abstract .txt and flagged every full-text snippet as absent. FULLTEXT_MARKER now counts as a real-content signal. Bug 2 - the curated-snippet stripper ate real text. SNIPPET_SECTION terminated on `\nURL:`, `\nDOI:`, `\n## ` or end-of-file, so in a cache laid out as notes -> snippets -> full text it consumed the full text too. The full-text markers are now terminators. Net: MISMATCH 166 -> 140, NOCONTENT 794 -> 756, and hCom2 goes 14 -> 0. New RENDERING bucket. The residual disagreement is snippets that match only after punctuation/whitespace/Greek normalisation -- record "10% CO 2 , 5% H 2" vs cached "10% CO2, 5% H2", "beta-5" vs "β-5". These are faithful quotes of the PAPER whose CACHE carries a PDF/XML extraction artefact, and they are exactly what validate-references reports. Counting them separately (62 repo-wide, `--list-rendering`) makes the two gates reconcile: validator errors == RENDERING + MISMATCH. Verified on hCom2: audit 1 RENDERING / 0 MISMATCH, validator 1 error, same snippet. Explicitly NOT done: rewriting those snippets to match the cache. A draft that did so produced "cDCE (4 to 5 mg/ L)" and "group 5 ( prmAB" -- propagating extraction artefacts into records to satisfy a tool. The script was discarded and the reasoning recorded in-code so it is not retried. "Total checks: N" counting issues rather than checks lives in the pip package and cannot be patched here; the justfile recipe now says so, alongside how to read a RENDERING hit. Added `just audit-snippets`. Verification: 266 tests pass; no new ruff findings (5 pre-existing, unchanged). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
realmarcin
added a commit
that referenced
this pull request
Jul 29, 2026
- PR #255 Suillus-Bacillus thiamine SynCom: merged after review (id 000312, no collision, validates, no snippet mismatches, dangling-check clean). Record count 305. Logged an optional causal-edge pass as follow-up since it carries no downstream edges. - #258 dangling edges: DONE — 7 retargeted, 7 dropped as self-referential restatements whose claims survive verbatim in the source node descriptions. 40 -> 26 integrity issues, dangling 14 -> 0. No nodes minted. - #257 snippet gates: DONE — fixed two audit cache bugs (a real full text discarded as a stub; the snippet stripper eating full text past a notes section), MISMATCH 166 -> 140, NOCONTENT 794 -> 756, and added the RENDERING bucket so validator errors == RENDERING + MISMATCH. - 000031 rename: DONE (#266), id unchanged, derived artifacts regenerated. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #257.
evidence_snippet_audit.pyandjust validate-referencesdisagreed 1 vs 14 on the same file. The cause was two bugs in the audit, both producing false MISMATCHes, plus one genuine difference in matching strictness that is now surfaced instead of hidden.Bug 1 — a real full text was discarded as a stub
A
.mdcache was trusted only withcontent_type:frontmatter or a## Contentheading. Several were fetched with neither — they beginFull text (re-fetched … via NCBI BioC PMC)— including a 200 KB, a 138 KB and an 88 KB full text. 55 files, 642 KB total. The audit silently fell back to the short abstract.txtand flagged every full-text snippet as absent.Bug 2 — the curated-snippet stripper ate real text
SNIPPET_SECTIONterminated on\nURL:,\nDOI:,\n##or EOF. In a cache laid out notes → snippets → full text, it consumed the full text too. The full-text markers are now terminators.New RENDERING bucket — this is the actual reconciliation
The residual disagreement is snippets matching only after punctuation/whitespace/Greek normalization — record
10% CO 2 , 5% H 2vs cached10% CO2, 5% H2;beta-5vsβ-5. These are faithful quotes of the paper whose cache carries a PDF/XML extraction artefact, and they are precisely whatvalidate-referencesreports.Counting them separately (62 repo-wide,
--list-rendering) makes the gates reconcile:Verified on hCom2: audit reports 1 RENDERING / 0 MISMATCH, validator reports 1 error — the same snippet, now correctly classified as an artefact rather than a fabrication suspect.
What I deliberately did NOT do
I drafted a script to rewrite those 62 snippets to match the cache exactly. Its dry run produced rewrites like
cDCE (4 to 5 mg/L)→cDCE (4 to 5 mg/ L)andgroup 5 (prmABC→group 5 ( prmAB.That propagates extraction artefacts into the records to satisfy a tool. The script was discarded and the reasoning left in-code so it isn't retried.
Total checks label
Counting issues rather than checks lives in the pip package and can't be patched here. The justfile recipe now says so, and explains how to read a RENDERING hit. Added
just audit-snippets.Verification
266 tests pass; no new ruff findings (5 pre-existing in
scripts/, unchanged).🤖 Generated with Claude Code