#479 shrink-guard compares TOTAL node count, so an AST-layer gain can mask total collapse of the semantic layer — and its 'you deleted files' message misreads a dedup win as data loss

[Abdallah01]

Summary

The #479 shrink-guard in to_json() refuses to overwrite graph.json when the
new graph has fewer total nodes. But a graph built through SKILL.md has two
layers produced by completely independent paths — AST (deterministic, free) and
semantic (LLM or Agent-tool subagents, expensive) — and a guard on the sum cannot
protect either one. A gain in the AST layer will mask a total collapse of the
semantic layer, and the write goes through reporting success.

Why this matters

The two layers do not fail together, and they cost wildly different amounts to
produce. On our most recent full re-extraction the AST layer grew from 18,079 to
19,938 nodes. Had the semantic pass returned nothing at all — every subagent
dying on the output ceiling described in #1758, say, which is a real and
observed failure mode — the merged total would still have comfortably cleared
the previous graph's node count. The guard would have written a graph.json
that had silently lost the entire layer that took millions of tokens to build,
and reported nothing wrong.

We didn't hit that, but only by luck of arithmetic: the guard fired on our run
for a reason unrelated to the risk it exists to protect against (the AST layer
also shrank, from deduplication).

Second problem: the message points at the wrong conclusion

When the guard does fire, it says:

ERROR: refused to shrink graphify-out/graph.json (existing graph has more nodes; #479).
If this shrink is intentional (you deleted files), re-run a full build with --force.

Deleted files is offered as the explanation for a shrink. Our shrink had no
deleted files. It was two opposite effects netting out, and the larger one was
a bug fix:

layer	before	after	cause
code	14,568	13,209	duplicate path-base copies collapsing — the fix
doc	3,505	2,061	composition inverted, not lost

The code layer shrank because a full re-extraction gave every node a canonical
ID, collapsing duplicates that a previous extractor had created under a second
path base (absolute-path IDs 1,486 → 92; one class appearing as 9 separate nodes
→ 1). That is precisely the outcome you want, and the guard's wording steers the
operator away from recognising their own fix.

The doc layer's shrink was likewise an improvement — its composition inverted:

file_type	before	after
document	3,162	114
concept	339	1,657
rationale	4	290

with INFERRED links 52 → 315 and doc↔code bridge links ~15 → 140. The old bulk
was per-heading fragments; the new layer is fewer, denser, better-connected
nodes. Node count went down, information went up.

Third problem (subtle): where you assert matters

Our own gate had this bug too, which is how we noticed it. We asserted a
doc-layer floor against .graphify_extract.json — the pre-build extraction —
and it passed at 2,167 against a floor of 2,103. Then build_from_json deduped
22,085 → 15,270 and the built graph's doc layer came in at 2,061, under our
own floor, silently. The extraction's node count is not the shipped node count.

Suggested fix

1. Assert per-layer, not on the total. Before writing, compare the previous
   graph's per-file_type counts to the new one's and fail on any layer that
   has an independent production path, rather than on the sum.
2. Measure the built graph, not the extraction. build_from_json dedups, so
   any assertion has to run after the build. Building to a scratch path and
   comparing before writing is cheap and makes the check honest.
3. Widen the message. (you deleted files) → (e.g. you deleted files, or duplicate nodes collapsed). The second cause is what a canonical-ID fix
   produces, and it will become more common as ID normalisation improves.
4. Consider reporting composition on a refusal. Printing the per-file_type
   delta alongside the refusal would let the operator see in one line whether
   they are looking at data loss or a dedup win. Right now both look identical:
   one number went down.

The general shape

A size guard can only ever ask did this shrink? — never did this get worse?,
which is the question that actually matters. "Fewer" and "worse" are independent
axes. For a merged artifact whose layers can move in opposite directions, the
aggregate is exactly the statistic that hides the failure.

Relation to existing issues

• Prevent partial-chunk overwrite in --update: pre-build node-count assertion + build_merge() helper #479 — the issue that introduced the guard. This isn't a regression in it;
  it's that a total-node-count assertion was always the wrong granularity for a
  two-layer artifact.
• _rebuild_code indexes markdown by AST but the /graphify skill does not — the two corpora fight over the shrink-guard #2089 — also about the shrink-guard, but a different mechanism: two
  corpora (_rebuild_code's markdown-by-AST vs the skill's) fighting over the
  same guard. This report is about two layers within one build, so the fixes
  are complementary rather than overlapping.
• Part B2 semantic-extraction subagents silently crash past the ~64k output-token turn limit on large/deep-mode chunks; SKILL.md chunking is file-count-based, not output-size-aware #1758 — the output-ceiling crash that makes semantic-layer collapse a
  realistic scenario rather than a hypothetical one. That issue is the cause
  this guard should be catching and currently can't.

Environment

• graphifyy 0.9.27, Windows 11
• Full re-extraction (not --update), default mode
• Semantic extraction via SKILL.md Part B2 Agent-tool subagents, no Gemini key
• Final graph 15,270 nodes / 22,527 links, written with force=True after
  verifying the reduction per-layer by hand — which is the manual work this
  report is asking the guard to do