Skip to content

fix(cache): re-anchor cached ids so a warm hit can't replay another root (#2257) - #2285

Open
Kaushik2003 wants to merge 1 commit into
Graphify-Labs:v8from
Kaushik2003:fix/2257-ast-cache-id-portability
Open

fix(cache): re-anchor cached ids so a warm hit can't replay another root (#2257)#2285
Kaushik2003 wants to merge 1 commit into
Graphify-Labs:v8from
Kaushik2003:fix/2257-ast-cache-id-portability

Conversation

@Kaushik2003

Copy link
Copy Markdown

Fixes #2257.

Problem

Extractors mint node ids from the path string they are handed (_make_id(str(path)), _file_node_id(path)), so an AST cache entry written under root A embeds A's slug in every id and edge endpoint. save_cached relativized only source_file — never the ids.

Because extract()'s id-remap / final-canonicalization passes key their rewrites off the current run's paths, an A-derived id matches no key on a warm hit under root B (a clone, a moved checkout, a second mount), so the stale machine slug survives into graph.json.

Reproduced before touching any code — the same corpus, warm vs cold under root B:

run B (WARM cache): ['tmp_..._a_dir_slug_aaa_corpus_app_py',
                     'tmp_..._a_dir_slug_aaa_corpus_pkg_mod_base', ...]
run B (COLD):       ['app', 'app_run', 'pkg_mod', 'pkg_mod_base', 'pkg_mod_base_hello']
warm == cold: False

This is distinct from #2231/#2243 (which fix producers on a cold run) and from #2199 (stat-index portability) — it is the AST result cache replaying pre-canonicalization ids.

Fix

Store entries root-anchored, re-anchor on read — the same store-portable / re-anchor-on-load contract source_file (#777) and the stat index (#2199) already use:

  • _relativize_ids_in (in save_cached, after the source_file pass) replaces the root's contribution with a $graphify-root$ marker.
  • _absolutize_ids_in (in load_cached) restores what the current run's extractor would mint — the pre-remap form every downstream pass expects, so a replay reproduces a cold run exactly and no other pass changed.

Details worth reviewing:

  • The anchor is derived per entry rather than assumed equal to the scan root (normalize_id distributes over path joins), so a symlinked root (macOS /tmp/private/tmp) or relative inputs decompose exactly.
  • Only absolute root spellings may anchor an id. A relative root ("src") is a strip candidate but never a restore candidate, so admitting it would rewrite an already-canonical src_utils_foo into <abs-root-slug>_utils_foo on the save_semantic_cache path (which forwards root unresolved). Covered by a regression test.
  • The walk covers the whole payload rather than a bucket list, because the id form is self-identifying (a long casefolded path slug). Hand-enumeration was already incomplete: this also reaches raw_calls[].caller_nid, swift_extensions[].nid, edges[].target_file, bash_sources[].source_file and *_type_table.path — the last three being resolution inputs that would otherwise still point at root A and diverge warm-vs-cold. Values only, never dict keys.
  • save_cached's deepcopy is now unconditional. The old truthiness gate skipped it for a payload whose only content lives outside nodes/edges (empty nodes beside a populated bash_sources), which would have let the transform mutate the caller's dict — and extract()'s id_remap is keyed on the absolute form, so that would break cold runs.

A cache entry on disk now looks like:

{ "id": "$graphify-root$_run_sh__entry",
  "source_file": "run.sh",
  "target_file": "$graphify-root$/lib/common.sh" }

Tests

tests/test_cache.py:

  • test_warm_cache_from_another_root_does_not_leak_that_root — the issue's scenario. Extract a python/C/bash/markdown corpus under root A, copy the tree and graphify-out to root B, extract under B on the warm cache. Asserts the run is genuinely warm (zero extractor invocations), no node id or edge endpoint carries A's slug, the on-disk entries hold neither A's slug (case-insensitive — ids are casefolded, paths are not) nor an absolute path, a cold run still yields canonical ids, and warm ≡ cold.
    • The fixture deliberately avoids JS/TS: those suffixes are in _JS_CACHE_BYPASS_SUFFIXES and never cached, which would make every warm assertion vacuous.
  • test_cached_ids_round_trip_under_the_same_root — the stored form must restore to the exact absolute-derived id, or a same-root warm hit would break extract()'s remap.
  • test_relative_root_does_not_reanchor_an_already_canonical_id — the strip/restore asymmetry above.

Full suite: 3787 passed. The 4 test_ollama_retry_cap.py failures are pre-existing in my environment (openai not installed) — confirmed by stashing this branch and re-running.

Additionally verified out-of-band on graphify's own 334-file tree (216 cached files, 0 warm misses, warm graph identical to cold at 3067 nodes / 5865 edges), across the default layout, --out (decoupled cache_root), a symlinked root, and the parallel process pool.

Note for the maintainer

Pre-fix entries carry no marker and their content hash never changes, so they cannot self-heal. They are swept when the release bumps the version, since AST entries live under cache/ast/v{version}/ — so the version bump is part of this fix landing, not incidental to it. Per repo convention this PR does not touch CHANGELOG.md or the version itself.

…oot (Graphify-Labs#2257)

Extractors mint node ids from the path STRING they are handed
(_make_id(str(path)), _file_node_id(path)), so an AST cache entry written
under root A embeds A's slug in every id and edge endpoint. save_cached
relativized only source_file, never the ids. Because extract()'s
id-remap / final-canonicalization passes key their rewrites off the
CURRENT run's paths, an A-derived id matches no key on a warm hit under
root B (a clone, a moved checkout, a second mount) and the stale
machine slug survives into graph.json. Distinct from Graphify-Labs#2231/Graphify-Labs#2243, which
fix producers on a cold run, and from Graphify-Labs#2199 (stat-index portability).

Entries are now stored root-anchored and re-anchored on read, the same
store-portable/re-anchor-on-load contract source_file (Graphify-Labs#777) and the
stat index (Graphify-Labs#2199) already use: _relativize_ids_in replaces the root's
contribution with a $graphify-root$ marker on write, _absolutize_ids_in
restores what the current run's extractor would mint on read. That is
the pre-remap form every downstream pass in extract() expects, so a
replay reproduces a cold run exactly and no other pass changed.

The anchor is derived per entry rather than assumed equal to the scan
root (normalize_id distributes over path joins), so a symlinked root or
relative inputs decompose exactly; only absolute root spellings may
anchor, or a relative root ("src") would rewrite an already-canonical
src_utils_foo into an absolute-derived id on the semantic path. The
walk covers the whole payload rather than a bucket list, since the id
form is self-identifying: that also reaches raw_calls[].caller_nid,
swift_extensions[].nid, edges[].target_file, bash_sources[].source_file
and *_type_table.path, the last three being resolution inputs that
would otherwise still point at root A. save_cached's deepcopy is now
unconditional; the old truthiness gate skipped it for a payload whose
only content lives outside nodes/edges, which would have let the
transform mutate the caller's dict and break cold-run remapping.

Pre-fix entries carry no marker and their content hash never changes,
so they cannot self-heal; they are swept when the release bumps the
version, since AST entries live under cache/ast/v{version}/.

Tests: extract a python/C/bash/markdown corpus under root A, copy the
tree and graphify-out to root B, extract under B on the warm cache, and
assert the run is genuinely warm (zero extractor calls), that no node id
or edge endpoint carries A's slug, that the on-disk entries hold neither
A's slug nor an absolute path, that a cold run still yields canonical
ids, and that warm and cold match exactly. The fixture avoids JS/TS on
purpose: those suffixes bypass the cache, which would make the warm
assertions vacuous.

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).


Graphify review — findings

This pull request introduces root-portability for cache entries by storing node ids and path values relative to a sentinel $graphify-root$ marker instead of the absolute root a cache was originally written under. New helpers (_id_anchor, _portability_anchors, _rewrite_strings, _relativize_ids_in, _absolutize_ids_in) strip the current root's slug on write and re-anchor it to the active run's root on read, with save_cached/load_cached wired to call them alongside the existing source_file relativization. It also changes the deep-copy in save_cached from being gated on populated node/edge buckets to being unconditional for dict results. The accompanying test suite additions cover portability across roots, symlinked roots, legacy unversioned/absolute entries, frontmatter/body hashing, semantic cache merge/prune/clear behavior, and namespace handling.

Worth a look

  • id restore drops marker when id_restore is empty, silently changing idgraphify/cache.py · Escalate · high
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
  • _absolutize_ids_in re-anchors any string starting with marker, including legitimate user datagraphify/cache.py · Escalate · medium
    • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 847 functions depend on the 192 node(s) this change touches.

Health — this change adds coupling hotspots:

  • worse: load_cached() — 39 callers, 5 callees
  • worse: save_cached() — 24 callers, 5 callees
  • worse: _flush_stat_index() — 6 callers, 2 callees

Verification — 847 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 562 function(s) in the blast radius were not formally verified this run

· 3 more finding(s) on lines outside this diff (see the check run).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AST cache replays node ids minted under the original root after a corpus move/clone

1 participant