Skip to content

fix(#890): oracle-wiring gate — 69 unwired oracles → 145 wired / 7 manual / 0 undeclared - #894

Merged
avrabe merged 9 commits into
mainfrom
fix/oracle-wiring-gate-890
Jul 30, 2026
Merged

fix(#890): oracle-wiring gate — 69 unwired oracles → 145 wired / 7 manual / 0 undeclared#894
avrabe merged 9 commits into
mainfrom
fix/oracle-wiring-gate-890

Conversation

@avrabe

@avrabe avrabe commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

#890 — fix the bug FACTORY, not the bugs

scripts/repro/*.py are synth's execution oracles. 69 of 150 were referenced by
no workflow
— nothing ran them. v0.53 hand-wired three and the unwired count
still went up, because instances were fixed while the mechanism that produces
them was untouched.

The defect was never "69 broken gates". Many are legitimately manual — a pinned
vendor drop, real silicon, a toolchain CI does not install. The defect is that
nothing distinguished "manual by design" from "forgotten"
, so telling them apart
meant reading all 150 and reasoning about each — the audit that kept
rediscovering this one instance at a time.

What landed

before after
wired 82 145
manual (with a real reason) 7
unwired debt 0
undeclared / indistinguishable 70 0
  1. A declared status per script — one # ci-status: header line:
    wired · manual (<category>) — reason · unwired — reason.
    Header, not a manifest: it lives in the file it describes, so it shows up in
    the diff of the PR that adds the script and cannot outlive its own file. The
    repo already reserves the central-ledger shape (claims.yaml) for claims with
    no natural home; a script's CI status has one.
  2. A gatescripts/oracle_wiring_check.py, run as a step of the
    already-required claim-check job. Fails on: wired with no workflow
    reference (the inert-gate case), manual/unwired referenced by a workflow,
    a missing/placeholder reason, an out-of-vocabulary manual category, and
    anything undeclared — so new oracles must choose. Also catches a workflow
    step pointing at a repro file that no longer exists.
  3. A ratcheted, visible countclaims.yaml SYNTH-ORACLE-WIRING-890:
    wired ≥ 145 (a floor, so the surface can't be emptied by deleting
    harnesses), manual ≤ 7 (a ceiling on the escape hatch), unwired ≤ 0 (new
    debt is a red build, not a backlog entry). The gate also writes the counts and
    the debt list to $GITHUB_STEP_SUMMARY.
  4. 63 oracles newly wired. Every one verified green locally against a freshly
    built synth first, using the compile line from its own documented Run: block.
    Four new sweep jobs (selector / memory / rv32 / wcet) + one step in the
    existing fact-spec job. All 62 sweep steps were then re-run extracted
    verbatim from ci.yml
    — 62/62 green.

What was not being gated

The six mem757_* red-first reconstructions of gale's wrong-segment miscompile
(#757). Every RV32 oracle for a regression gale found on real silicon — #220
callee-saved, #226 live-range clobber, #232 overflow-guard clobber, #317 i64
sign, #343 if-join. All four WCET soundness cross-checks — the only thing
that executes the fixtures and checks bound_cycles >= executed_instructions,
i.e. the evidence behind the sound-WCET claim. The #494 rem_u identity
differential behind the beat-clang headline.

The manual seven (each with a real reason)

wake_path (external-input: gale's gist, not in-tree) · size_attribution_390,
local_promotion_headroom, vcr_dec_001_join_alloc_measure,
spill_baseline_measure.sh (measurement: reports with no verdict; the first is
pinned by a cargo test that is the gate) · run204_unicorn,
i64_load_store_372 (scratch: print-only probes with stale hardcoded inputs).

Two defects this lane found in itself

  • sret_decide_differential.py was a gate that could not fail — it printed
    MISMATCH <-- BUG and exited 0. Wiring it as-is would have added a vacuous
    gate. The verdict is now the exit status (mutating the -35 expectation to
    -34rc=1).
  • The gate's own CI step was vacuous, and a mutation caught it.
    set -o pipefail without -e means the shell's status is its last
    command's — the inert-gate mutation greened the step while the gate printed
    FAIL and exited 1. Now set -euo pipefail explicitly (not leaning on
    Actions' default bash -e), and the verdict is re-derived from the JSON
    summary the gate wrote (total, wired, undeclared, wired_unreferenced,
    failures) instead of trusting exit 0.

Red-first evidence — in real CI, not just locally

A gate nobody has seen fire is indistinguishable from one that cannot — the #890
thesis, applied to #890. So both required mutations were run through GitHub on a
throwaway branch carrying only the mutations (PR #895, closed unmerged, branch
deleted). The real Claim Check job went red at the Oracle-wiring step
(run):

oracle wiring: 152 repro scripts — 145 wired, 6 manual, 0 unwired(debt), 1 UNDECLARED
FAIL scripts/repro/mem757_ptr_base_copy_differential.py: UNDECLARED — add a
     `# ci-status:` header line ... An oracle nothing runs must SAY so.
FAIL scripts/repro/wake_path_differential.py: declares `wired` but NO workflow
     references it — the gate is INERT ...
##[error]Process completed with exit code 1.

Red-first mutation evidence — the full matrix

Each run executes the step extracted verbatim from ci.yml (yaml.safe_load
→ the step's run: block → bash -e), so there is no drift between what was
proved and what CI runs. Full transcript in scripts/repro/ORACLE_WIRING.md.

mutation step exit
baseline 0
M1 un-declare mem757_ptr_base_copy_differential.py 1UNDECLARED
M2 declare wired on wake_path (no reference at all) 1the gate is INERT
M3 delete a wired oracle's CI step (re-inert it) 1the gate is INERT
M4 make the gate's glob match nothing 1VACUOUS or DRIFTED ['total<100']
M5 demote a reference to a comment 1only in a COMMENT — prose does not run an oracle
M6 replace a manual reason with TODO 1needs a REAL reason
restored 0

M5 exists because the first version of this gate derived "referenced" from a raw
grep of the .yml — so a mention in a comment satisfied it. A gate satisfiable
by prose is the failure shape #890 rejects; references now come from the parsed
workflow (a step's run: body and its with:/env: values). No script changed
status (checked: zero of the 145 were wired by comment alone) — pure hardening.

Ratchet legs proved separately: flipping one wired script to manual fails both
count-min 145 and count-max 7.

Verification

cargo fmt --check 0 · cargo clippy --workspace --all-targets -D warnings 0 ·
cargo test --workspace 0 (130 suites) · frozen anchors 10/10 ·
claim_check 35/35 · oracle-wiring gate 0. No shipping code touched — .text
unchanged by construction.

Honest residual — a follow-up for repo admin

The gate is a required context (it lives inside Claim Check, one of main's
nine). The 63 newly-wired oracles are not: they run in four new
repro-sweep-* jobs, and main's branch protection has strict=false with only
those nine names. So a red sweep job will not block gh pr merge --auto — it has
to be read by eye until someone adds the four names to branch protection.

I deliberately did not change branch protection mid-release: a required context
whose name does not match a job that actually runs deadlocks every merge in
the repo. Filing it as the follow-up rather than doing it unilaterally.

Note for the coordinator

This gate is now active for the whole release: a lane that adds a
scripts/repro/*.py without a # ci-status: header will turn Claim Check
red. That is the intended forcing function; the failure message says exactly what
to add.

Generated docs (FEATURE_MATRIX.md, status.json) deliberately untouched — no
new status_fields were added, so nothing to regenerate (#805).

🤖 Generated with Claude Code

https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L

@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

avrabe and others added 9 commits July 30, 2026 16:47
…(step 1/5)

Adds scripts/oracle_wiring_check.py: every scripts/repro/*.py|*.sh must carry
exactly one `# ci-status:` declaration (wired | manual (<category>) — reason |
unwired — reason), and a `wired` declaration is VERIFIED against
.github/workflows/*.yml so a green board can never sit on top of an inert gate.

Step 1 declares the 82 scripts a workflow already references. The remaining 70
are UNDECLARED and the gate is red on them by construction — classified next.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
…ns (step 2/5)

Read and classified the 70 scripts no workflow referenced. 63 of them run green
against a freshly built synth right now (verified locally, one at a time) and
are wired in the next step; 7 are genuinely manual and say WHY, in a closed
category vocabulary (measurement / scratch / external-input) rather than free
text.

Three enabling fixes fall out of the audit:
  * base_cse / leaf_dead_frame / load_store_big_offset_382 / uxth_fold hardcoded
    ./target/{debug,release}/synth; they now honour $SYNTH (same literal default)
    so CI can point them at its own build.
  * sret_decide_differential printed "MISMATCH <-- BUG" and still exited 0 —
    wiring it as-is would have added a gate that CANNOT fail. The verdict is now
    the exit status (proved: mutating the -35 expectation to -34 gives rc=1).

The gate is deliberately still red: 63 files declare `wired` while no workflow
references them. That is the exact defect being fixed, and it is now MACHINE
VISIBLE instead of requiring a 150-file read.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
… 3/5)

Four new sweep jobs (selector / memory / rv32 / wcet) plus one step in the
existing fact-spec job run the 63 differentials that were sitting in
scripts/repro/ with nothing executing them. Every one was verified green
locally against a freshly built synth before being wired — the compile line
comes from each harness's own documented Run: block.

Highlights of what was NOT being gated: the six mem757_* red-first
reconstructions of gale's wrong-segment miscompile (#757); every RV32 oracle
for a regression gale found on real silicon (#220/#226/#232/#317/#343); all
four WCET soundness cross-checks — the only thing that EXECUTES the fixtures
and checks bound_cycles >= executed_instructions, i.e. the evidence behind the
"sound WCET bound" claim; and the #494 rem_u identity differential behind the
beat-clang headline.

The gate itself is wired IN THE SAME COMMIT, as a step in claim-check — an
ALREADY-REQUIRED context on main. A brand-new job would not be required, so it
could sit red for weeks without blocking anything: the exact failure mode.
It uses `set -o pipefail` and re-reads its own JSON summary to assert a
non-zero wired count over a non-empty script set, so a run that measured
nothing fails instead of greening on exit 0.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
claims.yaml SYNTH-ORACLE-WIRING-890 pins the totals the header comments cannot
express: wired >= 145 (a FLOOR — the oracle surface must not be emptied by
deleting harnesses instead of fixing them), manual <= 7 (a ceiling on the
legitimate escape hatch), unwired <= 0 (a ceiling of ZERO — a new un-wired
oracle is a red build, not a backlog entry). Proved by mutation: flipping one
wired script to manual fails BOTH the floor and the ceiling legs.

This generalizes SYNTH-GPIO-846-ORACLE-CI-WIRED, which hand-pins exactly one
script's CI wiring — the instance-at-a-time shape #890 exists to replace.

Also hardens the gate after a mutation found a real defect in it: `pipefail`
without `-e` let the INERT-GATE mutation green the CI step, because the shell's
status is its last command's. The step now sets `-euo pipefail` explicitly and
the summary carries `wired_unreferenced` + `failures` so the verdict is
reachable from the summary alone, never from exit 0.

scripts/repro/ORACLE_WIRING.md documents the mechanism, argues header-over-
manifest, itemizes the manual seven, and carries the four-mutation transcript
(each run executes the step extracted VERBATIM from ci.yml, so there is no
drift between what was proved and what CI runs).

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
The gate derived "referenced by a workflow" from a raw grep of the .yml, so a
mention in a COMMENT satisfied it — a gate satisfiable by prose, which is the
exact failure shape #890 exists to reject. References now come from the PARSED
workflow: each step's `run:` body plus its `with:`/`env:` values. A workflow
that will not parse is a hard error, never a silent pass.

No script changed status (verified before the change: zero of the 145 wired were
wired by a comment alone), so this is pure future-proofing — but the diagnostic
now says WHY: "It IS mentioned in ci.yml, but only in a COMMENT — prose does not
run an oracle."

Mutation matrix against the final gate, each run executing the step extracted
VERBATIM from ci.yml (no drift between what was proved and what CI runs):

  baseline                                          EXIT 0
  M1 un-declare a script                            EXIT 1  UNDECLARED
  M2 declare wired, no reference at all             EXIT 1  the gate is INERT
  M3 delete a wired oracle's CI step                EXIT 1  the gate is INERT
  M4 gate's glob matches nothing                    EXIT 1  VACUOUS ['total<100']
  M5 reference demoted to a comment                 EXIT 1  only in a COMMENT
  M6 manual reason replaced with "TODO"             EXIT 1  needs a REAL reason
  restored                                          EXIT 0

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
A gate that has never been seen to fire is indistinguishable from one that
cannot — which is the #890 thesis applied to #890 itself. So both required
mutations were run through GitHub, not just locally: a throwaway branch carrying
only the two mutations turned the real `Claim Check` job red at the Oracle
wiring gate step (PR #895, closed unmerged, branch deleted).

  oracle wiring: 152 repro scripts — 145 wired, 6 manual, 0 unwired(debt), 1 UNDECLARED
  FAIL ... mem757_ptr_base_copy_differential.py: UNDECLARED ...
  FAIL ... wake_path_differential.py: declares `wired` but NO workflow references it
       — the gate is INERT ...
  ##[error]Process completed with exit code 1.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
The reverse check (a workflow step running a repro script that no longer exists)
scanned the RAW .yml, so a stale mention in a COMMENT would have been reported as
a "dangling CI step" — a false red, and the wrong diagnosis: untidy prose is not
a broken gate. It now scans the same executable surface the forward check uses.
Proved still live by mutation: adding a step that runs a nonexistent script
fails with the dangling-step message (EXIT 1).

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
…NGELOG after rebase

Two fixes on top of the rebase onto main (which now carries L4 #896 and the
bare-pipefail hardening #900).

1. **wat2wasm was missing.** Both failing sweeps died with
   `FileNotFoundError: 'wat2wasm'`. That is the gate DOING ITS JOB: these 63
   oracles had never run in CI, so their host dependencies had never been
   discovered — exactly the class that made the #881 VFP oracle fail on the
   runner while passing locally (#850). Several fixtures build their .wasm from
   .wat at test time. wabt installed in all four sweep jobs.

2. **Restored this lane's CHANGELOG entries.** My conflict resolution kept the
   `# ci-status:` header lines plus main's side, which is correct for the repro
   scripts and WRONG for an additive CHANGELOG — it silently dropped all four
   of the lane's bullets. Recovered verbatim from the pre-rebase branch and
   merged with the existing entry. Caught by counting bullets after the rebase
   rather than trusting that a clean `rebase --continue` meant a clean result.

Rebase conflict in vcr_dec_001_join_alloc_execution_differential.py resolved
keeping BOTH sides: L4's updated "increments 2+3" docstring and this lane's
ci-status header.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
@avrabe
avrabe force-pushed the fix/oracle-wiring-gate-890 branch from 3875f98 to cdca031 Compare July 30, 2026 14:48
@avrabe
avrabe merged commit c2aa8f8 into main Jul 30, 2026
50 checks passed
@avrabe
avrabe deleted the fix/oracle-wiring-gate-890 branch July 30, 2026 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant