test(golden): lock rendered behavior in a committed snapshot (#154 Phase 0) - #179
Open
defangdevs wants to merge 1 commit into
Open
test(golden): lock rendered behavior in a committed snapshot (#154 Phase 0)#179defangdevs wants to merge 1 commit into
defangdevs wants to merge 1 commit into
Conversation
, Phase 0) Snapshot every module-generated artifact — systemd units (text plus wantedBy/requiredBy trailers), published /etc files (sudoers, Caddyfile, fail2ban locals, agent guides), tmpfiles rules, and every referenced agent-box/agent-web store payload, followed recursively — from two golden configurations: nixosConfigurations.vm's modules, and the same plus a tests/golden-web.nix overlay (vm never enables web/selfUpdate, so the Caddy/ttyd/settings/webhook/update surface would otherwise go unlocked). Store hashes are normalized to a fixed placeholder, so the fixture is identical whichever system renders it (verified: aarch64 and x86_64 manifests are byte-identical after normalization) and content changes in generated scripts are caught by their bytes in payloads/, not hidden behind hash churn. The golden-snapshot flake check diffs the committed tests/golden/ against a fresh render; `nix run .#update-golden` regenerates the fixture after an intentional change so the PR diff IS the behavior review. Phases 1-2 of the portability refactor must keep this check green with a byte-identical fixture; flake.lock bumps regenerate it mechanically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NPHNnT2DsWHLhbbhsq2Wc7
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 0 of #154: a golden snapshot of every module-generated artifact, committed as a fixture and enforced by a new
golden-snapshotflake check, so Phases 1–3 (de-inlining payloads, purging antiquotes, template units) provably keep the rendered configuration byte-stable.What is snapshotted
Two configurations are rendered into
tests/golden/:vm/—nixosConfigurations.vm's modules, as named in the issue.web/— the same plustests/golden-web.nix. This is an addition to the issue's plan, called out here deliberately:hosts/vm.nixnever enablesweb/selfUpdate, so a vm-only snapshot would leave the entire Caddy/ttyd/settings-daemon/webhook/self-update surface unlocked — most of what Phases 1–3 touch. The overlay pins two terminal users (one codex, exercising sorted ttyd port assignment, rootUser selection and the codex remote-control path) plusselfUpdate.enable.Per configuration: systemd unit texts (with
wantedBy/requiredByas comment trailers), published/etcfiles (sudoers, the generated Caddyfile, fail2ban locals + the module's filter, agent guides), the module's tmpfiles rules, and — followed recursively from those texts — the bytes of every/nix/storepayload whose name mentions agent-box/agent-web: supervisor, session CLI, env-exec wrapper, attach scripts, password helpers, settings daemon, webhook CLI/spawner, spot monitor, update script, seeds. 60 files.How it stays arch- and hash-stable
Store hashes are normalized to a fixed placeholder. The golden systems are evaluated for the check's system (on x86_64 the vm entry is exactly
nixosConfigurations.vm's eval), and I verified the aarch64 and x86_64 manifests are byte-identical after normalization, so one committed fixture serves CI and Graviton dev boxes alike. Payload contents are in the fixture directly, so a one-byte script change fails the check even though hashes are normalized away.Workflow
nix build -L .#checks.<system>.golden-snapshot(wired into ci.yml after vm-closure, which already realizes the shared closure).nix run .#update-golden, review thetests/golden/diff, commit it with the change. Documented in AGENTS.md.Checks run
checks.aarch64-linux.golden-snapshot(green, matches fixture)checks.aarch64-linux.{multi-user,module-single-file,download-route,webhook-route,module-generated-up-to-date}(green)nixosConfigurations.vm+packages.x86_64-linux.vmdrvPath evals (green)No behavior change; the module itself is untouched.
Closes nothing; first checkbox of #154.
🤖 Generated with Claude Code
https://claude.ai/code/session_01NPHNnT2DsWHLhbbhsq2Wc7