v3.0.0: provenance you can check instead of trust

[whystrohm]

Closes the defects in AUDIT-v2.md, an audit of v2.0.0 that found 50 issues: 7 blocking, 28 major, 15 minor.

Most of them shared one root cause. Every artifact in the QA loop had a single fixed path, and every reference between artifacts was a filename. Both hold until someone regenerates a frame, edits a brand-lock mid-project, or runs the loop at the same time as a colleague. Then a filename still resolves while the thing behind it has changed, and nothing notices.

Breaking

The output directory layout moved, and the layout is this project's public interface.

Before	Now
output/critique.json	output/critiques/round-N/{shot_id}.critique.json
output/prompts/{generator}.txt	output/prompts/round-N/{generator}.txt
output/prompts/revised-{generator}.txt	output/prompts/round-N/revised-{generator}.txt
output/generated/{shot_id}.png	output/frames/round-N/{shot_id}.png

Existing trees still read: the tools fall back to generated/ and to a root critique.json, and critique schema 1.0 still validates with a warning. Nothing auto-migrates. That needs to be in the release note, not only the changelog.

A 12-shot project over 3 rounds used to produce 36 critiques and keep one, and which one depended on review order.

What to review closely

Ranked by how much damage a mistake would do, not by diff size.

1. tools/_template.py. A hand-rolled template engine, which is a classic place for bugs. Check the escaping and _find_block_end's nesting.
2. tools/validate_provenance.py. The hash-comparison logic is what the audit trail's credibility rests on. If it can be made to pass on a tampered tree, the rest is decoration.
3. critique.schema.json at 1.1. Every provenance field is required and nullable, deliberately: null records that an input was genuinely unavailable, a missing key records nothing. Worth arguing about.
4. tools/validate_prompts.py Rule 3. Exact substring matching on series_lock anchors. Strict by design. If it is too strict in practice, it will be found here first.

Added

• run.json, written once per run, pinning every input by SHA-256. Schema in skills/storyboard-architect/templates/run.schema.json.
• tools/validate_provenance.py, recomputes every recorded hash. Catches a frame regenerated after its critique, a brand-lock edited mid-project, a frame never reviewed, two operators colliding on one shot, skipped rounds. --require-accept is the pipeline stop condition as an exit code.
• tools/validate_shots.py, the first instance validator in the repo. shots.json and text-overlays.json had schemas and no way to check a file against them, so every rule lived in a SKILL.md as a checkbox.
• tools/validate_prompts.py, enforces the forge's hard rules, including verbatim series_lock anchors.
• tools/check.sh, one entry point for all 18 checks, called by CI.
• A --selftest on every validator, each constructing failing fixtures and failing if the check does not catch them.
• A worked run example, two shots across two rounds with real hashes and a rendered preview.

The finding worth reading

The forge documents four hard rules. Rule 3, series_lock anchors appear verbatim in every prompt, is the rule that makes shots in a series look like the same room. Nothing checked it.

Driving the pipeline through a real seven-shot brief drifted on it in all seven shots, with every other validator green. The worked-run fixture committed earlier in this same branch had drifted on it too.

environment was "minimalist home office, white walls, oak desk, single houseplant" and it got written as "a minimal home office with white walls and an oak desk". That reads better, which is the problem: paraphrasing an anchor is what writing good prose feels like, so it happens every time and nobody notices until six shots show six different rooms.

That is now validate_prompts.py, and it is the reason this branch has three commits after the first one.

Verification

• 18 checks pass in CI on Python 3.11, having been developed on 3.14
• the four bundled previews re-render byte-identically on Linux, so the determinism claim survives a change of OS and interpreter
• full pipeline driven end to end: 7 shots, 2 rounds, all four project validators pass, --require-accept returns 1 then 0, the rendered preview passes 15 deliverable checks
• the committed tree alone is sufficient, verified by exporting HEAD clean and running the suite there
• install, uninstall, and a no-dependencies first run all behave
• all 12 tools compile under Python 3.9 and defer annotations

Not done

Named here rather than left to be discovered:

• No approval log. run.json records what was built and reviewed. Who approved it, and when, is recorded nowhere.
• No migration tool for a v2.0.0 output tree.
• The v2.0.0 tag is still published one commit behind main, and its LICENSE is the short-form notice rather than the full text.
• The explainer video and demo GIF are v0.1.0 and still show Runway/Sora. remotion/src/ShotkitExplainer.tsx is deliberately left matching the artifact it produced; both are labelled.
• storyboard.md.tpl uses dotted paths ({{enter.at}}) that _template.py does not support, and a single {{on_screen_text_resolved}} that cannot represent a multi-overlay shot. Nothing is broken today because a model renders it, not the Python. It needs a decision about whether that template should be machine-renderable at all.
• Nothing enforces any of this at runtime. The validators are CLI tools a human chooses to run. A team can skip the critic and ship, and no mechanism objects. Turning conventions into linters is progress, not a control.

Review note

This branch was written by Claude and its selftests were written by the same author, so they share the same blind spots. The Rule 3 finding is the argument for why a second reader matters: it was invisible to inspection, including its own, and only surfaced by executing the pipeline rather than reading it.