P0: require authoritative evidence for policy findings#272
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This closes the P0 where names, descriptions, regexes, protocol defaults, and other heuristic evidence could be upgraded into high-confidence or blocking policy findings by rule metadata.
The release-affecting contract is now evidence-backed:
matched | not_matched | indeterminate | conflictingwith typed support;block: truecan only cap or consume evidence strength, never upgrade it;support_hash, so unchanged fingerprints cannot silently accept changed evidence support;--no-heuristics, suppressions, acknowledgements, baselines, and severity overrides cannot hide policy-evidence gaps;insufficient_evidence,would_fail_ci=true, exit20).The same support model now covers policy packs, action policies, current-surface controls, MCP audit, API readiness, auth/owner/ADK/evidence checks, release decisions, reports, packets, verifier/handoff projections, and process exit policy.
Public contract changes
0.16.0b5160.330.110.4v40.40.80.50.6; diff0.70.4v35Prior schemas remain frozen and readable. Finding fingerprints still exclude support metadata; removed heuristic-only findings are intentionally represented as non-baselineable evidence gaps.
Qualification
3380 passed, 5 skipped, coverage88.60%396 passed48/48, plus an end-to-end boundedcreate_refundregression proving strictinsufficient_evidence/ exit20Fixed-SHA 19-PR corpus re-evaluation
I re-ran the unchanged W27 labels and exact base/head SHAs against this branch's source in fresh temporary clones. The result does not support an accuracy claim for
0.16.0b5:Only
3/19cases produced scored release decisions;15/19stopped atinit_failedand one atscan_failed. Bothmust_blockcases were unscored, so the scorer reportsblocked_recall=0.0andmust_block_caught=0.0. There were no evaluated auto-passes, but that is not evidence of safety because 16 cases never reached the release gate.This failure is upstream of the policy-evidence change. A reproduced Stripe case fails during cold-start
init --writebecause generatedtool_sources[].idvalues collide; the Goose scan-failed case retains an unresolvedCHANGE_ME.yaml. Those harness/adoption defects must be fixed and all 19 fixed-SHA cases re-run before making beta accuracy, recall, or migration-burden claims. This PR remains draft and does not treat the old W27 numbers as qualification for b5.Review notes
This changes release trust roots and versioned schemas, so Shipgate correctly self-reports
control.state=human_review_required,merge_verdict=insufficient_evidence, andcan_merge_without_human=false. A coding agent cannot self-approve this PR.No agent execution, user-code import, tool call, LLM inference, network lookup, or runtime-safety claim was added. This remains deterministic static analysis only.
Please focus review on:
all_of/any_of/none_ofreduction and authoritative branch attribution;support_hashmigration, including the explicit pre-v0.8 re-save requirement;