Skip to content

P0: require authoritative evidence for policy findings#272

Merged
pengfei-threemoonslab merged 2 commits into
mainfrom
codex/p0-heuristic-evidence
Jul 14, 2026
Merged

P0: require authoritative evidence for policy findings#272
pengfei-threemoonslab merged 2 commits into
mainfrom
codex/p0-heuristic-evidence

Conversation

@pengfei-threemoonslab

@pengfei-threemoonslab pengfei-threemoonslab commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

This closes the P0 where names, descriptions, regexes, protocol defaults, and other heuristic evidence could be upgraded into high-confidence or blocking policy findings by rule metadata.

The release-affecting contract is now evidence-backed:

  • every semantic claim carries a typed evidence basis, deterministic claim ID, and derived policy eligibility;
  • policy predicates resolve to matched | not_matched | indeterminate | conflicting with typed support;
  • heuristic-only, mixed, unknown, and conflicting applicability becomes an unsuppressible evidence gap, never a blocking Finding;
  • a non-authoritative effect claim that outranks the proven effect now emits a mixed/unknown applicability gap instead of disappearing from current-surface controls;
  • rule severity, confidence, and block: true can only cap or consume evidence strength, never upgrade it;
  • conservative risk upper bounds remain available for diagnostics, while hard controls consume only authoritative claims;
  • baselines bind to support_hash, so unchanged fingerprints cannot silently accept changed evidence support;
  • --no-heuristics, suppressions, acknowledgements, baselines, and severity overrides cannot hide policy-evidence gaps;
  • strict mode keeps the evidence-aware tuple consistent (insufficient_evidence, would_fail_ci=true, exit 20).

The same support model now covers policy packs, action policies, current-surface controls, MCP audit, API readiness, auth/owner/ADK/evidence checks, release decisions, reports, packets, verifier/handoff projections, and process exit policy.

Public contract changes

  • package 0.16.0b5
  • runtime contract 16
  • report 0.33
  • packet 0.11
  • verifier 0.4
  • agent handoff v4
  • policy pack 0.4
  • baseline 0.8
  • capability standard 0.5
  • capability lock 0.6; diff 0.7
  • action snapshot 0.4
  • safety corpus/receipt/qualification v3
  • downstream local contract 5

Prior schemas remain frozen and readable. Finding fingerprints still exclude support metadata; removed heuristic-only findings are intentionally represented as non-baselineable evidence gaps.

Qualification

  • exact CI command: 3380 passed, 5 skipped, coverage 88.60%
  • static-only adapter suite: 396 passed
  • P0 policy-evidence matrix: 48/48, plus an end-to-end bounded create_refund regression proving strict insufficient_evidence / exit 20
  • existing 64 semantic and 48 binding canaries: green
  • schema generation check and Ruff: green
  • large-scan 10-second merge budget: green under both normal execution and CI's xdist/coverage path
  • four dedicated latency budgets: green
  • repeated packet/report goldens and generated contract copies: byte-stable

Fixed-SHA 19-PR corpus re-evaluation

I re-ran the unchanged W27 labels and exact base/head SHAs against this branch's source in fresh temporary clones. The result does not support an accuracy claim for 0.16.0b5:

label allow review IE block unscored
safe_to_merge (14) 0 1 1 0 12
needs_human (3) 0 1 0 0 2
must_block (2) 0 0 0 0 2

Only 3/19 cases produced scored release decisions; 15/19 stopped at init_failed and one at scan_failed. Both must_block cases were unscored, so the scorer reports blocked_recall=0.0 and must_block_caught=0.0. There were no evaluated auto-passes, but that is not evidence of safety because 16 cases never reached the release gate.

This failure is upstream of the policy-evidence change. A reproduced Stripe case fails during cold-start init --write because generated tool_sources[].id values collide; the Goose scan-failed case retains an unresolved CHANGE_ME.yaml. Those harness/adoption defects must be fixed and all 19 fixed-SHA cases re-run before making beta accuracy, recall, or migration-burden claims. This PR remains draft and does not treat the old W27 numbers as qualification for b5.

Review notes

This changes release trust roots and versioned schemas, so Shipgate correctly self-reports control.state=human_review_required, merge_verdict=insufficient_evidence, and can_merge_without_human=false. A coding agent cannot self-approve this PR.

No agent execution, user-code import, tool call, LLM inference, network lookup, or runtime-safety claim was added. This remains deterministic static analysis only.

Please focus review on:

  1. the producer allowlist and evidence-basis assignments;
  2. compensating gaps when higher non-authoritative effects are excluded from hard controls;
  3. tri-state all_of / any_of / none_of reduction and authoritative branch attribution;
  4. conversion of heuristic check matches into unsuppressible gaps;
  5. baseline support_hash migration, including the explicit pre-v0.8 re-save requirement;
  6. release-decision / exit-code consistency and schema projections;
  7. the cold-start corpus harness regression documented above.

@pengfei-threemoonslab pengfei-threemoonslab marked this pull request as ready for review July 14, 2026 03:16
@pengfei-threemoonslab pengfei-threemoonslab merged commit 358f7ee into main Jul 14, 2026
4 checks passed
@pengfei-threemoonslab pengfei-threemoonslab deleted the codex/p0-heuristic-evidence branch July 14, 2026 03:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant