Skip to content

fix(gate): block untested-data-access via the shared policy engine#183

Open
veksen wants to merge 1 commit into
mainfrom
fix-untested-data-access-gate
Open

fix(gate): block untested-data-access via the shared policy engine#183
veksen wants to merge 1 commit into
mainfrom
fix-untested-data-access-gate

Conversation

@veksen

@veksen veksen commented Jul 16, 2026

Copy link
Copy Markdown
Member

Goal

The CI⇄MCP verdict loop (Query-Doctor/Site epic #3493) wants each condition's pass/fail to come from the shared taxonomy and policy, not each surface's own severity. This PR makes the analyzer's test-presence gate stop deciding its own severity and adopt @query-doctor/core's taxonomy + policy engine. Closes Query-Doctor/Site#3541.

What

Before: a PR that changed data-access code with no related real-DB test got a "Warning" callout and a green check. A warning beside a passing check is invisible to downstream reviewers — human or agent — who key on the check status, so the finding was silently dropped (documented on a live PR in the issue thread).

After: the same PR gets a failing check by default. The comment block loses its "Warning" heading — it renders [!CAUTION] when blocking, framed as "could not verify", never "your query is broken". A repo can soften the gate to non-blocking (warn[!NOTE], neutral) or suppress it (off); those are config, not the default. Adding a real-DB test or triaging the flagged queries and re-running still goes green, unchanged.

How

Read in this order:

  • src/gate/policy.ts (new) — resolveVerdict maps a gate verdict to its CI conclusion via core's conclusionForVerdict (taxonomy #3498) + policyFor (policy #3500). uncertain-conservative-flag concludes failure; warn caps that at a surfaced-but-non-blocking neutral; off reports the condition as unsurfaced.
  • src/main.ts — resolves the verdict before posting the report (so off also drops the comment block and the comment can render the right framing), then carries the conclusion into the check: core.setFailed on failure, core.warning on neutral. Replaces the old always-core.warning path.
  • src/reporters/github/success.md.j2 — drops [!WARNING]; an inline conditional picks [!CAUTION] (blocking) or [!NOTE] (softened) from the resolved conclusion, kept on one line so the GitHub callout stays contiguous.
  • src/config.ts / src/reporters/reporter.tsAnalyzerConfig.conditionPolicies (optional) carries per-repo overrides; ReportContext.testPresenceConclusion drives the comment framing.

Note: conditionPolicies is read from the repo config ("conditionPolicies" in config) and is empty until Site adds it to RepoConfig/getRepoConfig — an additive, non-breaking follow-up. Until then every repo gets the safe default (fail); the gate honours an override the moment it arrives.

Scope note: this keeps the analyzer's inline TestPresenceVerdict (a precursor to the full CiVerdict contract #3497) rather than migrating to it, so the change stays surgical. It uses the two functions the issue names directly rather than evaluateRun, which would require the full contract.

Tests

  • src/gate/policy.test.ts (new) — resolveVerdict: blocks by default (failfailure), softens under warn (→neutral, surfaced), suppresses under off (unsurfaced), and keeps a pass verdict green even under fail (proving the taxonomy, not a blanket fail, drives it).
  • src/reporters/github/github.test.ts — the block renders [!CAUTION] for a blocking verdict and [!NOTE] for a softened one, and never [!WARNING].
  • Full suite green: 311 passed (30 files). Typecheck clean, build succeeds.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Query Doctor Analysis

3 queries analyzed

0 regressed · 0 improved · 0 new · 0 removed

2 pre-existing issues

Using assumed statistics (10000000 rows/table). For better results, sync production stats.

More detail → get_ci_run({ runId: "019f6e83-7bd6-77c5-aa11-6176a13ed686" }) · view run · docs

The test-presence gate flagged an unverified data-access change with a
"Warning" callout while the check stayed green. A warning beside a green
check is invisible to downstream reviewers — human or agent — who key on
the check status, so the finding got silently dropped (Site#3541).

Adopt the failure taxonomy (#3498) and policy engine (#3500) from
@query-doctor/core instead of a local severity: resolveVerdict maps the
verdict through conclusionForVerdict + policyFor, so
untested-data-access concludes failure by default and fails the check.
A repo can soften it to a non-blocking neutral (warn) or suppress it
(off) via conditionPolicies, threaded from the repo config once Site
sends it.

The comment block drops the [!WARNING] heading: a blocking conclusion
renders [!CAUTION], a softened one [!NOTE], framed as "could not verify",
never "your query is broken". Rerun after a real-DB test is added or the
queries are triaged still goes green, unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@veksen
veksen force-pushed the fix-untested-data-access-gate branch from a7436a0 to 57a8f00 Compare July 17, 2026 05:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant