test(stackone-defender): QA fixture regression suite

[hiskudin]

Summary

Adds an automated regression test for the stackone-defender plugin that pins production scan decisions on a canonical set of 12 fixtures. Replaces the manual-checklist portion of QA handoff with a CI-runnable assertion.

• 4 benign/ fixtures → expected allowed: true
• 3 realistic/ fixtures (real injection attacks) → expected allowed: false
• 5 tricky/ fixtures (FP-bait content discussing/quoting attacks) → expected allowed: true, with one known-FP override pinned to current behavior

The test loads @stackone/defender directly using the same defender-daemon.config.json the daemon consumes in production. Runs offline — no daemon, no socket, no network.

How to run

cd plugins/security/stackone-defender
npm install
npm test

Currently green locally: 12/12 pass in ~0.5s after warmup.

What this replaces in the QA handoff

QA no longer needs to manually verify "does defender catch the demo fixtures?" — CI does it. What's left for QA is the behavioral layer:

• Install path works end-to-end
• TP fixture → Claude refuses + notifies
• FP fixture → Claude proceeds silently + proposes label
• Feedback loop writes to jsonl + posts to collector
• Daemon kill → auto-restart works

Test plan

• npm install && npm test from plugins/security/stackone-defender/ passes 12/12
• Adding a known-attack fixture to tests/fixtures/realistic/ causes the suite to assert against allowed: false automatically
• Bumping @stackone/defender and re-running surfaces any behavior drift

🤖 Generated with Claude Code

---

Summary by cubic

Adds a fast, offline regression test suite for the stackone-defender plugin that pins production decisions on 12 canonical fixtures to catch behavior drift in CI. Uses the daemon’s defender-daemon.config.json, adds an npm test script, and pins Node >=22.

• New Features

  • Loads @stackone/defender with production config; no daemon, socket, or network.
  • 12 fixtures: 4 benign (allowed), 3 realistic attacks (denied), 5 tricky FP-bait (allowed); one pinned override for a known FP.
  • Easy to extend by adding files under tests/fixtures/*; assertions adapt automatically.

• Bug Fixes

  • Deterministic, file-only fixture loading to avoid flaky runs across filesystems.

^{Written for commit 17eaf7c. Summary will update on new commits. Review in cubic}

[cubic-dev-ai]

No issues found across 14 files

<sub>Auto-approved: This PR only adds a test suite with fixture files and a single test script; no production code or configuration is modified, so the blast radius is limited to the test pipeline and cannot affect runtime behavior.

Re-trigger cubic</sub>

[Copilot]

Pull request overview

Adds an offline Node.js regression test suite for the stackone-defender plugin that runs a canonical set of QA fixtures through @stackone/defender using the production daemon config, asserting allowed decisions to catch behavior drift.

Changes:

• Added qa-fixtures.test.mjs to execute all fixtures under tests/fixtures/{benign,realistic,tricky} and assert decisions (with one pinned override).
• Added 12 new fixture files spanning benign content, realistic attacks, and tricky FP-bait content.
• Added an npm test script for running the Node test runner.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
plugins/security/stackone-defender/tests/qa-fixtures.test.mjs	New regression suite that loads production config and asserts allowed decisions per fixture bucket/override.
plugins/security/stackone-defender/package.json	Adds npm test script to run the new test suite.
plugins/security/stackone-defender/tests/fixtures/benign/sourdough-recipe.md	Benign fixture expected to be allowed.
plugins/security/stackone-defender/tests/fixtures/benign/lms-training-modules.txt	Benign fixture expected to be allowed.
plugins/security/stackone-defender/tests/fixtures/benign/hiking-trail.md	Benign fixture expected to be allowed.
plugins/security/stackone-defender/tests/fixtures/benign/git-log.txt	Benign fixture expected to be allowed.
plugins/security/stackone-defender/tests/fixtures/realistic/support-ticket.txt	Realistic injection fixture expected to be blocked.
plugins/security/stackone-defender/tests/fixtures/realistic/slack-thread.txt	Realistic injection fixture expected to be blocked.
plugins/security/stackone-defender/tests/fixtures/realistic/document-summary.md	Realistic injection fixture expected to be blocked.
plugins/security/stackone-defender/tests/fixtures/tricky/research-note-on-injection.md	Tricky FP-bait fixture with override pinned to current blocked behavior.
plugins/security/stackone-defender/tests/fixtures/tricky/release-notes-2.5.md	Tricky FP-bait fixture expected to be allowed.
plugins/security/stackone-defender/tests/fixtures/tricky/incident-postmortem.md	Tricky FP-bait fixture expected to be allowed.
plugins/security/stackone-defender/tests/fixtures/tricky/employee-policy.md	Tricky FP-bait fixture expected to be allowed.
plugins/security/stackone-defender/tests/fixtures/tricky/api-response-listing.json	Tricky structured-output fixture expected to be allowed.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Auto-approved: This PR adds only test fixtures and a regression test suite that runs the existing defender library against canonical examples; no production code, config, or dependencies are modified, and the blast radius is limited to CI assertion changes.

Re-trigger cubic</sub>