From 1e119d0e8c963aebc084f4b9c09abadfbb7b4608 Mon Sep 17 00:00:00 2001 From: BohdanMar Date: Thu, 23 Jul 2026 12:57:20 +0200 Subject: [PATCH 1/2] feat(EC-1862): add AI skills for ec-cli Add 6 skills covering test execution, debugging, build/lint, PR checklist, test authoring, and benchmarking. Jira: EC-1862 --- .claude/skills/benchmark/SKILL.md | 68 +++++++++++++++ .claude/skills/build-and-lint/SKILL.md | 60 +++++++++++++ .claude/skills/debug-failure/SKILL.md | 76 ++++++++++++++++ .claude/skills/pr-checklist/SKILL.md | 45 ++++++++++ .claude/skills/run-tests/SKILL.md | 82 ++++++++++++++++++ .claude/skills/write-tests/SKILL.md | 115 +++++++++++++++++++++++++ 6 files changed, 446 insertions(+) create mode 100644 .claude/skills/benchmark/SKILL.md create mode 100644 .claude/skills/build-and-lint/SKILL.md create mode 100644 .claude/skills/debug-failure/SKILL.md create mode 100644 .claude/skills/pr-checklist/SKILL.md create mode 100644 .claude/skills/run-tests/SKILL.md create mode 100644 .claude/skills/write-tests/SKILL.md diff --git a/.claude/skills/benchmark/SKILL.md b/.claude/skills/benchmark/SKILL.md new file mode 100644 index 000000000..1d2adf8f0 --- /dev/null +++ b/.claude/skills/benchmark/SKILL.md @@ -0,0 +1,68 @@ +--- +name: benchmark +description: > + Run performance benchmarks for the Conforma CLI. Use when users ask "run benchmark", + "performance test", "stress test", "how fast", "benchmark data", "make benchmark", + or need help with performance measurement and profiling. +--- + +# Run Performance Benchmarks + +## Prerequisites + +- Docker or Podman running (testcontainers for local OCI registry) +- Benchmark data prepared (see below) + +## Two Benchmarks + +### Simple (single-component) + +Validates one golden-container image against the `@redhat` policy collection. + +```bash +make benchmark_data # prepare offline data (pulls from quay.io, ~760MB) +make benchmark # run the simple benchmark +``` + +Or directly: +```bash +cd benchmark/simple +./prepare_data.sh # one-time data prep +go run . # run benchmark +go run . -benchnum 5 # run 5 iterations +``` + +### Stress (multi-component, parallel) + +Validates a multi-component snapshot with configurable parallelism. + +```bash +cd benchmark/stress +./prepare_data.sh +go run . +``` + +Configure via environment variables: + +| Variable | Default | Purpose | +|----------|---------|---------| +| `EC_STRESS_COMPONENTS` | 10 | Number of components in the snapshot | +| `EC_STRESS_WORKERS` | 35 | Number of parallel workers | + +```bash +EC_STRESS_COMPONENTS=50 EC_STRESS_WORKERS=20 go run . +``` + +## Output Format + +Both benchmarks use `golang.org/x/benchmarks/driver` and output in the standard Go benchmark format (ns/op, memory stats). + +## Runtime Profiling + +The CLI has built-in profiling via the `--trace` flag: + +```bash +ec validate image --trace perf ... # Go runtime trace file +ec validate image --trace cpu ... # pprof CPU profile +ec validate image --trace mem ... # heap profile +``` diff --git a/.claude/skills/build-and-lint/SKILL.md b/.claude/skills/build-and-lint/SKILL.md new file mode 100644 index 000000000..a2e9716fb --- /dev/null +++ b/.claude/skills/build-and-lint/SKILL.md @@ -0,0 +1,60 @@ +--- +name: build-and-lint +description: > + Build and lint the Conforma CLI. Use when users ask "how to build", "make build", + "lint errors", "lint fix", "golangci-lint", "make generate", "generated files", + "go mod tidy", or need help with the build process and code quality checks. +--- + +# Build and Lint + +## Build + +```bash +make build # builds dist/ec__ for current platform +make dist # builds for all supported platforms +DEBUG_BUILD=1 make build # includes debugger symbols +``` + +## Lint + +```bash +make lint # golangci-lint + addlicense + tekton-lint (0 warnings enforced) +make lint-fix # auto-fix lint issues +make tekton-lint # lint Tekton task YAML only +``` + +Single-file verification: +```bash +golangci-lint run internal/evaluator/evaluator.go +gofmt -l internal/evaluator/evaluator.go +``` + +## Code Generation + +```bash +make generate +``` + +CI checks that `make generate` produces no uncommitted changes. If CI fails with "File was modified in build", run `make generate` locally and commit the results. + +## Multi-Module Project + +Three independent Go modules — run `go mod tidy` in the correct one: + +| Module | Path | What it covers | +|--------|------|----------------| +| Root | `go.mod` | CLI source code | +| Acceptance | `acceptance/go.mod` | Acceptance tests | +| Tools | `tools/go.mod` | Development tool dependencies | + +Adding a dependency to the wrong module causes build failures. + +## Container Images + +```bash +make build-image # build container image +make push-image # push to default registry +make task-bundle # push Tekton Task bundle +make dev # push ec + task bundle to kind cluster +``` diff --git a/.claude/skills/debug-failure/SKILL.md b/.claude/skills/debug-failure/SKILL.md new file mode 100644 index 000000000..5e605d0ee --- /dev/null +++ b/.claude/skills/debug-failure/SKILL.md @@ -0,0 +1,76 @@ +--- +name: debug-failure +description: > + Debug Conforma CLI test failures and runtime issues. Use when users ask "test + failed", "debug failure", "why is this failing", "preserve temp dir", "podman + error", "DNS resolution", "container failure", or need help troubleshooting. +--- + +# Debug a Failing Test or CLI Issue + +## Step 1: Preserve Debug State + +```bash +# CLI: preserve ec-work-* temp directories +ec validate image --debug ... +# or +EC_DEBUG=1 ec validate image ... + +# Acceptance: keep containers running after failure +make acceptance -- -persist + +# Reattach to persisted containers later +make acceptance -- -restore +``` + +## Step 2: Read the Output + +- **Unit/integration tests**: check the assertion message and stack trace +- **Acceptance tests**: check `features/__snapshots__/` for expected vs actual output. Snapshots are stored per feature file. +- **CLI runtime**: `ec-work-*` temp dirs (when `--debug` used) contain downloaded policies, OPA data, and evaluation artifacts + +## Step 3: Common Failure Patterns + +### DNS resolution failures +Binary is built with `CGO_ENABLED=0` (native Go DNS resolver). It cannot resolve second-level localhost domains. + +**Fix:** Add to `/etc/hosts`: +``` +127.0.0.1 apiserver.localhost +127.0.0.1 rekor.localhost +``` + +### Podman container failures +```bash +# Enable user podman socket +systemctl enable --user --now podman.socket + +# macOS: setup podman machine +./hack/macos/setup-podman-machine.sh +``` + +### inotify / key limit errors +```bash +# Linux +sudo sysctl fs.inotify.max_user_watches=524288 +sudo sysctl kernel.keys.maxkeys=1000 +``` + +### Go checksum mismatch +```bash +go env -w GOPROXY='https://proxy.golang.org,direct' +``` + +### Snapshot mismatch in acceptance tests +```bash +UPDATE_SNAPS=true make acceptance +``` + +### `make generate` produces uncommitted changes +CI runs `make generate` then checks `git diff --exit-code`. If it fails, run `make generate` locally and commit the changes. + +## Step 4: CI-Specific Issues + +- Harden Runner is disabled for acceptance tests due to DNS resolution conflicts +- CI installs tkn and kubectl from pinned versions (not from Go tools) +- CI runs `hack/ubuntu-podman-update.sh` before acceptance tests diff --git a/.claude/skills/pr-checklist/SKILL.md b/.claude/skills/pr-checklist/SKILL.md new file mode 100644 index 000000000..9f667943e --- /dev/null +++ b/.claude/skills/pr-checklist/SKILL.md @@ -0,0 +1,45 @@ +--- +name: pr-checklist +description: > + Definition of done checklist for Conforma CLI pull requests. Use when users ask + "is this PR ready", "definition of done", "PR checklist", "before merging", + "review checklist", or when preparing a PR for review. +--- + +# PR Definition of Done for Conforma CLI + +## Before Submitting + +- [ ] `make ci` passes (test + lint-fix + acceptance + tools-ci) +- [ ] `make generate` produces no uncommitted changes +- [ ] `go mod tidy` run in the correct module (root, acceptance/, or tools/) +- [ ] Acceptance test snapshots updated if behavior changed (`UPDATE_SNAPS=true make acceptance`) + +## Commit Messages + +Use conventional commits with Jira key: +``` +feat(EC-1234): add support for new attestation format +fix(EC-5678): handle empty predicate in bundle path +``` + +## PR Description + +Follow the template (`.github/pull_request_template.md`): +- **What:** What is this change doing? +- **Why:** Context and background +- **Tickets:** Link to Jira issue + +## Code Quality + +- [ ] No hardcoded values that should be configurable +- [ ] New code has appropriate test coverage (unit and/or acceptance) +- [ ] Build tags (`//go:build unit`) on all new test files +- [ ] No new lint warnings (`make lint` enforces 0 warnings) + +## Multi-Module Awareness + +If you changed dependencies: +- Root module: `go mod tidy` in repo root +- Acceptance tests: `cd acceptance && go mod tidy` +- Tools: `cd tools && go mod tidy` diff --git a/.claude/skills/run-tests/SKILL.md b/.claude/skills/run-tests/SKILL.md new file mode 100644 index 000000000..cd16ddbd6 --- /dev/null +++ b/.claude/skills/run-tests/SKILL.md @@ -0,0 +1,82 @@ +--- +name: run-tests +description: > + Run Conforma CLI tests. Use when users ask "how to run tests", "run unit tests", + "run acceptance tests", "make test", "test tags", "test timeout", "run a single + test", "ginkgo", or need help with test execution and environment setup. +--- + +# Run Conforma CLI Tests + +## Quick Start + +```bash +make test # unit + integration + generative tests +make acceptance # all acceptance tests (Cucumber/Gherkin, ~20min) +make ci # full CI: test + lint-fix + acceptance + tools-ci +``` + +## Test Tags and Timeouts + +| Tag | Timeout | What it covers | +|-----|---------|----------------| +| `unit` | 10s | Isolated function tests, mocked dependencies | +| `integration` | 15s | Tests with real OPA engine | +| `generative` | 30s | Property-based/fuzz tests | +| `acceptance` | 20m | End-to-end Cucumber scenarios with Testcontainers | + +## Running Specific Tests + +```bash +# Single test by name +go test -tags=unit ./internal/evaluator -run TestName + +# Single acceptance feature +make feature_validate_image + +# Single acceptance scenario (replace spaces with underscores) +make scenario_inline_policy + +# Focused acceptance tests (tag scenarios with @focus) +make focus-acceptance +``` + +## Acceptance Test Options + +```bash +# Keep test containers running after failure for debugging +make acceptance -- -persist + +# Reattach to persisted containers +make acceptance -- -restore + +# Update snapshot files +UPDATE_SNAPS=true make acceptance +``` + +## macOS Setup for Acceptance Tests + +Acceptance tests need Podman with a properly configured machine: + +```bash +./hack/macos/setup-podman-machine.sh # 4 CPUs, 8GB RAM +./hack/macos/run-acceptance-tests.sh +``` + +Also add to `/etc/hosts`: +``` +127.0.0.1 apiserver.localhost +127.0.0.1 rekor.localhost +``` + +## Environment Variables + +| Variable | Purpose | +|----------|---------| +| `UPDATE_SNAPS=true` | Update acceptance test snapshots | +| `E2E_INSTRUMENTATION=true` | Build coverage-instrumented binary | +| `EC_DEBUG=1` | Preserve `ec-work-*` temp directories | + +## Multi-Module Note + +This repo has 3 independent Go modules: root, `acceptance/`, and `tools/`. Run `go test` from the correct module root. `make test` handles the root module; `make acceptance` handles the acceptance module. diff --git a/.claude/skills/write-tests/SKILL.md b/.claude/skills/write-tests/SKILL.md new file mode 100644 index 000000000..fc70f439e --- /dev/null +++ b/.claude/skills/write-tests/SKILL.md @@ -0,0 +1,115 @@ +--- +name: write-tests +description: > + Write tests for the Conforma CLI. Use when users ask "add a test", "write a test", + "new test case", "how to test", "test pattern", "build tags", "snapshot testing", + "acceptance test", "cucumber", or need guidance on the test framework. +--- + +# Write Tests for Conforma CLI + +## Unit Tests + +Every test file starts with a build tag: +```go +//go:build unit + +package mypackage + +import ( + "testing" + "github.com/stretchr/testify/assert" +) + +func TestMyFunction(t *testing.T) { + result := MyFunction("input") + assert.Equal(t, "expected", result) +} +``` + +Use `testify/assert` for assertions and `testify/mock` for mocking. Timeout: 10s. + +Run: `go test -tags=unit ./internal/mypackage -run TestMyFunction` + +## Integration Tests + +Same pattern but with `//go:build integration` tag. These tests use the real OPA engine. Timeout: 15s. + +```go +//go:build integration + +package evaluator + +func TestEvaluatorWithRealOPA(t *testing.T) { + // test with actual OPA evaluation, no mocks +} +``` + +## Snapshot Testing + +Use `go-snaps` for snapshot assertions: +```go +import "github.com/gkampitakis/go-snaps/snaps" + +func TestOutput(t *testing.T) { + result := GenerateOutput() + snaps.MatchJSON(t, result) // or snaps.MatchSnapshot(t, result) +} +``` + +Update snapshots: `UPDATE_SNAPS=true go test -tags=unit ./...` + +## Acceptance Tests (Cucumber/Gherkin) + +Feature files live in `features/`. Step definitions in `acceptance/`. + +**Feature file** (`features/my_feature.feature`): +```gherkin +Feature: My feature + Scenario: it works + Given a known good image + When ec validates the image + Then the exit status is 0 +``` + +**Step definitions** are Go functions in `acceptance/` subdirectories, registered via Godog. The test framework uses: +- `framework.NewFramework()` for test setup +- Testcontainers for infrastructure (registry, API server, Rekor) +- WireMock for HTTP stubs +- Context values for state passing between steps (no global state) + +Run: `make scenario_it_works` + +## Test File Naming + +Follow existing patterns: +- `*_test.go` with appropriate build tag +- Unit tests alongside source: `evaluator.go` → `evaluator_test.go` +- Separate test files per concern: `evaluator_unit_core_test.go`, `evaluator_unit_data_test.go` + +## Table-Driven Tests + +Preferred pattern for multiple cases: +```go +cases := []struct { + name string + input string + expected string + expectErr string +}{ + {name: "valid input", input: "good", expected: "result"}, + {name: "invalid input", input: "bad", expectErr: "error message"}, +} + +for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + result, err := MyFunc(c.input) + if c.expectErr != "" { + assert.ErrorContains(t, err, c.expectErr) + return + } + assert.NoError(t, err) + assert.Equal(t, c.expected, result) + }) +} +``` From 6e8cbed33103d19474a6efa31f8ae8fc2663b412 Mon Sep 17 00:00:00 2001 From: BohdanMar Date: Thu, 23 Jul 2026 13:17:49 +0200 Subject: [PATCH 2/2] fix: correct acceptance test persist/restore commands Use `go test ./acceptance -args -persist` instead of `make acceptance -- -persist` which doesn't forward args. Jira: EC-1862 --- .claude/skills/debug-failure/SKILL.md | 6 +++--- .claude/skills/run-tests/SKILL.md | 9 +++++++-- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/.claude/skills/debug-failure/SKILL.md b/.claude/skills/debug-failure/SKILL.md index 5e605d0ee..8c6700a66 100644 --- a/.claude/skills/debug-failure/SKILL.md +++ b/.claude/skills/debug-failure/SKILL.md @@ -16,11 +16,11 @@ ec validate image --debug ... # or EC_DEBUG=1 ec validate image ... -# Acceptance: keep containers running after failure -make acceptance -- -persist +# Acceptance: keep containers running after failure (must use go test, not make) +cd acceptance && go test ./acceptance -args -persist # Reattach to persisted containers later -make acceptance -- -restore +cd acceptance && go test ./acceptance -args -restore ``` ## Step 2: Read the Output diff --git a/.claude/skills/run-tests/SKILL.md b/.claude/skills/run-tests/SKILL.md index cd16ddbd6..946f89981 100644 --- a/.claude/skills/run-tests/SKILL.md +++ b/.claude/skills/run-tests/SKILL.md @@ -43,12 +43,17 @@ make focus-acceptance ## Acceptance Test Options +Note: `-persist`/`-restore` require `go test` directly (not `make`), and use `./acceptance` not `./...`: + ```bash # Keep test containers running after failure for debugging -make acceptance -- -persist +cd acceptance && go test ./acceptance -args -persist # Reattach to persisted containers -make acceptance -- -restore +cd acceptance && go test ./acceptance -args -restore + +# Run with specific tags +cd acceptance && go test ./acceptance -args -tags=@focus # Update snapshot files UPDATE_SNAPS=true make acceptance