Skip to content

feat(EC-1862): add AI skills for ec-cli #3434

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
BohdanMar wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat(EC-1862): add AI skills for ec-cli #3434

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1e119d0
feat(EC-1862): add AI skills for ec-cli
BohdanMar Jul 23, 2026
6e8cbed
fix: correct acceptance test persist/restore commands
BohdanMar Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions .claude/skills/benchmark/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
name: benchmark
description: >
Run performance benchmarks for the Conforma CLI. Use when users ask "run benchmark",
"performance test", "stress test", "how fast", "benchmark data", "make benchmark",
or need help with performance measurement and profiling.
---

# Run Performance Benchmarks

## Prerequisites

- Docker or Podman running (testcontainers for local OCI registry)
- Benchmark data prepared (see below)

## Two Benchmarks

### Simple (single-component)

Validates one golden-container image against the `@redhat` policy collection.

```bash
make benchmark_data # prepare offline data (pulls from quay.io, ~760MB)
make benchmark # run the simple benchmark
```

Or directly:
```bash
cd benchmark/simple
./prepare_data.sh # one-time data prep
go run . # run benchmark
go run . -benchnum 5 # run 5 iterations
```

### Stress (multi-component, parallel)

Validates a multi-component snapshot with configurable parallelism.

```bash
cd benchmark/stress
./prepare_data.sh
go run .
```

Configure via environment variables:

| Variable | Default | Purpose |
|----------|---------|---------|
| `EC_STRESS_COMPONENTS` | 10 | Number of components in the snapshot |
| `EC_STRESS_WORKERS` | 35 | Number of parallel workers |

```bash
EC_STRESS_COMPONENTS=50 EC_STRESS_WORKERS=20 go run .
```

## Output Format

Both benchmarks use `golang.org/x/benchmarks/driver` and output in the standard Go benchmark format (ns/op, memory stats).

## Runtime Profiling

The CLI has built-in profiling via the `--trace` flag:

```bash
ec validate image --trace perf ... # Go runtime trace file
ec validate image --trace cpu ... # pprof CPU profile
ec validate image --trace mem ... # heap profile
```
60 changes: 60 additions & 0 deletions .claude/skills/build-and-lint/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
name: build-and-lint
description: >
Build and lint the Conforma CLI. Use when users ask "how to build", "make build",
"lint errors", "lint fix", "golangci-lint", "make generate", "generated files",
"go mod tidy", or need help with the build process and code quality checks.
---

# Build and Lint

## Build

```bash
make build # builds dist/ec_<os>_<arch> for current platform
make dist # builds for all supported platforms
DEBUG_BUILD=1 make build # includes debugger symbols
```

## Lint

```bash
make lint # golangci-lint + addlicense + tekton-lint (0 warnings enforced)
make lint-fix # auto-fix lint issues
make tekton-lint # lint Tekton task YAML only
```

Single-file verification:
```bash
golangci-lint run internal/evaluator/evaluator.go
gofmt -l internal/evaluator/evaluator.go
Comment on lines +29 to +30

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== files =="
git ls-files | sed -n '1,120p' | rg 'Makefile|go.mod|SKILL\.md|tools/go.mod|evaluator.go' || true

echo
echo "== relevant skill excerpt =="
if [ -f .claude/skills/build-and-lint/SKILL.md ]; then
  nl -ba .claude/skills/build-and-lint/SKILL.md | sed -n '1,80p'
fi

echo
echo "== Makefile lint references =="
for f in Makefile build/*; do
  [ -f "$f" ] && echo "--- $f" && rg -n "golangci|lint|tools/go.mod" "$f" || true
done

echo
echo "== tools/go.mod if present =="
if [ -f tools/go.mod ]; then
  nl -ba tools/go.mod | sed -n '1,220p'
fi

echo
echo "== golangci references in repo =="
rg -n "golangci|tools/go.mod"|sed -n '1,200p' || true

Repository: conforma/cli

Length of output: 478

🏁 Script executed:

#!/bin/sh
set -euo pipefail

echo "== relevant skill excerpt =="
if [ -f .claude/skills/build-and-lint/SKILL.md ]; then
  awk '{printf "%6d\t%s\n", NR, $0}' .claude/skills/build-and-lint/SKILL.md | sed -n '1,80p'
fi

echo
echo "== Makefile lint references =="
awk '{printf "%6d\t%s\n", NR, $0}' Makefile 2>/dev/null | sed -n '1,220p' | rg -n "golangci|lint|tools/go.mod|.PHONY|run:" || true

echo
echo "== tools module files =="
git ls-files | rg '^tools/|go.mod|go.sum' || true

echo
echo "== golangci references in tracked files =="
rg -n "golangci|tools/go\.mod|cmd/golangci-lint|golangci-lint" . 2>/dev/null | sed -n '1,240p' || true

Repository: conforma/cli

Length of output: 10470

🏁 Script executed:

#!/bin/sh
set -euo pipefail

echo "== Makefile target definitions around lint =="
awk '{printf "%6d\t%s\n", NR, $0}' Makefile 2>/dev/null | sed -n '216,245p'

echo
echo "== root go.mod module line =="
awk 'NF && NR <= 6 {print}' go.mod

echo
echo "== tools.go import =="
awk '{printf "%6d\t%s\n", NR, $0}' tools/tools.go 2>/dev/null | sed -n '1,45p'

Repository: conforma/cli

Length of output: 3927

🌐 Web query:

golangci-lint v2 single file lint single file supported

💡 Result:

Yes, golangci-lint supports linting a single file [1]. You can execute it by passing the file path directly to the run command, for example: golangci-lint run file1.go [1]. However, there are important limitations to be aware of: 1. Compilation Dependencies: Golangci-lint typically requires the code to compile to perform analysis [2]. If the single file you are linting has dependencies on other files or packages within your project, and those dependencies are not properly resolved or the package does not compile as a whole, the linting may fail [2]. 2. Linter Capability: Not all linters are designed to work on a single file [3]. Some linters require information from the entire package or project to function correctly (e.g., they might need to see all usages of a constant or type) [4][3][5]. If a linter requires this broader scope, it may not produce accurate results or might fail when restricted to a single file [3]. 3. No Mixing: When specifying paths, you cannot mix individual files with package directories in a single command [1]. To ensure correct analysis, it is generally recommended to run golangci-lint on the package or project level (e.g., golangci-lint run ./...) to provide the necessary context for all linters to function as intended [1][2].

Citations:

Use the repository-pinned linter invocation.

The lint targets run golangci-lint through tools/go.mod, but this example uses an unqualified global binary, which can fail on a clean checkout or run a different version. Update it to pinned tooling and document an appropriate package path for reliable coverage.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.claude/skills/build-and-lint/SKILL.md around lines 29 - 30, Update the lint
commands in the build-and-lint skill to invoke golangci-lint through the
repository-pinned tooling defined by tools/go.mod rather than an unqualified
global binary. Replace the single-file example with an appropriate package path
that provides reliable coverage, and keep the gofmt check aligned with that
target.

```

## Code Generation

```bash
make generate
```

CI checks that `make generate` produces no uncommitted changes. If CI fails with "File was modified in build", run `make generate` locally and commit the results.

## Multi-Module Project

Three independent Go modules — run `go mod tidy` in the correct one:

| Module | Path | What it covers |
|--------|------|----------------|
| Root | `go.mod` | CLI source code |
| Acceptance | `acceptance/go.mod` | Acceptance tests |
| Tools | `tools/go.mod` | Development tool dependencies |

Adding a dependency to the wrong module causes build failures.

## Container Images

```bash
make build-image # build container image
make push-image # push to default registry
make task-bundle # push Tekton Task bundle
make dev # push ec + task bundle to kind cluster
```
76 changes: 76 additions & 0 deletions .claude/skills/debug-failure/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
---
name: debug-failure
description: >

@fullsend-ai-review fullsend-ai-review Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] description-field-formatting

Trigger phrases in description fields vary in style across files: some use quoted command strings, others mix quoted commands with bare phrases. Minor inconsistency.

Debug Conforma CLI test failures and runtime issues. Use when users ask "test
failed", "debug failure", "why is this failing", "preserve temp dir", "podman
error", "DNS resolution", "container failure", or need help troubleshooting.
---

# Debug a Failing Test or CLI Issue

## Step 1: Preserve Debug State

```bash
# CLI: preserve ec-work-* temp directories
ec validate image --debug ...
# or
EC_DEBUG=1 ec validate image ...

# Acceptance: keep containers running after failure (must use go test, not make)
cd acceptance && go test ./acceptance -args -persist

# Reattach to persisted containers later
cd acceptance && go test ./acceptance -args -restore
```

## Step 2: Read the Output

- **Unit/integration tests**: check the assertion message and stack trace
- **Acceptance tests**: check `features/__snapshots__/` for expected vs actual output. Snapshots are stored per feature file.
- **CLI runtime**: `ec-work-*` temp dirs (when `--debug` used) contain downloaded policies, OPA data, and evaluation artifacts

## Step 3: Common Failure Patterns

### DNS resolution failures
Binary is built with `CGO_ENABLED=0` (native Go DNS resolver). It cannot resolve second-level localhost domains.

**Fix:** Add to `/etc/hosts`:
```
127.0.0.1 apiserver.localhost
127.0.0.1 rekor.localhost
```

### Podman container failures
```bash
# Enable user podman socket
systemctl enable --user --now podman.socket

# macOS: setup podman machine
./hack/macos/setup-podman-machine.sh
```

### inotify / key limit errors
```bash
# Linux
sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl kernel.keys.maxkeys=1000
```

### Go checksum mismatch
```bash
go env -w GOPROXY='https://proxy.golang.org,direct'
```

### Snapshot mismatch in acceptance tests
```bash
UPDATE_SNAPS=true make acceptance
```

### `make generate` produces uncommitted changes
CI runs `make generate` then checks `git diff --exit-code`. If it fails, run `make generate` locally and commit the changes.

## Step 4: CI-Specific Issues

- Harden Runner is disabled for acceptance tests due to DNS resolution conflicts
- CI installs tkn and kubectl from pinned versions (not from Go tools)
- CI runs `hack/ubuntu-podman-update.sh` before acceptance tests
45 changes: 45 additions & 0 deletions .claude/skills/pr-checklist/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
name: pr-checklist
description: >
Definition of done checklist for Conforma CLI pull requests. Use when users ask
"is this PR ready", "definition of done", "PR checklist", "before merging",
"review checklist", or when preparing a PR for review.
---

# PR Definition of Done for Conforma CLI

## Before Submitting

- [ ] `make ci` passes (test + lint-fix + acceptance + tools-ci)
- [ ] `make generate` produces no uncommitted changes
- [ ] `go mod tidy` run in the correct module (root, acceptance/, or tools/)
- [ ] Acceptance test snapshots updated if behavior changed (`UPDATE_SNAPS=true make acceptance`)

## Commit Messages

Use conventional commits with Jira key:
```
feat(EC-1234): add support for new attestation format
fix(EC-5678): handle empty predicate in bundle path
```
Comment on lines +21 to +24

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add a language identifier to the commit example fence.

This violates markdownlint MD040. Use text (or another appropriate language) for the example block.

🧰 Tools
🪛 markdownlint-cli2 (0.23.0)

[warning] 21-21: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.claude/skills/pr-checklist/SKILL.md around lines 21 - 24, Update the
commit-message example fenced block in SKILL.md to include the text language
identifier, preserving its existing example content.

Source: Linters/SAST tools


## PR Description

Follow the template (`.github/pull_request_template.md`):
- **What:** What is this change doing?
- **Why:** Context and background
- **Tickets:** Link to Jira issue

## Code Quality

- [ ] No hardcoded values that should be configurable
- [ ] New code has appropriate test coverage (unit and/or acceptance)
- [ ] Build tags (`//go:build unit`) on all new test files
- [ ] No new lint warnings (`make lint` enforces 0 warnings)

## Multi-Module Awareness

If you changed dependencies:
- Root module: `go mod tidy` in repo root
- Acceptance tests: `cd acceptance && go mod tidy`
- Tools: `cd tools && go mod tidy`
87 changes: 87 additions & 0 deletions .claude/skills/run-tests/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---

@fullsend-ai-review fullsend-ai-review Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] content-duplication-with-AGENTS.md

Significant portions of the skill files duplicate content already in AGENTS.md (test tags/timeouts, build commands, lint commands, troubleshooting, CGO/DNS info). CLAUDE.md designates AGENTS.md as the single source of truth. This creates a dual-maintenance burden. Consider documenting duplication as intentional for skill self-containment, or referencing AGENTS.md sections.

Suggested fix: Add a note documenting duplication as intentional for self-containment, or reference AGENTS.md sections instead of duplicating.

name: run-tests
description: >
Run Conforma CLI tests. Use when users ask "how to run tests", "run unit tests",
"run acceptance tests", "make test", "test tags", "test timeout", "run a single
test", "ginkgo", or need help with test execution and environment setup.
---

# Run Conforma CLI Tests

## Quick Start

```bash
make test # unit + integration + generative tests
make acceptance # all acceptance tests (Cucumber/Gherkin, ~20min)
make ci # full CI: test + lint-fix + acceptance + tools-ci
```

## Test Tags and Timeouts

| Tag | Timeout | What it covers |
|-----|---------|----------------|
| `unit` | 10s | Isolated function tests, mocked dependencies |
| `integration` | 15s | Tests with real OPA engine |
| `generative` | 30s | Property-based/fuzz tests |
| `acceptance` | 20m | End-to-end Cucumber scenarios with Testcontainers |

## Running Specific Tests

```bash
# Single test by name
go test -tags=unit ./internal/evaluator -run TestName

# Single acceptance feature
make feature_validate_image

# Single acceptance scenario (replace spaces with underscores)
make scenario_inline_policy

# Focused acceptance tests (tag scenarios with @focus)
make focus-acceptance
```

## Acceptance Test Options

Note: `-persist`/`-restore` require `go test` directly (not `make`), and use `./acceptance` not `./...`:

@fullsend-ai-review fullsend-ai-review Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] incorrect command invocation

The command 'cd acceptance && go test ./acceptance -args -persist' is inconsistent with the Makefile which uses 'go test .'. The skill copies this from acceptance/README.md — a pre-existing documentation inconsistency, not introduced by this PR.

Suggested fix: Align with the Makefile's 'go test .' form. Note this also requires updating acceptance/README.md.


```bash
# Keep test containers running after failure for debugging
cd acceptance && go test ./acceptance -args -persist

# Reattach to persisted containers
cd acceptance && go test ./acceptance -args -restore

# Run with specific tags
cd acceptance && go test ./acceptance -args -tags=@focus

# Update snapshot files
UPDATE_SNAPS=true make acceptance
```

## macOS Setup for Acceptance Tests

Acceptance tests need Podman with a properly configured machine:

```bash
./hack/macos/setup-podman-machine.sh # 4 CPUs, 8GB RAM
./hack/macos/run-acceptance-tests.sh
```

Also add to `/etc/hosts`:
```
127.0.0.1 apiserver.localhost
127.0.0.1 rekor.localhost
```
Comment on lines +72 to +75

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add language identifiers to both fenced blocks.

markdownlint MD040 requires every fenced code block to specify a language.

  • .claude/skills/run-tests/SKILL.md#L67-L70: mark the /etc/hosts block as text.
  • .claude/skills/debug-failure/SKILL.md#L38-L41: mark the /etc/hosts block as text.
🧰 Tools
🪛 markdownlint-cli2 (0.23.0)

[warning] 67-67: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

📍 Affects 2 files
  • .claude/skills/run-tests/SKILL.md#L67-L70 (this comment)
  • .claude/skills/debug-failure/SKILL.md#L38-L41
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.claude/skills/run-tests/SKILL.md around lines 67 - 70, Update the fenced
/etc/hosts blocks in .claude/skills/run-tests/SKILL.md lines 67-70 and
.claude/skills/debug-failure/SKILL.md lines 38-41 to declare the text language.
No other content changes are needed.

Source: Linters/SAST tools


## Environment Variables

| Variable | Purpose |
|----------|---------|
| `UPDATE_SNAPS=true` | Update acceptance test snapshots |
| `E2E_INSTRUMENTATION=true` | Build coverage-instrumented binary |
| `EC_DEBUG=1` | Preserve `ec-work-*` temp directories |

## Multi-Module Note

This repo has 3 independent Go modules: root, `acceptance/`, and `tools/`. Run `go test` from the correct module root. `make test` handles the root module; `make acceptance` handles the acceptance module.
Loading
Loading