144: Load-simulation harness: in-cluster load generation with SLO gates and evidence

[agent-relay-code]

Part of Epic: Factory Verification & Environments (#141). Depends on the EnvironmentProvider (#142) and stack deployer (#143).

Why

"Fully tested end to end" includes behavior under load, not just a single happy-path request. The Factory needs to drive realistic traffic at a deployed stack and decide pass/fail against SLOs — otherwise a feature that works for one request but collapses at concurrency ships green.

What

1. Load runner — run a load generator (k6 recommended; Locust acceptable) against the environment's endpoints. Run it from a Cloudflare Container in the environment (or an external CI runner) so traffic exercises the real deployed Workers/Containers; Cloudflare Workers autoscale and Container instances scale within the env's cap.
2. Profile — a declarable load profile (per the verification-stack descriptor or a sidecar file): target endpoints + request shapes, RPS / virtual users / ramp / duration.
3. SLO gate — assert thresholds: p95/p99 latency, error rate, throughput floor. Return a structured pass/fail with the measured numbers.
4. Evidence — capture the run summary (latency histogram, error rate, RPS) as evidence JSON so the merge-gate and humans can inspect it.
5. Export a runLoad(env, profile): LoadResult surface from src/index.ts.

Acceptance

• Load run drives the provisioned environment's endpoints with a configurable profile.
• SLO gate returns pass/fail with measured p95/error-rate/throughput.
• Evidence JSON captured; thresholds are declarative.
• Exported from src/index.ts.

✅ End-to-end verification — Factory success criteria (REQUIRED)

"Compiles" is not done. Fire real load at a real deployed service and prove the SLO gate flips correctly.

Run it

npm run build && npm test

CLOUDFLARE_API_TOKEN=... npx tsx test/e2e/load-harness.e2e.ts

The E2E must:

1. Provision (EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142) + deploy (Verification-stack descriptor contract + deployer: repos declare their full stack, Factory brings it up #143) a sample service into a real dispatch namespace.
2. Run a load profile (e.g. 50 VUs, 30s) against the deployed endpoint → assert it completes and emits evidence JSON with real numbers.
3. Pass case: a lenient SLO (e.g. p95 < 2s, errors < 1%) against a healthy service → gate returns pass.
4. Fail case: a strict SLO (e.g. p95 < 1ms) OR load against a deliberately throttled/broken service → gate returns fail with the violated metric. (Proves the gate actually gates — not always-green.)
5. Tear down.

Deliverable

• test/e2e/load-harness.e2e.ts running a real load run (Container/CI k6) against a deployed Cloudflare environment with both a passing and a failing SLO assertion, one command in CI.

Anchor files

• new src/environments/load-harness.ts (+ k6 Container/CI runner)
• src/ports/environment.ts / src/environments/* (from EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142/Verification-stack descriptor contract + deployer: repos declare their full stack, Factory brings it up #143)
• src/observability/* (evidence shape precedent), src/index.ts

Fixes #144

---

Summary by cubic

Adds an in-cluster load harness that runs k6 as a Kubernetes Job, checks SLOs, and emits machine-readable evidence. Hardened to fail closed on zero-traffic and validate evidence integrity; this gates deployed stacks and fulfills #144 in the Factory Verification & Environments epic.

• New Features

  • runLoad(environment, profile) creates a k6 Job, parses results, applies SLOs, and returns pass/fail with measurements and evidence JSON (factory.load.evidence.v1).
  • Declarative profiles (JSON/YAML via loadLoadProfile(path)): targets (endpoint or URL), VU/RPS with optional ramp, thresholds, and latency histogram; arrival-rate and VU scenarios supported.
  • Evidence validation: strict schema checks (errorRate vs counts, latency ordering, cumulative histogram integrity); zero-traffic runs add a requestCount violation.
  • Kubernetes integration: KubectlLoadJobClient, hardened Job/ConfigMap with pinned DEFAULT_K6_IMAGE, and renderK6Script.
  • E2E + CI: kind-based test/e2e/load-harness.e2e.ts deploys a sample service, runs pass/fail SLOs, writes artifacts/load-e2e/*.json; GitHub Actions job creates a kind cluster, pins the kubectl context for the run, and uploads evidence artifacts bound to the head SHA.
  • Docs: docs/load-harness.md with usage and profile examples.

• Dependencies

  • Dev: add tsx and test:e2e:load script.

^{Written for commit 94d2454. Summary will update on new commits.}

[kjgbot]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Warning

Review limit reached

@kjgbot, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 29 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9a7e5a99-e1e7-43ce-9eb5-26ff1eef77fa

📥 Commits

Reviewing files that changed from the base of the PR and between aeec56c and 94d2454.

⛔ Files ignored due to path filters (1)

• package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (10)

• .github/workflows/ci.yml
• docs/load-harness.md
• package.json
• src/environments/k6-job.ts
• src/environments/load-harness.test.ts
• src/environments/load-harness.ts
• src/environments/load-profile.ts
• src/fleet/internal-fleet-client.test.ts
• src/index.ts
• test/e2e/load-harness.e2e.ts

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch factory/144-agentworkforce-factory-81a11a73

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[kjgbot]

@cubic-dev-ai ultrareview: focus on Kubernetes context isolation, cleanup, untrusted profile/resource inputs, k6 evidence correctness, and CI reproducibility.

[cubic-dev-ai]

@cubic-dev-ai ultrareview: focus on Kubernetes context isolation, cleanup, untrusted profile/resource inputs, k6 evidence correctness, and CI reproducibility.

@kjgbot Couldn't start the ultrareview: please upgrade to a paid plan to review bot-authored PRs.