From de35e09d67702b6ce3b4ed77e743e449fdc5ea2f Mon Sep 17 00:00:00 2001 From: yyyyaaa Date: Mon, 6 Jul 2026 17:20:33 +0700 Subject: [PATCH] chore(perf): cperf regression harness + scale-validate rig + capacity docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tooling + documentation only — zero runtime code changes. This is the apparatus that produced every number cited in this stack, landed so the results are reproducible and regressions are catchable. packages/perf-harness (cperf): - Reusable perf/regression CLI: fleet provision/teardown (tenant factories, canary seeding, drift detection), load harness (authenticated zipf traffic, churn driver), measurement (per-instance retained heap via heap snapshots, PG-side sampler, JSON-line collector), regression suites with baselines + compare, ramp/soak/soakOps runners, capacity prediction from the sizing model. 21 jest suites over the pure logic (histograms, RNG, fleetfiles, merge/summarize, scenario matrices). scripts/scale-validate: - The V1-V4 validation rig used for #1325: plans, drivers, samplers, summarizers. Kept as scripts (not a package) — they orchestrate a live fleet against a dedicated PG and document the exact invocations used. - SIZING.md: catalog-driven capacity model — the heap-aware instance budget, the R(H) law with worked examples, shard guidance, and the measured constants (filtered instance ~15MB, unfiltered ~1.3GB at 61k pg_class). - VALIDATION-REPORT.md / V2-RESULTS.md: findings ledger V1-V4 — every crash, root cause, fix, and the final verdicts (47-tenant headline run; churn storm; 24h-soak OOM root cause; heap floor 1024MB guidance). .gitignore: rig output artifacts (fleet manifests, metrics.jsonl, perf-out). Extracted from #1325 (commits 2eeaefbc1, f13857373, e54596547, 143f4795a, 8eec3cd7c, aeb8c35b4, acc035c98, f95d59fda and the rig fix-ups). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb --- .gitignore | 6 + packages/perf-harness/.gitignore | 3 + packages/perf-harness/DESIGN.md | 168 +++ packages/perf-harness/README.md | 465 +++++++ packages/perf-harness/jest.config.js | 18 + packages/perf-harness/package.json | 58 + packages/perf-harness/src/commands/fleet.ts | 120 ++ packages/perf-harness/src/commands/load.ts | 31 + packages/perf-harness/src/commands/measure.ts | 34 + .../perf-harness/src/commands/regression.ts | 44 + packages/perf-harness/src/commands/report.ts | 34 + packages/perf-harness/src/commands/run.ts | 28 + .../src/core/__tests__/args.test.ts | 65 + .../src/core/__tests__/config.test.ts | 83 ++ .../src/core/__tests__/fixtures/drifted.json | 6 + .../core/__tests__/fixtures/empty-fleet.json | 1 + .../src/core/__tests__/fixtures/fleet.json | 13 + .../src/core/__tests__/fleetfile.test.ts | 84 ++ .../src/core/__tests__/histogram.test.ts | 55 + .../src/core/__tests__/proc.test.ts | 86 ++ .../src/core/__tests__/rng.test.ts | 88 ++ packages/perf-harness/src/core/args.ts | 63 + packages/perf-harness/src/core/config.ts | 96 ++ packages/perf-harness/src/core/fleetfile.ts | 116 ++ packages/perf-harness/src/core/gql.ts | 271 ++++ packages/perf-harness/src/core/histogram.ts | 74 + packages/perf-harness/src/core/pgc.ts | 46 + packages/perf-harness/src/core/proc.ts | 163 +++ packages/perf-harness/src/core/rng.ts | 71 + .../src/fleet/__tests__/canary.test.ts | 44 + .../src/fleet/__tests__/provision.test.ts | 93 ++ .../src/fleet/__tests__/teardown.test.ts | 27 + packages/perf-harness/src/fleet/canary.ts | 170 +++ packages/perf-harness/src/fleet/discover.ts | 105 ++ packages/perf-harness/src/fleet/drift.ts | 105 ++ packages/perf-harness/src/fleet/provision.ts | 433 ++++++ packages/perf-harness/src/fleet/teardown.ts | 175 +++ packages/perf-harness/src/index.ts | 141 ++ .../src/load/__tests__/harness.test.ts | 126 ++ packages/perf-harness/src/load/churn.ts | 127 ++ packages/perf-harness/src/load/harness.ts | 784 +++++++++++ .../src/measure/__tests__/collector.test.ts | 115 ++ .../src/measure/__tests__/pgSampler.test.ts | 35 + .../perf-harness/src/measure/collector.ts | 266 ++++ .../perf-harness/src/measure/instanceHeap.ts | 206 +++ .../perf-harness/src/measure/pgSampler.ts | 146 ++ .../regression/__tests__/baselines.test.ts | 45 + .../src/regression/__tests__/compare.test.ts | 105 ++ .../regression/__tests__/scenarios.test.ts | 200 +++ .../src/regression/__tests__/sql.test.ts | 136 ++ .../perf-harness/src/regression/baselines.ts | 94 ++ .../perf-harness/src/regression/compare.ts | 143 ++ .../perf-harness/src/regression/scenarios.ts | 1209 +++++++++++++++++ packages/perf-harness/src/regression/sql.ts | 191 +++ packages/perf-harness/src/regression/suite.ts | 410 ++++++ .../src/report/__tests__/merge.test.ts | 164 +++ .../src/report/__tests__/summarize.test.ts | 98 ++ packages/perf-harness/src/report/merge.ts | 387 ++++++ packages/perf-harness/src/report/predict.ts | 99 ++ packages/perf-harness/src/report/summarize.ts | 83 ++ .../src/run/__tests__/ramp.test.ts | 48 + .../src/run/__tests__/soak.test.ts | 20 + .../src/run/__tests__/soakOps.test.ts | 55 + packages/perf-harness/src/run/ramp.ts | 376 +++++ packages/perf-harness/src/run/server.ts | 200 +++ packages/perf-harness/src/run/soak.ts | 363 +++++ packages/perf-harness/src/run/soakOps.ts | 149 ++ packages/perf-harness/tsconfig.esm.json | 9 + packages/perf-harness/tsconfig.json | 9 + pnpm-lock.yaml | 161 +++ scripts/scale-validate/SIZING.md | 205 +++ scripts/scale-validate/V2-RESULTS.md | 115 ++ scripts/scale-validate/VALIDATION-REPORT.md | 189 +++ scripts/scale-validate/_lib.mjs | 456 +++++++ scripts/scale-validate/canary-seed.mjs | 159 +++ scripts/scale-validate/churn-driver.mjs | 116 ++ scripts/scale-validate/collector.mjs | 207 +++ scripts/scale-validate/drift-tenants.mjs | 81 ++ scripts/scale-validate/drop-tenants.mjs | 155 +++ scripts/scale-validate/fleet.mjs | 107 ++ scripts/scale-validate/harness.mjs | 694 ++++++++++ .../scale-validate/measure-instance-heap.mjs | 235 ++++ scripts/scale-validate/pg-sampler.mjs | 112 ++ scripts/scale-validate/plans/v2-1152.json | 13 + scripts/scale-validate/plans/v2-1536.json | 13 + .../plans/v2-2048-rps-single.json | 16 + .../scale-validate/plans/v2-2048-tuned.json | 26 + scripts/scale-validate/plans/v2-2048.json | 25 + scripts/scale-validate/plans/v2-3584-r2.json | 13 + scripts/scale-validate/plans/v2-3584.json | 19 + scripts/scale-validate/plans/v2-896.json | 13 + scripts/scale-validate/soak-ops.mjs | 86 ++ scripts/scale-validate/subset-fleet.mjs | 102 ++ scripts/scale-validate/summarize-results.mjs | 46 + scripts/scale-validate/tenant-factory.mjs | 421 ++++++ scripts/scale-validate/v2-ramps.mjs | 342 +++++ 96 files changed, 14208 insertions(+) create mode 100644 packages/perf-harness/.gitignore create mode 100644 packages/perf-harness/DESIGN.md create mode 100644 packages/perf-harness/README.md create mode 100644 packages/perf-harness/jest.config.js create mode 100644 packages/perf-harness/package.json create mode 100644 packages/perf-harness/src/commands/fleet.ts create mode 100644 packages/perf-harness/src/commands/load.ts create mode 100644 packages/perf-harness/src/commands/measure.ts create mode 100644 packages/perf-harness/src/commands/regression.ts create mode 100644 packages/perf-harness/src/commands/report.ts create mode 100644 packages/perf-harness/src/commands/run.ts create mode 100644 packages/perf-harness/src/core/__tests__/args.test.ts create mode 100644 packages/perf-harness/src/core/__tests__/config.test.ts create mode 100644 packages/perf-harness/src/core/__tests__/fixtures/drifted.json create mode 100644 packages/perf-harness/src/core/__tests__/fixtures/empty-fleet.json create mode 100644 packages/perf-harness/src/core/__tests__/fixtures/fleet.json create mode 100644 packages/perf-harness/src/core/__tests__/fleetfile.test.ts create mode 100644 packages/perf-harness/src/core/__tests__/histogram.test.ts create mode 100644 packages/perf-harness/src/core/__tests__/proc.test.ts create mode 100644 packages/perf-harness/src/core/__tests__/rng.test.ts create mode 100644 packages/perf-harness/src/core/args.ts create mode 100644 packages/perf-harness/src/core/config.ts create mode 100644 packages/perf-harness/src/core/fleetfile.ts create mode 100644 packages/perf-harness/src/core/gql.ts create mode 100644 packages/perf-harness/src/core/histogram.ts create mode 100644 packages/perf-harness/src/core/pgc.ts create mode 100644 packages/perf-harness/src/core/proc.ts create mode 100644 packages/perf-harness/src/core/rng.ts create mode 100644 packages/perf-harness/src/fleet/__tests__/canary.test.ts create mode 100644 packages/perf-harness/src/fleet/__tests__/provision.test.ts create mode 100644 packages/perf-harness/src/fleet/__tests__/teardown.test.ts create mode 100644 packages/perf-harness/src/fleet/canary.ts create mode 100644 packages/perf-harness/src/fleet/discover.ts create mode 100644 packages/perf-harness/src/fleet/drift.ts create mode 100644 packages/perf-harness/src/fleet/provision.ts create mode 100644 packages/perf-harness/src/fleet/teardown.ts create mode 100644 packages/perf-harness/src/index.ts create mode 100644 packages/perf-harness/src/load/__tests__/harness.test.ts create mode 100644 packages/perf-harness/src/load/churn.ts create mode 100644 packages/perf-harness/src/load/harness.ts create mode 100644 packages/perf-harness/src/measure/__tests__/collector.test.ts create mode 100644 packages/perf-harness/src/measure/__tests__/pgSampler.test.ts create mode 100644 packages/perf-harness/src/measure/collector.ts create mode 100644 packages/perf-harness/src/measure/instanceHeap.ts create mode 100644 packages/perf-harness/src/measure/pgSampler.ts create mode 100644 packages/perf-harness/src/regression/__tests__/baselines.test.ts create mode 100644 packages/perf-harness/src/regression/__tests__/compare.test.ts create mode 100644 packages/perf-harness/src/regression/__tests__/scenarios.test.ts create mode 100644 packages/perf-harness/src/regression/__tests__/sql.test.ts create mode 100644 packages/perf-harness/src/regression/baselines.ts create mode 100644 packages/perf-harness/src/regression/compare.ts create mode 100644 packages/perf-harness/src/regression/scenarios.ts create mode 100644 packages/perf-harness/src/regression/sql.ts create mode 100644 packages/perf-harness/src/regression/suite.ts create mode 100644 packages/perf-harness/src/report/__tests__/merge.test.ts create mode 100644 packages/perf-harness/src/report/__tests__/summarize.test.ts create mode 100644 packages/perf-harness/src/report/merge.ts create mode 100644 packages/perf-harness/src/report/predict.ts create mode 100644 packages/perf-harness/src/report/summarize.ts create mode 100644 packages/perf-harness/src/run/__tests__/ramp.test.ts create mode 100644 packages/perf-harness/src/run/__tests__/soak.test.ts create mode 100644 packages/perf-harness/src/run/__tests__/soakOps.test.ts create mode 100644 packages/perf-harness/src/run/ramp.ts create mode 100644 packages/perf-harness/src/run/server.ts create mode 100644 packages/perf-harness/src/run/soak.ts create mode 100644 packages/perf-harness/src/run/soakOps.ts create mode 100644 packages/perf-harness/tsconfig.esm.json create mode 100644 packages/perf-harness/tsconfig.json create mode 100644 scripts/scale-validate/SIZING.md create mode 100644 scripts/scale-validate/V2-RESULTS.md create mode 100644 scripts/scale-validate/VALIDATION-REPORT.md create mode 100644 scripts/scale-validate/_lib.mjs create mode 100644 scripts/scale-validate/canary-seed.mjs create mode 100644 scripts/scale-validate/churn-driver.mjs create mode 100644 scripts/scale-validate/collector.mjs create mode 100644 scripts/scale-validate/drift-tenants.mjs create mode 100644 scripts/scale-validate/drop-tenants.mjs create mode 100644 scripts/scale-validate/fleet.mjs create mode 100644 scripts/scale-validate/harness.mjs create mode 100644 scripts/scale-validate/measure-instance-heap.mjs create mode 100644 scripts/scale-validate/pg-sampler.mjs create mode 100644 scripts/scale-validate/plans/v2-1152.json create mode 100644 scripts/scale-validate/plans/v2-1536.json create mode 100644 scripts/scale-validate/plans/v2-2048-rps-single.json create mode 100644 scripts/scale-validate/plans/v2-2048-tuned.json create mode 100644 scripts/scale-validate/plans/v2-2048.json create mode 100644 scripts/scale-validate/plans/v2-3584-r2.json create mode 100644 scripts/scale-validate/plans/v2-3584.json create mode 100644 scripts/scale-validate/plans/v2-896.json create mode 100644 scripts/scale-validate/soak-ops.mjs create mode 100644 scripts/scale-validate/subset-fleet.mjs create mode 100644 scripts/scale-validate/summarize-results.mjs create mode 100644 scripts/scale-validate/tenant-factory.mjs create mode 100644 scripts/scale-validate/v2-ramps.mjs diff --git a/.gitignore b/.gitignore index f2471a36d3..74875645f9 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,9 @@ postgres/pgsql-test/output/ .env.local graphql/server/logs/ graphql/server/*.heapsnapshot +scripts/scale-validate/out/ +scripts/scale-validate/fleet*.json +scripts/scale-validate/drifted.json +scripts/scale-spike/ +packages/perf-harness/perf-out/ +/metrics.jsonl diff --git a/packages/perf-harness/.gitignore b/packages/perf-harness/.gitignore new file mode 100644 index 0000000000..c71e46e3f3 --- /dev/null +++ b/packages/perf-harness/.gitignore @@ -0,0 +1,3 @@ +dist/ +.smoke-out/ +perf-out/ diff --git a/packages/perf-harness/DESIGN.md b/packages/perf-harness/DESIGN.md new file mode 100644 index 0000000000..faf0846ec4 --- /dev/null +++ b/packages/perf-harness/DESIGN.md @@ -0,0 +1,168 @@ +# @constructive-io/perf-harness — Design + +Reusable performance / regression harness for the Constructive GraphQL server. +Port of the ad-hoc `scripts/scale-validate/*.mjs` tooling (built during the +2026-07 blueprint-pooling validation program) into a proper monorepo package, +so the next time a memory/scale regression appears it is re-run with one +command instead of hand-driven scripts. + +**The originals in `scripts/scale-validate/` are the behavioral source of +truth. Port faithfully: every CLI flag, default, output field, and exit code is +preserved unless this document says otherwise. Do not redesign algorithms.** + +## Package identity + +- Directory: `packages/perf-harness` (auto-included by `pnpm-workspace.yaml` glob `packages/*`; lerna derives from pnpm — no registration anywhere) +- Name: `@constructive-io/perf-harness`, version `0.1.0`, license MIT, author `Constructive ` +- Bin: `{ "perf-harness": "index.js", "cperf": "index.js" }` (dist-relative like `packages/cli`; shebang `#!/usr/bin/env node` as FIRST line of `src/index.ts` — makage preserves it; no makage config file exists or is needed) +- `main: index.js`, `module: esm/index.js`, `types: index.d.ts`, `publishConfig: { access: public, directory: dist }`, NO `files` field (no sibling has one) +- Scripts (verbatim sibling standard): + `clean: makage clean` / `prepack: npm run build` / `build: makage build` / `build:dev: makage build --dev` / `dev: ts-node ./src/index.ts` / `lint: eslint . --fix` / `test: jest --passWithNoTests` / `test:watch: jest --watch` +- dependencies: `pg ^8.21.0` (imported directly), `inquirerer ^4.9.1`, `@inquirerer/utils ^3.3.9`, `graphile-cache workspace:^` (lazy-required ONLY inside `report predict` — see below) +- devDependencies: `@types/pg ^8.20.0`, `@types/node ^22.19.11`, `makage ^0.3.0`, `ts-node ^10.9.2`, `@inquirerer/test ^1.4.3` +- Do NOT declare typescript / jest / ts-jest / eslint / graphql — root-hoisted or irrelevant. +- `tsconfig.json`, `tsconfig.esm.json`, `jest.config.js`: byte-identical to `packages/server-utils` versions (extends `../../tsconfig.json`; ESM variant outDir `dist/esm`, module es2022, declaration false; jest = ts-jest preset, testRegex `(/__tests__/.*|(\.|/)(test|spec))\.(jsx?|tsx?)$`, modulePathIgnorePatterns `['dist/*']`, NO trailing commas). +- NO eslint config file in the package (root `.eslintrc.json` governs). Code style: 2-space, single quotes, semicolons, no trailing commas, `_`-prefixed unused args. + +## Hard rules (safety + fidelity) + +1. **Hub guardrail**: `HUB_PORTS = [5432, 3000, 3001, 3002, 9000]`. EVERY command that dials Postgres or a server port calls `assertPortAllowed(port, allowHub)` and fails with a clear error naming the flag `--allow-hub` to override. PG port default is **5433, never 5432** (preserve the `_lib.mjs` comment's intent). +2. **Credentials**: never hardcoded. `--email` defaults to `seeder@gmail.com`; `--password` has NO default — resolution order: flag > `PERF_PASSWORD` env. Commands needing auth error out with guidance when it is missing. Never interpolate creds into shell strings (we spawn with arg arrays only). +3. **No absolute paths** in code. Repo root discovered by walking up from cwd to the first `pnpm-workspace.yaml` (`findRepoRoot()`); server command defaults to `['node', '/packages/cli/dist/index.js']`, overridable via `--server-cmd`. +4. All outputs under `--out-dir` (default `./perf-out`), created with mkdir -p. File naming: `metrics-