(DO NOT MERGE: superseded by stacked PRs) (WIP) proof of concept: feat(server): heap-safe instance cache + opt-in blueprint pooling (one instance per schema-shape) by yyyyaaa · Pull Request #1325 · constructive-io/constructive

yyyyaaa · 2026-07-02T10:43:03Z

Summary

Makes the multi-tenant PostGraphile v5 server survive and scale on small heaps. Two layers:

Availability hardening (always on): heap-aware instance cache with safe eviction — the server degrades gracefully under tenant churn instead of OOMing.
Blueprint pooling (opt-in, GRAPHILE_BLUEPRINT_POOLING=1): one shared PostGraphile instance per schema-shape, routed per request via search_path — collapsing N same-shape tenants from N×~0.5GB to one instance.

Also fixes three latent multi-tenant bugs found during the audit (one live RLS bypass).

Problem

Each query-serving PostGraphile instance retains ~0.5GB of heap (measured; ~51% strings, ~30% plan closures). The cache was keyed per svc_key (per tenant×API) with max: 50, TTL 1 year — a steady state of ~24GB for a fleet that runs in 2GB containers. Memory grew linearly with tenants and the process OOM'd long before "thousands of tenants."

What's included

1. Cache + eviction hardening (`graphile-cache`, `graphql/server`)

Heap-aware default cap: clamp(⌊heap×0.5 / 512MB⌋, 3, 50) instead of a fixed 50 (GRAPHILE_CACHE_MAX, GRAPHILE_CACHE_INSTANCE_HEAP_BYTES override/tune); resolved cap logged at startup.
Entry-identity disposal guard (fixes a same-key disposal race), pool-coupling by entry.dbname (the old substring match never fired).
Eviction drain: in-flight requests are refcounted (invokeEntryHandler); disposeEntry waits for them (bounded by GRAPHILE_CACHE_DRAIN_TIMEOUT_MS, default 30s) before pgl.release() — eviction can no longer tear a schema down mid-request.
Build admission control: cross-key builds serialize through a semaphore (GRAPHILE_BUILD_CONCURRENCY, default 1) and evict the LRU instance before building, so the build's transient peak lands on freed headroom.
Prod idle TTL 1 year → 6h; GraphiQL (ruru) off outside development (GRAPHILE_GRAPHIQL=true to force).

2. Multi-tenant bug fixes

graphile-i18n: live RLS bypass — the localeStrings query ran withPgClient(null, …) (no role, no claims). Now threads the request's pgSettings. Also resets the type cache per build (module-singleton leak).
meta-schema: cachedTablesMeta was a module global — concurrent builds could serve each other's _meta. Now keyed per build via WeakMap (build objects are frozen; the flat global remains only for single-build codegen consumers, documented).
graphile-llm agent-discovery: config cache was keyed by dbname with a LIMIT 1, no tenant filter — cross-tenant bleed in shared-DB topologies. Now filtered and keyed by database_id.
graphile-presigned-url: storage-module resolution matched build-time physical schema names; now matches logical names (hash prefix stripped) so it works under pooling and across re-hashed schemas.

3. Blueprint pooling (opt-in)

Key: bp:sha256({sorted logical schemas, shape fingerprint, database settings flags, api name, mode, dbname}). The shape fingerprint hashes the catalog's [logical schema, relname] pairs, so tenants that drifted (e.g. a half-provisioned tenant) automatically get their own instance. dbname is included so same-shape tenants in different physical databases never share a pool.
Mechanism: shared instances build with the stock gather: { pgIdentifiers: 'unqualified' } (search_path-relative SQL — GraphQL SDL is byte-identical to qualified builds, sha256-verified) plus schema: { constructiveUnqualified: true } so Constructive plugins (search chunk refs + BM25 index name, llm RAG chunk query, i18n localeStrings) emit search_path-relative SQL for tenant data. Control-plane (metaschema_*, services_public) references stay fully qualified by design.
Routing: grafast.context reads roles from req.api (de-closured in all modes) and, on pooled instances only, sets pgSettings.search_path = requesting tenant's physical schemas + public last (shared domains/extensions — SECURITY DEFINER functions like sign_in depend on it).
Safety fallbacks to today's per-tenant instances: realtime-enabled APIs; empty schema lists; unqualified relation-name collisions within the schema set (e.g. the intentional identity_providers table/view shadow — detected by catalog probe, logged, per-tenant instance used); failed probes (not memoized — re-probed next request).
Invalidation (v1): any tenant schema:update flushes all pooled instances + cached decisions (rebuilds are ~1–2s for tenant APIs); the manual /flush route does the same.
Flag off is verified behavior-identical (existing middleware suites unchanged; plugin emissions byte-identical).

Verification (isolated rig: full constructive-db schema + 8 seeded marketplace tenants as hashed schemas, server at 2GB heap)

Gate: SDL qualified vs unqualified — byte-identical (same sha256). Zero-bleed spike at the engine level — one unqualified instance served other tenants purely via search_path, canaries clean.
Flag OFF regression: 40/40 requests, one instance per host, no pooling lines; separately 360/360 requests across two eviction soaks with 251+ pre-build evictions, 0 drain timeouts, 0 5xx.
Flag ON: 7 same-shape tenants → one shared instance (6 builds → 1 for the same hosts; 56 requests served by one build); the half-provisioned tenant automatically split to its own instance by fingerprint. Zero-bleed via HTTP-authenticated canary queries through the shared instance (10 interleaved rounds; cross-token control correctly unauthenticated). RSS for the same traffic: 1475MB (per-tenant) → 711MB (pooled). Collision fallback observed live (warn + per-tenant instance). Pooled login verified end-to-end (signIn → bearer → tenant data).
Adversarial review: 3 dimensions (correctness / tenant isolation / perf-ops), 19 findings adjudicated by refute-by-default verifiers → 9 confirmed → 6 fixed in this PR (search_path public drop, dbname in key, transient-probe memoization, /flush bp: gap, double catalog scan, _meta physical-name leak), 3 documented below.

Env vars

Var	Default	Purpose
`GRAPHILE_BLUEPRINT_POOLING`	off	Opt-in instance sharing per schema-shape
`GRAPHILE_CACHE_MAX`	heap-aware	Max cached instances
`GRAPHILE_CACHE_INSTANCE_HEAP_BYTES`	512MB	Per-instance estimate for the heap-aware cap
`GRAPHILE_CACHE_TTL_MS`	6h prod / 5m dev	Idle TTL
`GRAPHILE_CACHE_DRAIN_TIMEOUT_MS`	30s	Max wait for in-flight requests before release
`GRAPHILE_BUILD_CONCURRENCY`	1	Concurrent schema builds
`GRAPHILE_GRAPHIQL`	off in prod	Force-enable GraphiQL

Known limitations / follow-ups

Fingerprint granularity: relation names only (not columns/functions/enum labels). Same-relname column drift between pooled tenants is not detected by the key; exposure is bounded by flush-all-on-schema:update. Follow-up: extend the fingerprint to attributes/procs.
v1 flush semantics are coarse (any tenant migration flushes all pooled instances). Cheap rebuilds make this acceptable; follow-up: blueprint-membership tracking for targeted eviction.
Build semaphore has no wall-clock timeout — a pathologically slow build delays queued builds (requests on cached instances are unaffected). Follow-up: per-build timeout + 503.
Headroom accounting counts cached instances only, not still-draining ones (bounded by the 30s drain timeout).
Realtime APIs are excluded from pooling (per-instance LISTEN topology); pooling them needs the channel scheme rework.
The /flush route's missing auth is pre-existing (TODO in code).

Rollout

Ship with flag off (behavior-identical; hardening + bug fixes active).
Staging: GRAPHILE_BLUEPRINT_POOLING=1, watch [pooling] logs (attach vs build), instance counts, RSS.
Production canary on public API pods; auth/admin endpoints pool automatically where collision-free.

🤖 Generated with Claude Code

https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

…of 50 Root cause of the schema-builder (public cnc server) heap OOM: each PostGraphile v5 instance that has served a GraphQL request retains ~0.5 GB of heap (fully-materialised schema + grafast plan machinery; a build-only instance is far smaller). graphileCache capped entries at a fixed 50, so the steady-state resident set was ~50 x 0.5 GB ~= 24 GB -- far beyond the heap -- and the process OOM'd as distinct app hosts filled the cache over days. Eviction was empirically confirmed to free instances correctly; the count cap was simply far too large for the per-instance footprint. - getCacheConfig: heap-aware default for GRAPHILE_CACHE_MAX -- budget ~50% of the V8 heap limit at ~0.5 GB/instance, clamped to [3, 50], instead of a fixed 50. Override with GRAPHILE_CACHE_MAX; tune the per-instance estimate with GRAPHILE_CACHE_INSTANCE_HEAP_BYTES. The resolved cap is logged at startup. - disposeEntry: guard double-disposal by ENTRY IDENTITY (WeakSet) instead of by cache key. The key-scoped guard skipped pgl.release() for a rebuilt entry that shared a key with an entry still mid-release (same-key disposal race), proven via a repro harness (1/12 -> 12/12 disposals run). Also close the http.Server unconditionally -- it is never .listen()ed, so the old `.listening` guard was dead -- and drop the now-needless key bookkeeping. - pgCache cleanup callback: match entries by entry.dbname === poolKey (pools are keyed by database name) instead of `cacheKey.includes(poolKey)`; cacheKey is the request host and never contained the db name, so that safety valve was dead. dbname is threaded onto the entry via createGraphileInstance. - Add regression tests for the disposal guard and the heap-aware cap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H3mDDgX8z6dE7kyaMERhin

… GraphiQL gating - disposeEntry drains in-flight requests (refcounted via invokeEntryHandler, bounded by GRAPHILE_CACHE_DRAIN_TIMEOUT_MS, default 30s) before pgl.release(), so eviction can no longer tear down a schema mid-request. - All handler invocations in the graphile middleware go through invokeEntryHandler; disposing entries are treated as cache misses. - Global BuildSemaphore (GRAPHILE_BUILD_CONCURRENCY, default 1) serializes cross-key schema builds; ensureCacheHeadroom evicts the LRU instance BEFORE each build so the build's transient peak lands on freed headroom. - Prod idle TTL drops from 1 year to 6h (GRAPHILE_CACHE_TTL_MS still overrides). - GraphiQL (ruru) only in development or with GRAPHILE_GRAPHIQL=true. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

…LS), reset type cache per build withPgClient(null, ...) executed the localeStrings query with no role and no jwt.claims — bypassing RLS on every translation read. Thread pgSettings from the grafast context into the runtime query (same pattern as graphile-llm's rag-plugin). Also reset localeTypeCache in init alongside i18nRegistry so the module-singleton I18nPlugin export cannot leak GraphQLObjectTypes across schema rebuilds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

…not module global Concurrent PostGraphile builds in one process interleave init/fields hooks, so the module-global cachedTablesMeta could bake build B's tables into build A's _meta resolver. Store per-build via WeakMap (build objects are frozen by graphile-build, so no own-property). Flat global retained solely for single-build codegen consumers (graphile-schema buildIntrospectionJSON, codegen DatabaseSchemaSource), documented as such. Also export invokeEntryHandler/ensureCacheHeadroom from graphile-cache barrel. Verified: graphile-settings 158/158 tests, 3 snapshots identical, against live PG on :5433. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

… per schema-shape GRAPHILE_BLUEPRINT_POOLING=1 keys the instance cache by a blueprint hash (sorted logical schema names + shape fingerprint over the catalog's [schema,table] pairs + database settings flags) instead of per-tenant svc_key, builds shared instances with stock gather.pgIdentifiers='unqualified', and routes each request via pgSettings search_path (requesting tenant's physical schemas, double-quoted). Safety fallbacks to today's per-tenant instances: realtime-enabled APIs, empty schema lists, unqualified relation-name collisions within the schema set (e.g. identity_providers table/view shadow), or failed catalog probes. Decisions memoized per svc_key; schema:update flushes all pooled instances + decisions (v1 semantics). Plugins honor schema.constructiveUnqualified for tenant-data SQL (search chunk refs + BM25 index name, llm RAG chunk query, i18n localeStrings) while control-plane metaschema references stay fully qualified. presigned-url resolves storage modules by logical schema name; llm agent-discovery is tenant-filtered and keyed by database_id (fixes a LIMIT 1 cross-tenant bleed). grafast.context now reads role/anonRole from req.api in all modes (de-closured). Flag off = behavior-identical (verified: 61 pre-existing middleware tests unchanged; plugin suites byte-identical emissions). Gate evidence: SDL qualified-vs-unqualified byte-identical (sha256 match); zero-bleed proven on live hashed-schema tenants via per-request search_path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

…ixes for pooling - tenantSearchPath: keep shared 'public' LAST on the pooled search_path. Replacing the path with only tenant schemas broke SECURITY DEFINER auth functions without their own SET search_path (sign_in's ::email cast → 'type email does not exist', HTTP 500 on every pooled login). Verified fixed live: pooled signIn returns a token; authenticated data reads flow through the shared instance. - computeBlueprintKey now includes dbname: same-shape tenants in DIFFERENT physical databases must never share an instance (its pool targets one DB). - Transient catalog-probe failures are no longer memoized as permanent per-tenant fallbacks — next request re-probes. - Manual /flush route now also clears bp: entries + pooling decisions. - Single catalog scan feeds both shape fingerprint and collision check (was two identical pg_class scans per decision). - _meta reports LOGICAL schema names on pooled instances (stops leaking the representative tenant's hashed schema identifier to other tenants). W3 rig evidence: 7 same-shape tenants → 1 shared instance (6 builds → 1), tenant2 auto-split by shape fingerprint, zero-bleed via HTTP-authenticated canaries (10 interleaved rounds + cross-token control), RSS 1475MB → 711MB, collision fallback fires with warn + per-tenant instance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

…e-validation tooling - cacheCounters (evictions by reason, disposals, drain timeouts) exported from graphile-cache; build/pooling counters + build-queue depth from the graphile middleware. - GRAPHILE_DEBUG_METRICS=1 starts a 10s JSON sampler (rss/heap/GC pauses/cache stats/counters) to GRAPHILE_DEBUG_METRICS_FILE — zero overhead when off; groundwork for a prod /metrics endpoint. - scripts/scale-validate/: fleet manifest, token-bucket workload harness (zipf/ uniform/burst, authenticated sessions, read/write/meta mix, cross-tenant bleed sentinel that hard-fails), churn driver (schema:update storms), RSS-slope collector, canary seeding, SQL tenant-factory (pooling-equivalent tenants at ~0.8/min, bypasses the GraphQL seeder's undici cap), drift tooling (table- drift → distinct blueprints; column-drift → documents fingerprint limits). - include presigned-url resolver unit tests from the pooling work (11 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

- measure-instance-heap.mjs: per-class instance heap probes (fresh cap=1 server per run, baseline -> cold build -> settle -> delta; dual-stack port probe since the server binds ::1 on macOS) - subset-fleet.mjs: diversity/tenant-count fleet subsets from fleet.json + drifted.json; group 0 restricted to the fingerprint-validated identical pool (factory* + marketplace_db_tenant1) - v2-ramps.mjs: plan-driven ramp executor (fresh server per step, harness child, pg_stat_activity sampling, metrics harvest, crash-tolerant, cold-burst mode) - drop-tenants.mjs: tenant teardown (control-plane CASCADE delete + physical schema drops + catalog VACUUM) after the PG introspection catalog wall: a ~100-tenant catalog (135k pg_class) OOM-killed the introspection backend even running alone; fleet trimmed to 40 factory tenants - tenant-factory.mjs: survive PG crash recovery (pool error handler + connection-class retry with long backoff) - plans/: V2 limits-discovery plans at 2048/896/3584MB heap Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

… + tuned plan - drift-tenants.mjs: resolve pg via _lib resolvePg (direct require('pg') fails — pg is not a root dependency) - v2-ramps.mjs: plan steps can inject server env (e.g. GRAPHILE_CACHE_INSTANCE_HEAP_BYTES for the tuned headroom rerun) - plans/v2-2048-tuned.json: mitigation rerun with instance-heap estimate set to the measured ~1.35GB (as-shipped 512MB estimate under-evicts at the 48-DB catalog where every instance retains ~1.3GB) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

… plans The as-shipped div-k1 run banked the expected result: auth blueprint + api blueprint (~1.3GB each at the 48-DB catalog) SIGABRT a 2GB heap. Remaining V2 runs use the measured instance-heap estimate; new 1536/1152/ 896MB single-step plans bracket the minimum viable heap per instance. The pg_stat_activity sampler now survives PG crash recovery (error handler + reconnect) instead of taking down the executor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

…BRT) V2 validation caught this live: on a 48-database catalog every instance retains ~1.3GB (introspection graph scales ~21KB per pg_class row), so computeHeapAwareMax's budget correctly computes 0-1 slots on a 2GB heap — but the floor of 3 overrode it, ensureCacheHeadroom (count-based) never evicted, and admitting a second build aborted the process with a V8 heap-limit SIGABRT before any eviction could run. Floor of 1 keeps the cache functional (something must be admitted) while honouring the per-instance estimate; eviction then makes the displaced graph unreachable so build-time GC reclaims it. New regression test: an estimate larger than half the heap must yield max=1, never the old floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

… the server V2 validation caught this live twice: a PostgreSQL crash-recovery (its introspection backend was OOM-killed) terminated every pooled connection, and one orphaned pg Client surfaced an unhandled 'error' event that took the whole multi-tenant node process down (exit 1). A ~15s PG recovery became a full outage for every tenant on the box. installConnectionErrorGuard() absorbs ONLY connection-class failures (57P01/08006/ECONNRESET/'connection terminated'/... — logged + counted, pools reconnect on next checkout); everything else stays fatal via process.exit(1). Counters surface in the metrics sampler as counters.connGuard. Opt out with GRAPHILE_CONNECTION_GUARD=0. Rig note: introspection backends retain multi-GB relcache for the whole catalog and accumulate across pool connections; idle_session_timeout on the database recycles them (prod guidance: pgbouncer server_lifetime). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

… follow-up plans V2-RESULTS.md consolidates the measured constants (21KB/pg_class-row instance heap, 37KB/row PG introspection spike, 1536MB min viable heap, zero marginal heap per tenant within a blueprint) and the regime matrix (healthy at R>=active blueprints; thrash-not-crash beyond; as-shipped SIGABRT control). soak-ops.mjs cycles provision+teardown for the 24h soak without net catalog growth (instance heap scales with catalog). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

…nt rps ceiling r2 probe: 3584MB heap with 896MB estimate holds api+auth resident (builds=2, evictions=0, heapMax 2750MB, p99 34ms) — capacity 2 works; only the 0.5-fraction budget formula prevents it by default. rps probe: 32 pooled tenants on one 2GB node at 188 achieved rps, errRate 0, p99 21ms, PG pool still at 5 connections — no knee found through 200rps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

…eaping) A persistent client sleeping 3600s between NOTIFYs is exactly what idle_session_timeout reaps; the socket error then killed the driver mid-soak. Fresh connection per NOTIFY tick instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

…ics + build wait timeout Hardening derived from V2 limits discovery: - computeCapacityFromBudget: replaces the blunt heap*0.5 budget with the two-constraint model validated live (residency bound + rebuild-with- evict-before-build bound): a 3584MB heap now correctly admits TWO ~1.35GB instances (was 1); a 2GB heap stays at 1. New tunables GRAPHILE_CACHE_BASE_RESERVE_BYTES / GRAPHILE_CACHE_BUILD_RESERVE_BYTES. - Memory governor: at >=85% heap (GRAPHILE_MEMORY_GOVERNOR_ELEVATED) a 10s timer evicts LRU idle instances; at >=92% (…_CRITICAL) the dispatcher refuses NEW builds with 503 SERVICE_OVERLOADED + Retry-After (resident instances keep serving) — a build transient at critical pressure is what converts degradation into a process abort. Counters: evictions.governor, buildRefusals. Disable: GRAPHILE_MEMORY_GOVERNOR=0. - Build wait timeout: requests stop waiting on a build after GRAPHILE_BUILD_TIMEOUT_MS (default 180s) with 503 BUILD_TIMEOUT; the build itself completes in the background and fills the cache (cache population + in-flight cleanup moved to build-settle, so a timed-out request no longer breaks coalescing for followers). - GET /metrics (GRAPHILE_METRICS_ENDPOINT=1, loopback-guarded): same sample shape as the JSON-line sampler + live memory pressure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

…rd guidance Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

partman.part_config rows are keyed by table NAME (not FK-cascaded), and schema hashes are deterministic per dbname — a leftover row blocks re-provisioning any reused tenant name with unique_violation (hit live by the soak's provision/drop cycler). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

… ledger) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

…e blueprint flush The 24h soak's first capacity-2 attempt died at hour 2, seconds after a tenant provision/teardown cycle: flushService cleared ALL bp: instances on any schema:update, and the immediate rebuild's ~1.4GB transient stacked on ~2.7GB of evicted-but-still-draining instances (invisible to ensureCacheHeadroom — they are no longer cache entries) => V8 SIGABRT. Two fixes: - graphile-cache: drainingCount tracking + waitForDrainSettle(); the dispatcher now waits (bounded, pressure-aware) for evicted instances to actually release memory before starting a build. - flushService: PoolDecision records its owning databaseId at the single memoization point; schema:update now invalidates only that database's decisions and rebuilds only the blueprint it was attached to. Tenant PROVISIONS become a no-op for resident blueprints (instances are shape-generic — a same-shape tenant attaches with zero rebuilds); a real schema change rebuilds exactly one blueprint. The explicit /flush admin route keeps clear-all semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

Ports scripts/scale-validate tooling into @constructive-io/perf-harness (bins: perf-harness, cperf): fleet discover/provision/drift/canary/ subset/teardown, load harness/churn, measure collector/pg/instance-heap, run ramp/soak/soak-ops, report summarize/merge/predict, and a regression suite with baselines from the 2026-07 blueprint-pooling validation program. Adds a hub-port guardrail (5432/3000-3002/9000 refused without --allow-hub), detached soak orchestration with ordered stop, and credential handling via PERF_PASSWORD (no hardcoded secrets). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014vjKpGWLc3Rw4jfNztY9Ni

Appends the 5h soak attempt-3 verdict (392,794 requests, 0 bleed across 235 sentinel checks, errRate 0.094% confined to one recovery blip, p50/p95/p99 13/21/55ms, like-for-like heap flat at ~2.78GB +/-45MB, all governor/guard/drain counters zero, 3 provision/drop cycles + 5 relogin batches survived), finding 11 (fleet-wide flush + drain-race -> surgical per-database flush + drain-aware admission), the perf-harness tooling note, and fixes the report merge/predict flag docs in the package README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014vjKpGWLc3Rw4jfNztY9Ni

pnpm install regenerated the lockfile for the new packages/perf-harness importer; pnpm 10.30.2 also normalizes/dedupes existing entries on write (same lockfileVersion 9.0, all importers verified present, validated with pnpm install --frozen-lockfile). Regenerate with the team's pinned pnpm if a minimal diff vs main is preferred. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014vjKpGWLc3Rw4jfNztY9Ni

First live dogfood run found fleet discover --out failing with ENOENT for a nested path. Adds ensureParentDir to core/proc and applies it at every user-path write site (discover, harness, collector, instanceHeap, suite x4, merge, ramp). Live acceptance so far: fleet teardown (soak3 + partman rows cleaned), fleet discover (48 tenants), regression quick suite PASS 4/4 (errRate 0, p99 34ms, zero-marginal delta 3.7MB, bleed 0) against baseline catalog61k-2026-07. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014vjKpGWLc3Rw4jfNztY9Ni

pyramation · 2026-07-03T09:25:15Z

+            : `"${schemaName}"."${baseTable}"`;
+          const translationTableRef = constructiveUnqualified
+            ? `"${translationTable}"`
+            : `"${schemaName}"."${translationTable}"`;


let's aim to use this where we can instead of just quoting stuff w/o checking:

https://www.npmjs.com/package/@pgsql/quotes

QuoteUtils.quoteQualifiedIdentifier('public', 'my_table');

pyramation · 2026-07-03T09:25:57Z

    acm.task_table_name
  FROM metaschema_modules_public.agent_chat_module acm
  JOIN metaschema_public.schema s ON s.id = acm.schema_id
+  WHERE s.database_id = $1


this is great, not sure if we can separate a few smaller PRs, would be great :)

if not, I also get it, we can take a look together then.

pyramation · 2026-07-03T09:28:27Z

+      // (schema.constructiveUnqualified), emit search_path-relative references for
+      // tenant-data tables/indexes so the per-request search_path resolves the
+      // tenant schema. Default (flag absent): fully schema-qualified, byte-identical.
+      const constructiveUnqualified = !!((build as any)?.options?.constructiveUnqualified);


wait, this means using unqualified schemas?

…ng axes Adds a deep suite tier with four scenarios covering the previously untested memory axes: multi-API residency (R=4 across api/auth/admin/ usage surfaces), settings-variant pool splits (database_settings flags are part of the blueprint key), realtime dedicated-instance cost (advisory), and partman partition-creep projection (advisory). All rig mutations are paired with finally-teardown. Documents the corrected scaling model (hot API surfaces x settings variants, priced by module-driven catalog size; blueprints immaterial) in SIZING.md and the package README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014vjKpGWLc3Rw4jfNztY9Ni

…nant identifier rewrite at the pool seam Replaces the banned mechanism (gather.pgIdentifiers 'unqualified' + per-request search_path) with fully-qualified SQL: the pooled instance is built against a canonical tenant, and a rewriting pool wrapper swaps canonical→tenant schema identifiers per request. - rewrite-pool.ts: single-pass SQL lexer (strings/dollar-quotes/comments-aware) rewriting only complete double-quoted identifiers; tenant identity via constructive.pool_schemas GUC inside the adaptor settings query; fail-closed when a rewritable query has no tenant map; per-tenant prepared-statement namespacing with wrapper-owned LRU (deallocate + node-postgres parsedStatements cleanup on eviction); memoized rewrites; 35 unit tests - graphile.ts: servicePool = createRewritingPool(...) when pooling; both pgSettings blocks stamp constructive.pool_schemas; pgIdentifiers override deleted; flag renamed to constructivePooled - blueprint.ts / pooling-decision.ts: search_path helpers and the relation-collision opt-out removed (qualified SQL cannot be ambiguous) - plugins (i18n, llm/rag, search adapters, meta-schema): unqualified branches reverted to always-qualified SQL - metrics-sampler: rewritePool counters exposed Validated live: SDL parity pooled-vs-dedicated byte-identical (MD5), zero search_path/canonical-schema leakage across tenants, cperf quick 4/4 PASS (errRate 0, p99 21ms, zero-marginal 1.9MB, bleed 0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…kup, visible teardown leftovers, fail-closed residency check - partition-creep: escape LIKE metacharacters in schema names before the partman parent_table LIKE ANY lookup (an unescaped underscore could match a different tenant's parents) - settings-variant-split + realtime-dedicated-cost: teardown leftovers (rowsRemaining > 0) now surface as suite warnings instead of silent data - multi-api-residency: fail closed — an abort before the gradeable check lands now pushes a failing CheckResult so the suite cannot PASS with the residency invariant untested 12 new unit tests (199 total). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… (GRAPHILE_INTROSPECTION_FILTER) PostGraphile v5 introspection is unscoped: every built instance ingests the entire pg_class catalog (~1.4GB retained heap at a 61k-row multi-tenant catalog, ~22KB/row, independent of how few schemas the instance serves). This intercepts the single static introspection statement at the pool wrappers we own and rewrites its four namespace gates (classes/constraints/procs/types) to a whitelist: served schemas ∪ public ∪ a discovered closure over cross-schema references (FK targets, attribute/domain/array-element types, proc return/argument types), keeping the stock pg_catalog types branch intact. - introspection-filter.ts: signature detection, 4-site gate swap (pinned by test against the installed pg-introspection@1.0.1 text), iterative closure discovery (schema-qualified pg_catalog SQL, 5-round cap), per-checkout memoized interceptor, thin filter-only pool wrapper for dedicated instances, counters - rewrite-pool.ts: introspectionFilter option — swap runs before settings parsing and identifier rewriting; callback-form queries pass through counted - graphile.ts: flag-gated wiring for pooled + dedicated service pools; flag-off behavior byte-identical - metrics-sampler: introspectionFilter counters sampled - Fail-open on unrecognized shapes (gate mismatch → stock text); fail-closed only on discovery errors (build fails and is retried) 15 new unit tests (189 total in package); pg-introspection added as devDependency for the pinning test only (runtime never imports it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…nt text The rewriting pool classified the pgSettings query by values-shape alone (values[0] containing the pool_schemas GUC). A data query whose first bound parameter merely contains that substring was misclassified: forwarded unrewritten (canonical SQL on the tenant connection) and able to remap or null the checkout's tenant map for sibling queries in the same transaction. Gate classification on the adaptor's fixed statement text (byte-matching @dataplan/pg's settings statement) in addition to the values shape, and guard applySettings itself against non-settings text. Adds 4 tests: misclassification, self-DoS map null, cross-tenant remap, pre-settings fail-closed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…acts Drop the hardcoded seeder password default in harness.mjs — resolution is now flag > PERF_PASSWORD env with a fail-fast when --auth has no credential (same contract as packages/perf-harness resolveCreds). Ignore locally generated rig artifacts (fleet manifests, drifted.json, perf-out, metrics JSONL, scale-spike scratch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…e timer; reachable heap-pressure governor Root cause (heap-snapshot proven): the build-wait Promise.race armed a 180s setTimeout that was never cleared when the build won. The armed Timeout pins the built instance through the race reaction chain (Timeout -> _onTimeout closure -> race result promise -> grafast context -> Grafserv -> GraphQLSchema), one timer per waiting request. Under eviction churn (blueprints > capacity, sustained traffic) every evicted ~15MB instance stayed live for 180s: ~2 builds/s x 180s x 15MB OOM-killed a 2GB heap in 65s (and 512MB in 160s). Extract raceBuildAgainstTimeout() which disarms the timer on win, timeout, and rejection. Companion governor fixes, since the OOM exposed them: - getMemoryPressure ratio was computed against heap_size_limit, which includes young-gen/reserved space old space can never use — at a 512MB heap the process aborts at ratio 0.69 and at 2048MB at 0.90, so the 0.85/0.92 watermarks were unreachable at every size. Ratio is now used / (used + total_available_size). - Re-check pressure inside the build critical section (post-semaphore, at the point of allocation) — the request-time gate is seconds stale under a build storm. New BuildRefusedError maps to 503 SERVICE_OVERLOADED + Retry-After for the requester and every coalesced waiter. - Pause once between build starts under elevated pressure (GRAPHILE_BUILD_PRESSURE_SPACING_MS, default 250ms) so mark-compact gets a window before the next transient commits. Validated live on the 48-tenant rig (fleet-k3, 12 hosts, 25rps round-robin): - forced GRAPHILE_CACHE_MAX=1 at 2048MB: was OOM at 65s; now survives 240s+ at 23 builds/s with bounded sawtooth (peaks ~1.4GB, troughs ~190MB), disposals tracking builds 3461/3462, clean post-stop recovery. - capacity-3-of-4-blueprints rotation at 2048MB (the realistic over-capacity mode): 1444/1444 HTTP 200, heap 151-336MB, zero errors. - warm path unchanged: cold build 443ms, warm 1-2ms. - sub-floor heaps (<=768MB) still die under 100%-miss churn — all-miss rebuild amplitude cannot fit; documented minimum heap remains 1024MB. Tests: +5 (build-timeout suite incl. timer-count assertions); graphql/server 201/201; graphile-cache 24/24. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ts, churn fix, compliance audit, ship verdict Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…m 503 for re-coalesced waiters, adaptor-text pin, exit-code hygiene Final pre-push sweep findings (adversarially verified): - Re-coalesce path (Phase C) now maps BuildRefusedError to 503 SERVICE_OVERLOADED + Retry-After like the owner path and request-time gate; previously a refusal surfacing on the unguarded await escaped to the outer catch as a generic 500 with no Retry-After. Other rejections fall through to retry, matching Phase B. Adds a setInFlightForTest seam + a regression test proven against the pre-fix behavior (500 -> 503). - Pin SETTINGS_QUERY_TEXT against the installed @dataplan/pg adaptor source (new devDependency @dataplan/pg@1.0.3, mirroring the pg-introspection pinning test): a dependency bump that changes the settings statement now fails loudly at test time instead of silently failing the pooling path closed at runtime. - harness.mjs missing-credential fail-fast now exits 3: exit 2 stays reserved exclusively for the bleed-sentinel (cross-tenant isolation) signal. - shouldRefuseBuild log no longer attributes the pressure ratio to the heap limit (ratio is measured against exhaustible headroom since 46636d4). - Remove dead export buildKeepClosureQuery (zero importers). Release note (from the sweep's stock-path review): the governor ratio fix in 46636d4 also applies to non-pooling deployments — previously its watermarks were unreachable, so stock servers never saw 503 SERVICE_OVERLOADED; they now can, but only at genuine near-OOM pressure (no idle false positives; verified). graphql/server: 200/200 (18 suites; count drop vs earlier reports is the removal of an untracked duplicate timer test that had inflated totals). graphile-cache: 24/24. Full monorepo build: 107 projects green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

yyyyaaa and others added 6 commits July 2, 2026 12:43

yyyyaaa changed the title ~~feat(server): heap-safe instance cache + opt-in blueprint pooling (one instance per schema-shape)~~ (WIP) proof of concept: feat(server): heap-safe instance cache + opt-in blueprint pooling (one instance per schema-shape) Jul 2, 2026

yyyyaaa and others added 20 commits July 2, 2026 19:53

chore: ignore scale-validate output artifacts

ea1275b

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

chore(scale-validate): results summarizer for v2-ramps JSONL

bf86ef3

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

docs(scale-validate): SIZING.md — catalog-driven capacity model + sha…

8eec3cd

…rd guidance Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

docs(scale-validate): consolidated validation report (V1-V4, findings…

aeb8c35

… ledger) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb

pyramation reviewed Jul 3, 2026

View reviewed changes

yyyyaaa and others added 9 commits July 3, 2026 16:30

docs(scale-validate): capacity-10x program results — filtered constan…

f95d59f

…ts, churn fix, compliance audit, ship verdict Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

(DO NOT MERGE: superseded by stacked PRs) (WIP) proof of concept: feat(server): heap-safe instance cache + opt-in blueprint pooling (one instance per schema-shape)#1325

(DO NOT MERGE: superseded by stacked PRs) (WIP) proof of concept: feat(server): heap-safe instance cache + opt-in blueprint pooling (one instance per schema-shape)#1325
yyyyaaa wants to merge 35 commits into
mainfrom
feat/scale-phase0

yyyyaaa commented Jul 2, 2026

Uh oh!

pyramation Jul 3, 2026

Uh oh!

pyramation Jul 3, 2026

Uh oh!

pyramation Jul 3, 2026

Uh oh!

pyramation Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

yyyyaaa commented Jul 2, 2026

Summary

Problem

What's included

1. Cache + eviction hardening (graphile-cache, graphql/server)

2. Multi-tenant bug fixes

3. Blueprint pooling (opt-in)

Verification (isolated rig: full constructive-db schema + 8 seeded marketplace tenants as hashed schemas, server at 2GB heap)

Env vars

Known limitations / follow-ups

Rollout

Uh oh!

pyramation Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

pyramation Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

pyramation Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

pyramation Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

1. Cache + eviction hardening (`graphile-cache`, `graphql/server`)