feat(graphile-cache,server): heap-safe instance cache, memory governor, build admission control (2/6)#1331
Open
yyyyaaa wants to merge 1 commit into
Open
Conversation
…l, memory governor, build admission control Each cached PostGraphile instance retains up to ~1.3GB on production-sized catalogs (61k pg_class rows), while the previous cache admitted a fixed 50 entries with a 1-year TTL and disposed entries that still had requests in flight. This PR makes residency heap-aware and makes every degradation mode a 503 + Retry-After instead of a process abort. Every mechanism below is traceable to a specific measured crash in the #1325 validation program (scripts/scale-validate/VALIDATION-REPORT.md there). graphile-cache: - Heap-aware capacity: R = clamp(min(⌊(H−B)/I⌋, ⌊(H−B−T)/I⌋+1), 1, 50) with B=256MB server baseline, T=768MB build transient, I=instance budget (GRAPHILE_CACHE_INSTANCE_HEAP_BYTES). Replaces the fixed max=50. - In-flight refcounting (invokeEntryHandler) + eviction DRAIN: disposal waits for the entry's live requests before teardown (bounded), instead of closing pools under them. - Memory governor: background eviction at 85% pressure, build refusal at 92%. Pressure = used/(used + total_available_size) — reachable, unlike a heap_size_limit denominator (measured death at ratio 0.69@512MB before the 92% trigger could ever fire). - Per-database entry indexing (dbname on entries) so pool disposal evicts exactly the entries it strands. graphql/server: - Process-wide build semaphore (GRAPHILE_BUILD_CONCURRENCY, default 1): only same-key builds were deduped before, so N tenants cold-starting stacked N transient build peaks. Evict-before-build (ensureCacheHeadroom) + drain-wait + a late pressure re-check with elevated-pressure spacing (GRAPHILE_BUILD_PRESSURE_SPACING_MS) bound the allocation rate. - Request build-wait timeout (GRAPHILE_BUILD_TIMEOUT_MS, default 180s) via raceBuildAgainstTimeout, which ALWAYS clears its timer: an armed race timer pins the built instance through the reaction chain for the full timeout — heap-snapshot-proven OOM under eviction churn (~2 builds/s × retained instance × 180s). .unref() does not help; only clearTimeout releases it. - Build settle owns cache population (single-flight map cleared when the BUILD settles, not when the first requester stops waiting), so timed-out waiters leave coalescing intact. - BuildRefusedError → uniform 503 SERVICE_OVERLOADED + Retry-After at all three surfaces (request gate, owner path, re-coalesced waiters); mid-disposal cache hits → 503 SERVICE_ROTATING instead of invoking a dying handler. - Connection-class error guard: a PG restart/crash must degrade requests, not kill the server process. - Observability: /metrics endpoint (GRAPHILE_METRICS_ENDPOINT=1, loopback gated), JSON-line metrics sampler (GRAPHILE_DEBUG_METRICS), GC pause tracking, build/cache counters. GraphiQL assets now dev-only (GRAPHILE_GRAPHIQL=true to force). Validated in #1325: 47 tenants on one 2048MB node, 30min authenticated zipf @50rps — 104,818/104,818 OK, p99 21ms, heap peak 195.7MB (9.6%), and a 240s+ eviction-churn storm at ~23 builds/s holding a bounded heap sawtooth. Extracted from #1325 (commits 9f78b69, 9ea25e3, 6745ea2, 620b011, 4659d52, b468b15, 46636d4, 2eeaefb, 3088062). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 2/6 of the #1325 split. Makes PostGraphile instance residency heap-aware and turns every memory degradation mode into a
503 + Retry-Afterinstead of a process abort. No pooling here — this hardens the existing per-service cache and is independently shippable; it's the "server stays alive" layer. Every mechanism traces to a specific measured crash in the validation program (scripts/scale-validate/VALIDATION-REPORT.md, lands with PR 6).Why: on a production-shaped catalog (61k
pg_classrows), one cached instance retains up to ~1.3GB. The previous cache admitted a fixed 50 entries with a 1-year TTL and disposed entries that still had requests in flight — on catalogs this size that configuration is the OOM, not a cache.graphile-cache
max=50:R = clamp(min(⌊(H−B)/I⌋, ⌊(H−B−T)/I⌋+1), 1, 50)with B=256MB server baseline, T=768MB build transient, I = per-instance budget (GRAPHILE_CACHE_INSTANCE_HEAP_BYTES).invokeEntryHandler): disposal waits (bounded) for the entry's live requests instead of closing pools under them.used / (used + total_available_size)— the earlierheap_size_limitdenominator was unreachable in practice (measured death at ratio 0.69 on a 512MB heap, long before a 0.92 trigger could fire).dbnameso pool disposal evicts exactly the entries it strands.graphql/server
GRAPHILE_BUILD_CONCURRENCY, default 1): the single-flight map only dedupes builds of the same key, so N cold-starting tenants used to stack N several-hundred-MB build transients. Plus evict-before-build, drain-wait, and a late pressure re-check with elevated-pressure spacing (GRAPHILE_BUILD_PRESSURE_SPACING_MS) so build starts can't outrun GC.GRAPHILE_BUILD_TIMEOUT_MS, default 180s) viaraceBuildAgainstTimeout, which always clears its timer. This is load-bearing: an armedPromise.racetimeout pins the freshly built instance through the timer's reaction chain for the full 180s. Under eviction churn that stacked ~2 builds/s × retained instance × 180s and OOM'd the process — root-caused by heap-snapshot retainer chains, and.unref()demonstrably does not help (it removes event-loop keepalive, not GC reachability).SERVICE_OVERLOADED(pressure refusal — at the request gate, the build owner path, and re-coalesced waiters alike),BUILD_TIMEOUT(wait expired; build continues in background and fills the cache for retries),SERVICE_ROTATING(mid-disposal hit). Never a generic 500 for a governed condition./metricsendpoint (GRAPHILE_METRICS_ENDPOINT=1, loopback-gated), JSON-line sampler (GRAPHILE_DEBUG_METRICS) with GC pause tracking and build/cache counters. GraphiQL (ruru) assets are now dev-only (GRAPHILE_GRAPHIQL=trueto force) — they were per-instance overhead and an unnecessary prod surface.Validation (from #1325)
Testing
graphile-cache24/24 (counters, disposal/cap, governor) ·graphql/server119/119 including newbuild-timeout(timer-count-zero assertions) andgraphile-recoalesce(503 contract, proven 500 pre-fix) suites, plus all pre-existing middleware suites.Stack
Stacked on #1330 (
feat/scale-s1-plugin-fixes); merge after it. Full stack map in #1330. Extracted from #1325 (kept open as the validated reference).🤖 Generated with Claude Code
https://claude.ai/code/session_0122xqM2VkNbuAmZshK1YNSb