Skip to content

Size the host memory-baseline hard gate to each module's observed swing#5598

Open
habdelra wants to merge 2 commits into
mainfrom
fix-host-memory-baseline-variance
Open

Size the host memory-baseline hard gate to each module's observed swing#5598
habdelra wants to merge 2 commits into
mainfrom
fix-host-memory-baseline-variance

Conversation

@habdelra

@habdelra habdelra commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

The "Host Memory Baseline" check compares each test module's post-GC heap boundary delta against a rolling window of recent main runs and hard-fails on a large regression. That metric is dominated by non-deterministic settle-GC drain timing at the module boundary, so heavy modules routinely swing run-to-run by well over 100MB without leaking anything — a negative reading just means the previous module's garbage got reclaimed inside this module's window.

The hard gate already anchors to the recent ceiling (max sample) rather than the mean, precisely so a high-variance module can't be blocked by a value it has already exhibited. But when a module's entire recent window is negative, both its mean baseline and its ceiling floor to zero, and the gate collapses to "any single reading over +50MB fails." A module that has been sitting at samples like [-73, -74, -8, -19, -120] then gets blocked the one run it lands at +54:

Module Baseline (mean of last 5) Current Change Recent samples
Integration | ai-assistant-panel | general 0.0 MB 54.3 MB +54.3 MB [-73.0, -74.1, -7.6, -19.4, -120.1]

The baseline showing 0.0 and the change being the raw measured value is the tell: nothing about the module's actual behaviour is informing the threshold. That window spans 112MB peak-to-peak, so a one-off +54 is squarely inside the module's own noise.

This folds the recent window's peak-to-peak spread into the hard threshold, alongside the existing absolute and relative terms. A run now has to clear the ceiling by more than the module has already shown it can move on its own before it blocks. Behaviour at the edges:

  • Tight-window modules (spread below the +50MB absolute threshold) are unchanged.
  • A genuine regression still fails — the delta has to clear the recent ceiling (the module's highest recent sample) by more than the module's own observed swing, so a step well past a demonstrated range is still caught.
  • The soft, mean-anchored warning is untouched, so a module trending upward still surfaces early; it just no longer blocks on within-noise excursions.

This is a strict relaxation of the current gate. The measured delta and the ceiling it's compared against are unchanged, and the threshold only ever grows (the added spread term is non-negative), so hardThreshold can only increase. No module that clears the hard gate today can begin to fail under this change — it lands with no re-baselining and introduces no new reds.

That monotonicity is also why the spread is layered on top of the ceiling anchor rather than replacing it with a single mean anchor. Anchoring the excursion at the mean would tighten the wide-swing modules, but it would begin blocking moderate-variance modules that pass today — e.g. a [40, 60, 40, 60, 50] window would hard-fail at +100MB, where both the current gate and this change pass. The cost of keeping the ceiling anchor is that a very wide window tolerates a correspondingly large excursion before it hard-blocks; that excursion still prints as a non-blocking warning, and a sustained regression re-crosses the gate once it folds into the rolling baseline.

Verified against reconstructed data from the failing run and synthetic leak/low-variance cases: the false failure drops to a non-blocking warning, while both a low-variance step-change and a noisy value that clears its own swing still hard-fail.

The per-module post-GC boundary delta is dominated by non-deterministic
settle-GC drain timing, so heavy modules swing run-to-run by >100MB with no
leak. When a module's entire recent window is negative, its mean and ceiling
both floor to zero and the hard gate collapses to "any single reading over
+50MB fails" — so a module that routinely swings from -120MB to -8MB gets
blocked the one run it happens to land at +54MB.

Fold the recent window's peak-to-peak spread into the hard threshold: a run
must clear the ceiling by more than the module has already shown it can move on
its own before it blocks. Tight-window modules (spread below the absolute
threshold) are unaffected; a genuine step past the observed swing still fails.
The soft, mean-anchored warning is untouched, so upward trends still surface.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Preview deployments

Host Test Results

    1 files      1 suites   2h 57m 50s ⏱️
3 629 tests 3 614 ✅ 15 💤 0 ❌
3 648 runs  3 633 ✅ 15 💤 0 ❌

Results for commit 7d75ab6.

Realm Server Test Results

    1 files      1 suites   13m 57s ⏱️
1 934 tests 1 934 ✅ 0 💤 0 ❌
2 013 runs  2 013 ✅ 0 💤 0 ❌

Results for commit 7d75ab6.

…math

The file header and the failures-table heading still described the gate as a
flat ">2x baseline or +50MB" rule, which stopped being accurate once the hard
gate moved to a ceiling-relative measurement. State the actual condition — a
delta clears the recent ceiling by more than the larger of +50MB, 100% of the
baseline, or the module's own observed swing — and correct the inline note that
implied the absolute term blocks any positive reading rather than one over 50MB.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@habdelra
habdelra requested a review from a team July 24, 2026 00:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant