Size the host memory-baseline hard gate to each module's observed swing#5598
Open
habdelra wants to merge 2 commits into
Open
Size the host memory-baseline hard gate to each module's observed swing#5598habdelra wants to merge 2 commits into
habdelra wants to merge 2 commits into
Conversation
The per-module post-GC boundary delta is dominated by non-deterministic settle-GC drain timing, so heavy modules swing run-to-run by >100MB with no leak. When a module's entire recent window is negative, its mean and ceiling both floor to zero and the hard gate collapses to "any single reading over +50MB fails" — so a module that routinely swings from -120MB to -8MB gets blocked the one run it happens to land at +54MB. Fold the recent window's peak-to-peak spread into the hard threshold: a run must clear the ceiling by more than the module has already shown it can move on its own before it blocks. Tight-window modules (spread below the absolute threshold) are unaffected; a genuine step past the observed swing still fails. The soft, mean-anchored warning is untouched, so upward trends still surface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
…math The file header and the failures-table heading still described the gate as a flat ">2x baseline or +50MB" rule, which stopped being accurate once the hard gate moved to a ceiling-relative measurement. State the actual condition — a delta clears the recent ceiling by more than the larger of +50MB, 100% of the baseline, or the module's own observed swing — and correct the inline note that implied the absolute term blocks any positive reading rather than one over 50MB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The "Host Memory Baseline" check compares each test module's post-GC heap boundary delta against a rolling window of recent main runs and hard-fails on a large regression. That metric is dominated by non-deterministic settle-GC drain timing at the module boundary, so heavy modules routinely swing run-to-run by well over 100MB without leaking anything — a negative reading just means the previous module's garbage got reclaimed inside this module's window.
The hard gate already anchors to the recent ceiling (max sample) rather than the mean, precisely so a high-variance module can't be blocked by a value it has already exhibited. But when a module's entire recent window is negative, both its mean baseline and its ceiling floor to zero, and the gate collapses to "any single reading over +50MB fails." A module that has been sitting at samples like
[-73, -74, -8, -19, -120]then gets blocked the one run it lands at+54:The baseline showing
0.0and the change being the raw measured value is the tell: nothing about the module's actual behaviour is informing the threshold. That window spans 112MB peak-to-peak, so a one-off +54 is squarely inside the module's own noise.This folds the recent window's peak-to-peak spread into the hard threshold, alongside the existing absolute and relative terms. A run now has to clear the ceiling by more than the module has already shown it can move on its own before it blocks. Behaviour at the edges:
This is a strict relaxation of the current gate. The measured delta and the ceiling it's compared against are unchanged, and the threshold only ever grows (the added spread term is non-negative), so
hardThresholdcan only increase. No module that clears the hard gate today can begin to fail under this change — it lands with no re-baselining and introduces no new reds.That monotonicity is also why the spread is layered on top of the ceiling anchor rather than replacing it with a single mean anchor. Anchoring the excursion at the mean would tighten the wide-swing modules, but it would begin blocking moderate-variance modules that pass today — e.g. a
[40, 60, 40, 60, 50]window would hard-fail at+100MB, where both the current gate and this change pass. The cost of keeping the ceiling anchor is that a very wide window tolerates a correspondingly large excursion before it hard-blocks; that excursion still prints as a non-blocking warning, and a sustained regression re-crosses the gate once it folds into the rolling baseline.Verified against reconstructed data from the failing run and synthetic leak/low-variance cases: the false failure drops to a non-blocking warning, while both a low-variance step-change and a noisy value that clears its own swing still hard-fail.