ci(prod-vectors): unpin rayon and lower xdist workers for prod fill#920
Merged
tcoratger merged 1 commit intoJun 11, 2026
Merged
Conversation
The prod fixture fill spends ~56m almost entirely in the real_crypto smoke vectors, which build genuine production XMSS proofs. Each proof parallelizes internally with rayon, but the job pinned RAYON_NUM_THREADS=1 while running one xdist worker per core, so every heavy proof was starved to a single thread. Add a FILL_WORKERS knob to the canonical fill recipe (default auto, so the test-scheme job is unchanged) and set it to 2 for the prod job, with rayon left unpinned. A heavy proof can now spread across cores instead of one thread, while two workers still overlap proofs and clear the mocked backlog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The "Fill production test fixtures" job runs ~56m on
main, while the test-scheme fill is ~4m. After #895 mocked the aggregation prover for both schemes, the prod run is dominated almost entirely by thereal_crypto(smoke=True)vectors — a handful of genuine production XMSS proofs that intentionally bypass the mock.Each of those proofs parallelizes internally via rayon. But the prod job pinned
RAYON_NUM_THREADS=1while running one xdist worker per core (-n auto), so every heavy proof was starved to a single thread, and idle cores were never used to speed up an individual proof.Change
FILL_WORKERSknob to the canonicalfill-cirecipe. It defaults toauto, so the test-scheme job is unchanged.FILL_WORKERS=2and drop theRAYON_NUM_THREADS=1pin.A heavy proof can now spread across all cores via rayon instead of one thread, while two xdist workers still overlap proofs and clear the (instant) mocked backlog.
Expected impact and caveat
The mocking from #895 is working correctly — this only retunes parallelism for the real-proof subset. The realistic win is on the tail (when fewer heavy proofs remain than cores, a single proof now uses all cores). The ~4m figure is the test-parameter floor; the prod floor is the inherent cost of real production SNARK proofs, so expect a moderate improvement rather than a return to single-digit minutes.
If this doesn't move the needle enough, the higher-leverage levers are a larger runner or trimming the
real_crypto(smoke=True)subset for the prod scheme.🤖 Generated with Claude Code