Skip to content

ci(prod-vectors): unpin rayon and lower xdist workers for prod fill#920

Merged
tcoratger merged 1 commit into
leanEthereum:mainfrom
tcoratger:ci/prod-vectors-rayon-threads
Jun 11, 2026
Merged

ci(prod-vectors): unpin rayon and lower xdist workers for prod fill#920
tcoratger merged 1 commit into
leanEthereum:mainfrom
tcoratger:ci/prod-vectors-rayon-threads

Conversation

@tcoratger

Copy link
Copy Markdown
Collaborator

Problem

The "Fill production test fixtures" job runs ~56m on main, while the test-scheme fill is ~4m. After #895 mocked the aggregation prover for both schemes, the prod run is dominated almost entirely by the real_crypto(smoke=True) vectors — a handful of genuine production XMSS proofs that intentionally bypass the mock.

Each of those proofs parallelizes internally via rayon. But the prod job pinned RAYON_NUM_THREADS=1 while running one xdist worker per core (-n auto), so every heavy proof was starved to a single thread, and idle cores were never used to speed up an individual proof.

Change

  • Add a FILL_WORKERS knob to the canonical fill-ci recipe. It defaults to auto, so the test-scheme job is unchanged.
  • In the prod-vectors job, set FILL_WORKERS=2 and drop the RAYON_NUM_THREADS=1 pin.

A heavy proof can now spread across all cores via rayon instead of one thread, while two xdist workers still overlap proofs and clear the (instant) mocked backlog.

Expected impact and caveat

The mocking from #895 is working correctly — this only retunes parallelism for the real-proof subset. The realistic win is on the tail (when fewer heavy proofs remain than cores, a single proof now uses all cores). The ~4m figure is the test-parameter floor; the prod floor is the inherent cost of real production SNARK proofs, so expect a moderate improvement rather than a return to single-digit minutes.

If this doesn't move the needle enough, the higher-leverage levers are a larger runner or trimming the real_crypto(smoke=True) subset for the prod scheme.

🤖 Generated with Claude Code

The prod fixture fill spends ~56m almost entirely in the real_crypto
smoke vectors, which build genuine production XMSS proofs.

Each proof parallelizes internally with rayon, but the job pinned
RAYON_NUM_THREADS=1 while running one xdist worker per core, so every
heavy proof was starved to a single thread.

Add a FILL_WORKERS knob to the canonical fill recipe (default auto, so
the test-scheme job is unchanged) and set it to 2 for the prod job, with
rayon left unpinned. A heavy proof can now spread across cores instead of
one thread, while two workers still overlap proofs and clear the mocked
backlog.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tcoratger tcoratger merged commit 53df0fe into leanEthereum:main Jun 11, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant