perf(hotblocks): prepare unspilled chunks in memory

[mo4islona]

Problem

At chain head every flush (= every block on 1s chains) builds a full out-of-core ChunkProcessor: 488 temp files for one EVM block (solana: 229), then tears them down. Measured on the prod node (a production fleet node): ~52 ms per head flush, ~90% kernel time, create+teardown alone = 86%. Fleet-wide that is ~10 CPU cores of syscall/inode churn, growing linearly with dataset count. Full investigation record: docs/adr/0001-in-memory-chunk-prepare.md + docs/measurements/ (in this PR).

Fix

DataBuilder::finish takes an in-memory prepared path when processor.is_none() — which by construction (maybe_flush trims the builder after every block) means the whole chunk still sits in the builder, under the 30 MiB spill bound. prepare_in_memory() sorts and downcasts straight from the builder slices into a new TableReader::Mem variant. No temp files, no syscalls.

Chunks that DID spill (backfill, fat bursts) keep the disk path bit-for-bit unchanged — that is the out-of-core design's home turf.

Numbers — measured on the prod node (production fleet node, overlay /tmp, image e96c877)

crates/data/benches/flush_spill interleaves the three modes and asserts identical record batches on every measured iteration:

rows	fresh p50 (old path)	mem p50 (this PR)	saves
1 (head block)	52.4 ms	0.224 ms	100%
100	53.4 ms	0.453 ms	99%
1,000	56.1 ms	2.1 ms	96%
10,000	116.7 ms	22.1 ms	81%

Cross-mode output equality held for the whole sweep, both on overlay /tmp and tmpfs.

Safety

• Differential tests (crates/data-core/tests/in_memory_prepare.rs): byte-equality with the spill path — sorted + plain tables, strings, lists, nulls, empty tables, partial read windows, chunk-wide downcast consistency (incl. >u32::MAX block numbers).
• Decision point is deterministic and local; no new threshold or config — finish() re-checks the existing 30 MiB spill bound, so an oversized unspilled chunk (a row-bound flush whose final block is fat) falls back to the disk path. Pinned by unit tests on both sides of the bound.
• Ownership model unchanged: a prepared chunk is still built once, consumed once.
• Memory: bounded by the existing 30 MiB spill bound × ≤3 in-flight chunks per dataset (ingest→controller mpsc cap 1); ~1 GiB steady estimate at 100 datasets. Estimate ≠ allocation bound (builders retain capacity) — peak-RSS to be validated on the canary.
• into_processor() on a mem-prepared table errors explicitly — processor reuse is a separate, riskier fix (see the ADR, Alternatives).

Rollout

testnet-dev → mainnet-internal (canary) → mainnet/morpho, judged by same-pod before/after per-thread profile (same-pod before/after per-thread profile). Rollback = image tag revert.

Before undrafting

• Re-run flush_spill on a production fleet node with the mem mode (image e96c877) — 52.4 → 0.224 ms, −99.6%
• Peak-RSS under harness load — plateaus ~110 MB above the spill path at 25k blocks / ~177 blocks/s (150× prod per-dataset head rate); heap-instead-of-page-cache, no leak shape; crates/hotblocks/tests/peak_rss.rs
• Tie-order caveat reviewed — ties are REAL (evm statediffs, solana instructions), but harmless: query layer re-sorts all output by a row-unique primary key (compute_order), and byte-equality on ties currently holds anyway (pdqsort sorted-run identity, pinned by test; multiset test is the semantic floor)

🤖 Generated with Claude Code