test(hotblocks): cover the multi-source shape production runs

[mo4islona]

Production configures several sources per dataset; every script in the suite ran exactly one. So StandardDataSource's multi-endpoint path — the fixed-order poll race, the fork consensus, the per-endpoint MaybeOnHead flush trigger — had never been executed by a test, and crates/data-source still carries 0 unit tests. This adds the missing shape.

Prompted by base-mainnet, where one endpoint has been sitting minutes behind two on the head.

What it found

A lagging source is harmless to the head. The hypothesis going in was that it is not — that the tip flush (DataEvent::MaybeOnHead) fires only when the endpoint whose stream just ended is itself the one that committed the head block (ep.last_committed_block == Some(prev_block), per-endpoint, while position is shared), so an interleaving laggard could hold the head back. It does not reproduce, in the minority shape or the majority one, and not because the healthy source wins the poll order: reversing the endpoint list changes nothing. A laggard sits below the shared cursor, so it has nothing to serve and never wins the race from any slot. The honest endpoints keep advancing the cursor, and a fork claim is dropped as soon as the cursor moves past it (standard.rs:118-124).

What wedges the service is a source that answers wrongly, not one that answers late — and one of three is enough. SimFaults::fork_signal_above_tip: an endpoint answers a parent assertion above its own tip with a fork signal, which RP-5b confines to from == tip + 1. At one signalling endpoint of three there is no majority (1 > 3/2 is false) and the honest endpoints stay active (1 == 3 is false); the door is the third clause, the 2 s fork_consensus_timeout, which fires on any poll where every endpoint returned Pending. Measured, 5/5 runs, by instrumenting poll_next_event:

forks=1 endpoints=3 active=3 majority=false all_active=false timeout=true

extract_fork adopts the liar's stale hints, compute_rollback rejects them as below fin, and the epoch parks for P-EPOCH-RETRY — head frozen, no alarm, no recovery, two healthy sources still offering the chain.

This is why the credulous consensus (GAP-41) is filed P1 and not P2. accept_new_block clears the fork-consensus timer only on a commit, so on a chain with block time ≥ 2 s every inter-block gap is a firing window. The 50 ms cadence in the first draft of this script is what hid it: at that speed the same single fault signals repeatedly and the service conforms.

The simulator was wrong in exactly the place under test. It answered a parent assertion naming any position above its tip with a fork signal. RP-5b confines that signal to from == tip + 1, the one position where the assertion is evaluable against a block the source holds; above it the answer is no-data. The old behavior made a source that is merely behind indistinguishable from one that disagrees — which would have silently turned every future lag script into a fork script. Kept as an explicit fault, pinned in both directions by a unit test.

Gaps

Nothing is closed — no production code changed. GAP-5 and GAP-30 stay open; GAP-5 gains the deterministic repro it lacked, GAP-30 gains the lag dimension: selection is per block, not per source (first arrival that links onto the cursor wins, ties to config order), an endpoint leaves the rotation only by erroring, so slowness is unnameable and the only lever is removing a source by hand. Lag-aware demotion needs per-source observability first — PR #81 is that substrate and has been in draft since 2026-07-08.

Tests

ct4_three_sources_in_step_conform ............................... ok
ct4_a_lagging_source_does_not_hold_the_head_back ................ ok   (1 of 3 behind — production's shape)
ct4_two_lagging_sources_of_three_do_not_hold_the_head_back ...... ok   (the margin)
ct4_a_caught_up_source_changes_nothing .......................... ok
ct4_a_single_source_signalling_a_fork_above_its_tip... .......... ignored (repro, fails in ~59s)
ct4_a_fork_signal_majority_above_the_tip... ..................... ignored (repro, other clause)

Both wedge repros are pinned because they reach the same terminal state through different clauses — a fix closing only the majority path would leave the single-source one green while a lone bad source still wedges.

Full -p sqd-hotblocks -p sqd-hotblocks-harness suite green (77 passed), cargo +nightly fmt --check clean, no new clippy warnings.

🤖 Generated with Claude Code