node: pause derivation while P2P block sync is catching up

[panos-xyz]

Problem Statement

When a Morph full node starts from empty L2 data, P2P block sync and L1 derivation start at the same time. Derivation can discover a committed batch whose L2 blocks have not arrived through P2P yet. In local verification mode it waits for P2P backfill without advancing the derivation L1 cursor, but continues polling eth_getLogs from the same L1 block every 15 seconds.

As the L1 head advances, the pinned fromBlock falls outside the recent-history window offered by non-archive RPC providers. The node then receives permanent 403 Archive requests require a personal token errors. This also generates unnecessary L1 RPC traffic during initial block sync.

Reproduction

Environment:

• Network: Hoodi
• Morph commit: 21d05b92b17623c4a6b9c2463e6bd1da78eb2bb7
• Morph Node: v0.5.7-37-g21d05b92
• derivation.verify-mode=local (default)
• Empty Tendermint state and empty Morph Node store
• Non-archive Hoodi L1 RPC

Steps:

1. Start a full node from block 0 with P2P block sync enabled.
2. Let derivation start with no persisted cursor.
3. Observe derivation initialize at the latest confirmed L1 block and initially advance normally.
4. When derivation discovers a batch whose L2 tip is not available locally, observe it return with waiting for P2P backfill instead of L1 derivation.
5. Leave P2P block sync running while the L1 head advances.
6. Observe derivation retry eth_getLogs from the same fromBlock every poll until the provider rejects it as an archive request.

Observed trace:

derivation startHeight defaulted to latest L1 confirmed block height=3210904
write latest derivation l1 height success l1BlockNumber=3210937
local verify: batch firstBlockNumber is pre-upgrade; waiting for P2P backfill instead of L1 derivation batchIndex=22144 firstBlockNumber=6855908
derivation start pull rollupData form l1 startBlock=3210938 end=<growing L1 head>
eth_getLogs failed: 403 Archive requests require a personal token

A direct RPC boundary check against the same endpoint returned:

fromBlock = confirmed head - 89: HTTP 200
fromBlock = confirmed head - 90: HTTP 403 archive required

A single-block query for the pinned historical block also returns 403, so reducing derivation.fetchBlockRange does not resolve the stalled state.

Solution

Treat initial P2P catch-up as a lifecycle phase before local-mode derivation starts:

• While Tendermint reports catching_up=true, do not run Rollup derivation or its finalizer. The L1 Message Queue syncer remains independent and continues running.
• For a fresh node with no configured start height and no persisted derivation cursor, resolve the default start height from the then-current latest confirmed L1 block only after catch-up completes. This preserves Morph Node's existing first-run semantics, applied when derivation becomes ready rather than when the process starts.
• Preserve an explicitly configured start height and every existing persisted cursor. Never silently advance or rewrite an existing cursor; operators recovering an already-stalled historical cursor may still need an archive RPC.
• Resume normal derivation, finalization, and L1 reorg handling after catch-up.
• Keep sequencer behavior, layer1 verification mode, and the store schema unchanged.
• Log the pause and resume transitions once.

For a fresh empty cursor, batches committed while the node is performing historical P2P catch-up are treated as pre-derivation history, just like batches preceding the old first-run default start height. Derivation verifies new commitments after it becomes ready.

User Stories

1. As a full-node operator, I want a node started from empty data to complete P2P block sync without requiring an archive L1 RPC solely because derivation started too early.
2. As an RPC operator, I want initial block sync to avoid repeated Rollup eth_getLogs requests whose results cannot yet be processed.
3. As a protocol operator, I want normal derivation and L1 reorg checks to start after P2P catch-up.
4. As an operator with an explicit or persisted cursor, I want it preserved rather than silently moved.
5. As a node developer, I want the catch-up transition covered by a deterministic regression test.

Testing Decisions

Use the derivation service seam with a fake Tendermint sync status and a recording L1 RPC service:

1. Across repeated cycles with catching_up=true, assert zero L1-head reads, zero Rollup FilterLogs calls, and no persisted derivation cursor.
2. Transition to catching_up=false, return successful empty Rollup logs and contract/header responses, then assert the first-run cursor advances from the latest confirmed L1 height.
3. Run the test under the race detector.
4. Keep existing L1 reorg and verification-mode tests passing.
5. In a real from-zero Hoodi run, confirm the pause log appears and no new derivation eth_getLogs calls occur while status reports catching_up=true.

Out of Scope

• Migrating or deleting an existing stalled derivation cursor.
• Removing L1 CommitBatch discovery after initial sync.
• Changing public RPC provider archive policies.
• Changing L1 Message Queue synchronization.
• Modifying Reth block execution or state-root verification.

Further Notes

The failing calls are from the derivation module's Rollup CommitBatch(uint256,bytes32) filter, not the L1 Message Queue syncer. Explicit db.dir configuration and smaller query ranges do not address the lifecycle mismatch.

[curryxbo]

A few notes to help scope this:

• Root cause: in local verify, startHeight defaults to the latest confirmed L1 head (node/derivation/derivation.go:166-172). Before P2P catch-up, a batch whose L2 tip isn't local returns without advancing the cursor, so getLogs(fromBlock) stays pinned; as the L1 head moves, fromBlock falls out of the non-archive window → permanent 403. getLogs runs before batch processing, so once it ages out the node can't re-read the log even after P2P catches up — self-locking.
• "Trusting history" is already the default, not newly introduced here: with startHeight defaulting to the confirmed head, everything below the first verified batch was always trusted, never L1-verified. So the real question is trust boundary vs. non-archive window, not whether to trust history.
• Not limited to from-zero nodes: any node with a persisted cursor that restarts after downtime longer than the non-archive window (~90 blocks) hits the same getLogs(stale cursor) 403.

Suggested direction: rather than patching only the from-zero path, make the trust height an explicit, adjustable, forward-resettable knob covering both from-zero and fell-behind restarts — with the trust shortcut explicit/auditable rather than a silent default.