feat(metrics): make ingest errors, fork signals and epoch parks countable

[mo4islona]

What

Three counters over the ingest path, which had none.

metric	labels
hotblocks_ingest_source_errors_total	source, kind — connect/timeout/http/decode/io/other
hotblocks_ingest_fork_signals_total	source, standing — at_tip/above_tip
hotblocks_dataset_epoch_failures_total	dataset, reason — unapplicable_fork/other

Why

Transport errors (DNS/connect failures, upstream 5xx, timeouts) were WARN logs only. A stalled ingest — every source failing to connect — was invisible on dashboards and unalertable. This came out of the recurring post-deploy freshness freezes (NET-408), where the pod's own logs weren't shipped (fluent-bit gap) while Prometheus kept scraping, so a metric is the only signal that survives.

Fork signals were not covered by that counter, and by construction: a 409 arrives as Ok(BlockStreamResponse::Fork), never reaches on_error, and resets error_counter on the way through. So the per-source error rate stays flat through the exact event that wedges ingestion — GAP-41, where one lying endpoint of three parks the dataset while two healthy sources keep offering the chain.

Epoch deaths were an error! log before a 60s park. A pod in a restart loop may never ship it, which is the case where knowing matters most.

The standing label

A plain fork counter would answer the wrong question — it fires on every ordinary reorg too. The split is strict rather than heuristic:

query/service.rs:128 answers 409 only at head.number + 1 == query.first_block(), otherwise it waits. So an in-spec hint chain always tops out at from - 1, and a top hint below that means the source contested a position it does not itself hold. That is exactly the harness fault's definition (sim.rs:370), and exactly the GAP-41 shape: the adopted hints end at the liar's stale tip, compute_rollback rejects them as below fin, ingestion parks.

# which source is signalling forks it has no standing for
sum by (source, standing) (rate(hotblocks_ingest_fork_signals_total[5m]))

# are we parking in production, and on what
sum by (dataset, reason) (rate(hotblocks_dataset_epoch_failures_total[5m]))

# distinct sources erroring — spikes on a real incident, flat against a chronic-dead baseline
count by (pod) (sum by (pod, source) (rate(hotblocks_ingest_source_errors_total{kind="connect"}[5m])) > 0)

Notes

reason buckets off a typed UnapplicableFork, not the error text — the message carries block numbers and hashes, so it can never become a label, and string matching would break silently on a rewording. It also gives the previously anonymous ensure!(prev[pos].hash == finalized_head.hash) a name.

DataClient::error_kind() / source_label() are trait methods with default impls ("other" / "unknown"); the reqwest-specific classification stays in ReqwestDataClient.

Tests

ct4_lagging_source.rs pins the discriminator from both sides — a defect lands in above_tip with at_tip empty, an honest reorg the reverse. One test would not do: a discriminator stuck on above_tip passes the first script and then cries source defect on every reorg in production. Verified by inverting the comparison, which fails the suite.

The observability test is not #[ignore]d, unlike its two GAP-41 neighbours — observability holds, conformance does not.

Scope

Nothing is fixed here. GAP-41 and GAP-5 both remain: one lying source of three still parks ingestion, with no alarm and no recovery. This only makes that state countable, so a head held back by a bad source reads differently from a hung service.

🤖 Generated with Claude Code