feat(metrics): make ingest errors, fork signals and epoch parks countable

[mo4islona]

What

Three counters over the ingest path, which had none.

metric	labels
hotblocks_ingest_source_errors_total	source, kind — connect/timeout/http/decode/io/request/other
hotblocks_ingest_fork_signals_total	source, standing — at_tip/above_tip
hotblocks_dataset_epoch_failures_total	dataset, reason — unapplicable_fork/other

source is the configured endpoint as host:port/path, not the host. Host alone collapses
endpoints that differ only by port or dataset path into a single series, which defeats the whole
point of a per-source label — and since the configured URL carries the dataset in its path, the
full endpoint gives per-dataset attribution without plumbing a dataset label through
data-source. It is assembled by hand rather than from Url::authority(), which would put
userinfo into a Prometheus label.

Why

Transport errors (DNS/connect failures, upstream 5xx, timeouts) were WARN logs only. A stalled
ingest — every source failing to connect — was invisible on dashboards and unalertable. A log line
cannot be alerted on, cannot be aggregated across pods, and is gone from the tail once a restart
loop has rolled it away; Prometheus keeps scraping through all three.

The head probe is counted alongside the stream loop, because otherwise the counter is blind in
exactly the worst case. fetch_chain_top gates ingestion, and its only exit on total failure is
the completed > 0 guard on the deadline arm: with every source down, completed stays zero, the
timeout is disabled and the probe respins on a 5s backoff forever. StandardDataSource is never
constructed, on_error never runs, and the counter reads zero through a total outage while a
partial one — strictly less severe — counts normally. This is not confined to Head retention;
HeadProbe runs whenever the init head is Some, which includes Api retention with
max_blocks set.

Fork signals were not covered by that counter, and by construction: a 409 arrives as
Ok(BlockStreamResponse::Fork), never reaches on_error, and resets error_counter on the way
through. So the per-source error rate stays flat through the exact event that wedges ingestion —
GAP-41, where one lying endpoint of three parks the dataset while two healthy sources keep
offering the chain.

Epoch deaths were an error! log before a 60s park. A pod in a restart loop may never ship
it, which is the case where knowing matters most.

The standing label

A plain fork counter would answer the wrong question — it fires on every ordinary reorg too. The
split is strict rather than heuristic:

query/service.rs:128 answers 409 only at head.number + 1 == query.first_block(), otherwise
it waits. So an in-spec hint chain always tops out at from - 1, and a top hint below that means
the source contested a position it does not itself hold. That is exactly the harness fault's
definition (sim.rs:370), and exactly the GAP-41 shape: the adopted hints end at the liar's stale
tip, compute_rollback rejects them as below fin, ingestion parks.

The other two hint-building paths agree: query/running.rs:172 emits
first_chunk.first_block() - 1 gated on equality with query.first_block(), and
plan.rs:136 upper-bounds its window at parent_number = from - 1 (sparse chains) or
number = from (dense), sorting ascending so last() is the top. One caveat, recorded in the
metric's help text: GAP-21 — the 100-position hint window can miss the parent across a wider
hole, and an honest source then lands in above_tip. Documented as unlikely when nothing consumed
it; worth knowing now that it is alertable.

# which source is signalling forks it has no standing for
sum by (source, standing) (rate(hotblocks_ingest_fork_signals_total[5m]))

# are we parking in production, and on what
sum by (dataset, reason) (rate(hotblocks_dataset_epoch_failures_total[5m]))

# distinct sources erroring — spikes on a real incident, flat against a chronic-dead baseline
count by (pod) (sum by (pod, source) (rate(hotblocks_ingest_source_errors_total{kind="connect"}[5m])) > 0)

Notes

reason buckets off a typed UnapplicableFork, not the error text — the message carries block
numbers and hashes, so it can never become a label, and string matching would break silently on a
rewording. It also gives the previously anonymous ensure!(prev[pos].hash == finalized_head.hash)
a name.

DataClient::error_kind() / source_label() are trait methods with default impls ("other" /
"unknown"); the reqwest-specific classification stays in ReqwestDataClient.

Recording stays at the call sites rather than moving inside ReqwestDataClient. A client-level
recorder would catch every call path automatically, but on_error also counts errors that never
came from the client — failed to parse a block is wrapped there — so it would either lose those
or need both layers recording and a rule about which owns what. The probe and the stream loop are
sequential states of one controller loop, so there is no double count.

Tests

ct4_lagging_source.rs pins the discriminator from both sides — a defect lands in above_tip with
at_tip empty, an honest reorg the reverse. One test would not do: a discriminator stuck on
above_tip passes the first script and then cries source defect on every reorg in production.
Verified by inverting the comparison, which fails the suite.

ct9_source_faults.rs covers the probe: a dataset whose only source is an unreachable port must
count the failure even though ingestion never starts. It keys on the source label rather than
kind, so it pins the endpoint label format at the same time.

The observability tests are not #[ignore]d, unlike their two GAP-41 neighbours — observability
holds, conformance does not.

Scope

Nothing is fixed here. GAP-41 and GAP-5 both remain: one lying source of three still parks
ingestion, with no alarm and no recovery. This only makes that state countable, so a head held
back by a bad source reads differently from a hung service.

🤖 Generated with Claude Code