feat(metrics): make ingest errors, fork signals and epoch parks countable#81
Merged
Merged
Conversation
mo4islona
force-pushed
the
feat/ingest-source-error-metric
branch
from
July 8, 2026 15:36
a197df6 to
88b8129
Compare
mo4islona
force-pushed
the
feat/ingest-source-error-metric
branch
from
July 20, 2026 15:29
88b8129 to
f36fa06
Compare
mo4islona
marked this pull request as ready for review
July 20, 2026 15:37
mo4islona
force-pushed
the
feat/ingest-source-error-metric
branch
from
July 20, 2026 16:15
f36fa06 to
d344daa
Compare
…able Transport errors were WARN logs only, so a stalled ingest was invisible on dashboards and unalertable: a log line cannot be alerted on, cannot be aggregated across pods, and is gone from the tail once a restart loop has rolled it away. The head probe is counted alongside the stream loop, because otherwise the counter is blind in the worst case. `fetch_chain_top` gates ingestion and its only exit on total failure is the `completed > 0` guard on the deadline arm, so with every source down it respins forever, `StandardDataSource` is never constructed and `on_error` never runs. A partial outage would count normally while a total one read zero. Fork signals needed a counter of their own: a 409 arrives as `Ok`, never reaches `on_error`, and resets `error_counter` on the way through, so the per-source error rate stays flat through the exact event that wedges ingestion. `standing` splits the two cases — a source answers 409 only at `head + 1 == from`, so an in-spec hint chain tops out at `from - 1` and anything lower contests a position the source does not hold. Without that split the counter would fire on every ordinary reorg. `source` carries the endpoint as `host:port/path` rather than the host, which collapsed endpoints differing only by port or dataset path into one series; it is assembled by hand because `Url::authority()` would put userinfo into a label. Epoch deaths bucket off a typed `UnapplicableFork` rather than the error text, which carries block numbers and hashes. Nothing is fixed here: one lying source of three still parks ingestion, with no alarm and no recovery. This only makes that state countable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mo4islona
force-pushed
the
feat/ingest-source-error-metric
branch
from
July 20, 2026 16:16
d344daa to
e4be68c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three counters over the ingest path, which had none.
hotblocks_ingest_source_errors_totalsource,kind—connect/timeout/http/decode/io/request/otherhotblocks_ingest_fork_signals_totalsource,standing—at_tip/above_tiphotblocks_dataset_epoch_failures_totaldataset,reason—unapplicable_fork/othersourceis the configured endpoint ashost:port/path, not the host. Host alone collapsesendpoints that differ only by port or dataset path into a single series, which defeats the whole
point of a per-source label — and since the configured URL carries the dataset in its path, the
full endpoint gives per-dataset attribution without plumbing a
datasetlabel throughdata-source. It is assembled by hand rather than fromUrl::authority(), which would putuserinfo into a Prometheus label.
Why
Transport errors (DNS/connect failures, upstream 5xx, timeouts) were WARN logs only. A stalled
ingest — every source failing to connect — was invisible on dashboards and unalertable. A log line
cannot be alerted on, cannot be aggregated across pods, and is gone from the tail once a restart
loop has rolled it away; Prometheus keeps scraping through all three.
The head probe is counted alongside the stream loop, because otherwise the counter is blind in
exactly the worst case.
fetch_chain_topgates ingestion, and its only exit on total failure isthe
completed > 0guard on the deadline arm: with every source down,completedstays zero, thetimeout is disabled and the probe respins on a 5s backoff forever.
StandardDataSourceis neverconstructed,
on_errornever runs, and the counter reads zero through a total outage while apartial one — strictly less severe — counts normally. This is not confined to
Headretention;HeadProberuns whenever the init head isSome, which includesApiretention withmax_blocksset.Fork signals were not covered by that counter, and by construction: a 409 arrives as
Ok(BlockStreamResponse::Fork), never reacheson_error, and resetserror_counteron the waythrough. So the per-source error rate stays flat through the exact event that wedges ingestion —
GAP-41, where one lying endpoint of three parks the dataset while two healthy sources keep
offering the chain.
Epoch deaths were an
error!log before a 60s park. A pod in a restart loop may never shipit, which is the case where knowing matters most.
The
standinglabelA plain fork counter would answer the wrong question — it fires on every ordinary reorg too. The
split is strict rather than heuristic:
query/service.rs:128answers 409 only athead.number + 1 == query.first_block(), otherwiseit waits. So an in-spec hint chain always tops out at
from - 1, and a top hint below that meansthe source contested a position it does not itself hold. That is exactly the harness fault's
definition (
sim.rs:370), and exactly the GAP-41 shape: the adopted hints end at the liar's staletip,
compute_rollbackrejects them as belowfin, ingestion parks.The other two hint-building paths agree:
query/running.rs:172emitsfirst_chunk.first_block() - 1gated on equality withquery.first_block(), andplan.rs:136upper-bounds its window atparent_number = from - 1(sparse chains) ornumber = from(dense), sorting ascending solast()is the top. One caveat, recorded in themetric's help text: GAP-21 — the 100-position hint window can miss the parent across a wider
hole, and an honest source then lands in
above_tip. Documented as unlikely when nothing consumedit; worth knowing now that it is alertable.
Notes
reasonbuckets off a typedUnapplicableFork, not the error text — the message carries blocknumbers and hashes, so it can never become a label, and string matching would break silently on a
rewording. It also gives the previously anonymous
ensure!(prev[pos].hash == finalized_head.hash)a name.
DataClient::error_kind()/source_label()are trait methods with default impls ("other"/"unknown"); the reqwest-specific classification stays inReqwestDataClient.Recording stays at the call sites rather than moving inside
ReqwestDataClient. A client-levelrecorder would catch every call path automatically, but
on_erroralso counts errors that nevercame from the client —
failed to parse a blockis wrapped there — so it would either lose thoseor need both layers recording and a rule about which owns what. The probe and the stream loop are
sequential states of one controller loop, so there is no double count.
Tests
ct4_lagging_source.rspins the discriminator from both sides — a defect lands inabove_tipwithat_tipempty, an honest reorg the reverse. One test would not do: a discriminator stuck onabove_tippasses the first script and then cries source defect on every reorg in production.Verified by inverting the comparison, which fails the suite.
ct9_source_faults.rscovers the probe: a dataset whose only source is an unreachable port mustcount the failure even though ingestion never starts. It keys on the
sourcelabel rather thankind, so it pins the endpoint label format at the same time.The observability tests are not
#[ignore]d, unlike their two GAP-41 neighbours — observabilityholds, conformance does not.
Scope
Nothing is fixed here. GAP-41 and GAP-5 both remain: one lying source of three still parks
ingestion, with no alarm and no recovery. This only makes that state countable, so a head held
back by a bad source reads differently from a hung service.
🤖 Generated with Claude Code