feat(metrics): make ingest errors, fork signals and epoch parks countable#81
Open
mo4islona wants to merge 2 commits into
Open
feat(metrics): make ingest errors, fork signals and epoch parks countable#81mo4islona wants to merge 2 commits into
mo4islona wants to merge 2 commits into
Conversation
mo4islona
force-pushed
the
feat/ingest-source-error-metric
branch
from
July 8, 2026 15:36
a197df6 to
88b8129
Compare
Add hotblocks_ingest_source_errors_total{source, kind} recording upstream
data-source ingestion errors (connect/dns/timeout/http/...), which were
previously only WARN logs and thus invisible on dashboards / unalertable.
- DataClient::error_kind() / source_label(): new trait methods with default
impls; the reqwest-specific classification stays in ReqwestDataClient
- recorded in StandardDataSource::on_error, registered in hotblocks metrics
- the `source` label separates a chronically-dead endpoint (constant error
spam, healthy fallback) from a real incident where many sources fail at once
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UBLDJBHu2zvs75C4XdDFrL
The 409 that triggers GAP-41 is an `Ok` response, so it never reaches `on_error` and the per-source error counter stays flat through the exact event that wedges ingestion. The park it causes was likewise only an `error!` log, which a crash-looping pod may never ship. `standing` separates the two cases that matter: a source answers 409 only at `head + 1 == from`, so an in-spec hint chain tops out at `from - 1` and anything lower contests a position the source does not hold. Without that split the counter would fire on every ordinary reorg. The epoch-death cause is bucketed off a typed `UnapplicableFork` rather than the error text, which carries block numbers and hashes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mo4islona
force-pushed
the
feat/ingest-source-error-metric
branch
from
July 20, 2026 15:29
88b8129 to
f36fa06
Compare
mo4islona
marked this pull request as ready for review
July 20, 2026 15:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three counters over the ingest path, which had none.
hotblocks_ingest_source_errors_totalsource,kind—connect/timeout/http/decode/io/otherhotblocks_ingest_fork_signals_totalsource,standing—at_tip/above_tiphotblocks_dataset_epoch_failures_totaldataset,reason—unapplicable_fork/otherWhy
Transport errors (DNS/connect failures, upstream 5xx, timeouts) were WARN logs only. A stalled ingest — every source failing to connect — was invisible on dashboards and unalertable. This came out of the recurring post-deploy freshness freezes (NET-408), where the pod's own logs weren't shipped (fluent-bit gap) while Prometheus kept scraping, so a metric is the only signal that survives.
Fork signals were not covered by that counter, and by construction: a 409 arrives as
Ok(BlockStreamResponse::Fork), never reacheson_error, and resetserror_counteron the way through. So the per-source error rate stays flat through the exact event that wedges ingestion — GAP-41, where one lying endpoint of three parks the dataset while two healthy sources keep offering the chain.Epoch deaths were an
error!log before a 60s park. A pod in a restart loop may never ship it, which is the case where knowing matters most.The
standinglabelA plain fork counter would answer the wrong question — it fires on every ordinary reorg too. The split is strict rather than heuristic:
query/service.rs:128answers 409 only athead.number + 1 == query.first_block(), otherwise it waits. So an in-spec hint chain always tops out atfrom - 1, and a top hint below that means the source contested a position it does not itself hold. That is exactly the harness fault's definition (sim.rs:370), and exactly the GAP-41 shape: the adopted hints end at the liar's stale tip,compute_rollbackrejects them as belowfin, ingestion parks.Notes
reasonbuckets off a typedUnapplicableFork, not the error text — the message carries block numbers and hashes, so it can never become a label, and string matching would break silently on a rewording. It also gives the previously anonymousensure!(prev[pos].hash == finalized_head.hash)a name.DataClient::error_kind()/source_label()are trait methods with default impls ("other"/"unknown"); the reqwest-specific classification stays inReqwestDataClient.Tests
ct4_lagging_source.rspins the discriminator from both sides — a defect lands inabove_tipwithat_tipempty, an honest reorg the reverse. One test would not do: a discriminator stuck onabove_tippasses the first script and then cries source defect on every reorg in production. Verified by inverting the comparison, which fails the suite.The observability test is not
#[ignore]d, unlike its two GAP-41 neighbours — observability holds, conformance does not.Scope
Nothing is fixed here. GAP-41 and GAP-5 both remain: one lying source of three still parks ingestion, with no alarm and no recovery. This only makes that state countable, so a head held back by a bad source reads differently from a hung service.
🤖 Generated with Claude Code