feat(hotblocks): name the serving instance on every response#99
Merged
Conversation
A Service load-balances across replicas behind one ClusterIP, and the kernel DNATs transparently, so the portal sees only the virtual IP and cannot tell which pod answered — nor can it ask a request that never returned. Computed once: HOSTNAME is fixed for the process's life. The portal strips every `x-internal-*` header (sqd-portal#133), released in 0.11.8. That release is the deploy gate, not a property of this branch: portals below it forward upstream headers verbatim, and mainnet-public still runs 0.11.3 behind an nginx that hides only Date/Server/X-Accel-*. Rolling this out to network-hotblocks-mainnet ahead of those portals would put pod names in front of public clients. Resolution is a pure function so the HOSTNAME mapping is testable without an env race: empty and unrepresentable names collapse to `unknown` alongside absent, since an empty value would reintroduce the ambiguity the fallback exists to kill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mo4islona
force-pushed
the
worktree-hotblocks-instance-header
branch
from
July 16, 2026 16:15
e7b9a52 to
2fe819a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
hotblocks-dbis a StatefulSet behind a ClusterIP Service. The portal connects to the Service's virtual IP and kube-proxy DNATs to a pod in the kernel, sogetpeername()— and therefore anything reqwest can report — returns the ClusterIP. The portal cannot tell which replica answered.This adds
x-internal-hotblocks-instance: <pod>to every response, so it can.Merging is safe. Deploying is gated.
sqd-portal strips every
x-internal-*header before responding (subsquid/sqd-portal#133, released in 0.11.8) and logs this one on a 5xx. But that strip is released, not deployed, and the deploy is what matters:Only 0.11.8 carries the strip (
git tag --contains 9d08fba→v0.11.8alone); below itforward_responsecopies every upstream header verbatim, with no filter at all. Every hotblocks namespace is fronted by at least one portal that does not strip, andportal.sqd.devis one of them.Implementation
One line in the existing
middleware, which wraps every route —.fallback()is applied before.layer(), andaxum::serveputs nothing outside it, so 404s are covered too. No new layer, no tower-http feature.HOSTNAMEis read once viaLazyLock(Kubernetes sets it to the pod name). Resolution lives in a pureinstance_header()so the mapping is testable without racing on process-global env. Absent, empty, and unrepresentable names all collapse tounknown: the header is always present and never empty, so the portal never has to distinguish "absent" from "unset" — and an empty value would have reintroduced exactly that ambiguity.Test
tests/instance_header.rsdrives the real binary through the harness and asserts the header is present. Verified it catches the regression — with the insert removed it fails withevery response must name its instance.That test can't pin the
HOSTNAME→ header mapping: the harness inherits the parent env, andHOSTNAMEis unexported on dev machines and on bare CI runners, so it only ever exercises the fallback.instance_header_testspins the mapping directly instead — deleting the empty-guard fails them withleft: "" / right: "unknown".Scope — worth being clear
This names the replica only for requests that got a response. It does not help the case that motivated it: a request that stalls before headers has no response to carry any header, and that is exactly the
/headtimeout a tenant hit on 2026-07-16. For a wedged replica, hotblocks' own per-pod metrics remain the only signal. This earns its place for 5xx attribution.🤖 Generated with Claude Code