fix(server): log why a readiness probe failed instead of discarding the cause - #332
Merged
Conversation
…he cause
`/readyz` collapsed each dependency probe with `.is_ok()`, throwing the
error away. The probes build good ones — MinIO's says which bucket and
that it is "missing or credentials invalid" — but the response can only
ever say the single word "fail", and nothing was logged.
So a degraded deployment is undiagnosable: an unreachable endpoint, a
missing bucket, and a rotated credential are indistinguishable from the
outside AND from the logs. That is live right now — production has been
returning 503 with `minio: fail` and there is no recoverable cause
anywhere to say which of the three it is.
Each probe now logs its full cause chain (`{e:#}`, single-line for the
JSON pipeline) before being collapsed to the bool the response needs.
Postgres and SpiceDB get the same treatment — they have the identical
blind spot, and a silent Postgres readiness failure would be worse.
Logged server-side ONLY. The cause is deliberately NOT added to the
response body: `/readyz` is publicly reachable and the text carries
bucket names and endpoints. The body keeps its coarse
ok/fail/skipped/degraded vocabulary, so no schema change and no new
information disclosure.
Same failure shape as the mailer logging fix: the diagnostic was
already written, and the call site dropped it on the floor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production is currently returning 503 from
/readyzwithminio: "fail"— and there is no way to find out why.{"status":"degraded","version":"1.8.132","checks":{"postgres":"ok","spicedb":"ok","minio":"fail"}}The gap
ready()collapsed each probe with.is_ok(), discarding the error:The probes build genuinely useful errors.
MinioMirror::ping()is aHeadBucketwrapped with:None of that is logged, and the response vocabulary is one word. So an unreachable endpoint, a missing bucket, and a rotated credential are indistinguishable — from outside and from the logs. That is the state right now: the deployment is degraded and nothing recorded which of the three it is.
Change
Each probe logs its full cause chain before being collapsed to the bool the response needs. Postgres and SpiceDB get the same treatment — they have the identical blind spot, and a silently unexplained Postgres readiness failure would be worse than this one.
Uses
{e:#}(single-line, JSON-pipeline friendly) — consistent withsmtp_admin_routes.rs:351and #331.Logged server-side only — deliberately
The cause is not added to the response body.
/readyzis publicly reachable (I read the 503 above from the open internet) and the text carries bucket names and endpoints. The body keeps its coarseok/fail/skipped/degradedvocabulary — no schema change, no new information disclosure.Note on scope
This does not fix MinIO. It makes the next probe say what is wrong, which is the blocker for fixing it. Given the mirror is best-effort (Postgres is the audit source of truth and writes still succeed), the practical impact today is that every mirror PUT is failing silently — the
/readyz503 is the only signal, and it is unexplained.cargo test -p starstats-server959 + 931, clippy-D warnings+cargo fmt --all --checkclean. No migration, no behaviour change beyond log output.