Skip to content

fix(server): log why a readiness probe failed instead of discarding the cause - #332

Merged
ntatschner merged 1 commit into
nextfrom
fix/readyz-log-dep-failures
Jul 26, 2026
Merged

fix(server): log why a readiness probe failed instead of discarding the cause#332
ntatschner merged 1 commit into
nextfrom
fix/readyz-log-dep-failures

Conversation

@ntatschner

Copy link
Copy Markdown
Collaborator

Production is currently returning 503 from /readyz with minio: "fail" — and there is no way to find out why.

{"status":"degraded","version":"1.8.132","checks":{"postgres":"ok","spicedb":"ok","minio":"fail"}}

The gap

ready() collapsed each probe with .is_ok(), discarding the error:

let minio_status = match minio.as_ref() {
    Some(mirror) => Some(mirror.ping().await.is_ok()),   // <- cause dropped here
    None => None,
};

The probes build genuinely useful errors. MinioMirror::ping() is a HeadBucket wrapped with:

MinIO HeadBucket failed for '{bucket}' (bucket missing or credentials invalid)

None of that is logged, and the response vocabulary is one word. So an unreachable endpoint, a missing bucket, and a rotated credential are indistinguishable — from outside and from the logs. That is the state right now: the deployment is degraded and nothing recorded which of the three it is.

Change

Each probe logs its full cause chain before being collapsed to the bool the response needs. Postgres and SpiceDB get the same treatment — they have the identical blind spot, and a silently unexplained Postgres readiness failure would be worse than this one.

WARN readyz: minio probe failed  error="MinIO HeadBucket failed for `starstats-audit` (bucket missing or credentials invalid): dispatch failure: io error: ..."

Uses {e:#} (single-line, JSON-pipeline friendly) — consistent with smtp_admin_routes.rs:351 and #331.

Logged server-side only — deliberately

The cause is not added to the response body. /readyz is publicly reachable (I read the 503 above from the open internet) and the text carries bucket names and endpoints. The body keeps its coarse ok/fail/skipped/degraded vocabulary — no schema change, no new information disclosure.

Note on scope

This does not fix MinIO. It makes the next probe say what is wrong, which is the blocker for fixing it. Given the mirror is best-effort (Postgres is the audit source of truth and writes still succeed), the practical impact today is that every mirror PUT is failing silently — the /readyz 503 is the only signal, and it is unexplained.

cargo test -p starstats-server 959 + 931, clippy -D warnings + cargo fmt --all --check clean. No migration, no behaviour change beyond log output.

…he cause

`/readyz` collapsed each dependency probe with `.is_ok()`, throwing the
error away. The probes build good ones — MinIO's says which bucket and
that it is "missing or credentials invalid" — but the response can only
ever say the single word "fail", and nothing was logged.

So a degraded deployment is undiagnosable: an unreachable endpoint, a
missing bucket, and a rotated credential are indistinguishable from the
outside AND from the logs. That is live right now — production has been
returning 503 with `minio: fail` and there is no recoverable cause
anywhere to say which of the three it is.

Each probe now logs its full cause chain (`{e:#}`, single-line for the
JSON pipeline) before being collapsed to the bool the response needs.
Postgres and SpiceDB get the same treatment — they have the identical
blind spot, and a silent Postgres readiness failure would be worse.

Logged server-side ONLY. The cause is deliberately NOT added to the
response body: `/readyz` is publicly reachable and the text carries
bucket names and endpoints. The body keeps its coarse
ok/fail/skipped/degraded vocabulary, so no schema change and no new
information disclosure.

Same failure shape as the mailer logging fix: the diagnostic was
already written, and the call site dropped it on the floor.
@ntatschner
ntatschner merged commit 5686a94 into next Jul 26, 2026
12 checks passed
@ntatschner
ntatschner deleted the fix/readyz-log-dep-failures branch July 26, 2026 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant