Skip to content

fix(server): retry all PD peers while waiting for storage - #3129

Open
bitflicker64 wants to merge 1 commit into
apache:masterfrom
bitflicker64:fix/wait-storage-pd-failover-3123
Open

fix(server): retry all PD peers while waiting for storage#3129
bitflicker64 wants to merge 1 commit into
apache:masterfrom
bitflicker64:fix/wait-storage-pd-failover-3123

Conversation

@bitflicker64

@bitflicker64 bitflicker64 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Kept in sync with hugegraph#175

Purpose of the PR

wait-storage.sh selected the first PD answering /v1/health, then polled
/v1/stores only on that peer. Since the health endpoint can succeed before
the peer has raft-backed store state, a storeless first peer could block Server
startup even when another configured PD already reported an Up store.

Main Changes

  • Probe /v1/stores across every configured PD peer on every retry.
  • Succeed as soon as any peer reports a store in state Up, without a separate
    /v1/health selection stage.
  • Preserve authentication, configured peer order, custom REST endpoints, retry
    timing, and the existing outer timeout failure.
  • Add a deterministic shell regression suite and an isolated Server CI job.

Verifying these changes

  • Need tests and can be verified as follows:
    • test-wait-storage.sh: 4 scenarios passed, 0 failed.
    • The same suite fails against the unmodified script on the pinned first peer.
    • bash -n passed for the production and test scripts.
    • Server CI workflow YAML parsing and git diff --check passed.
    • Apache RAT and EditorConfig checks passed for the 20-module dependency set.

Does this PR potentially affect the following parts?

  • Dependencies
  • Modify configurations
  • The public API
  • Other affects (HStore Server startup readiness)
  • Nope

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need

Out of scope

  • PD and Store health endpoint semantics
  • wait-partition.sh
  • Environment-variable and timeout configurability cleanup
  • Docker Compose health checks and documentation

@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. ci-cd Build or deploy labels Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-cd Build or deploy size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] wait-storage.sh binds to the first PD answering /v1/health and never fails over to other peers

1 participant