ci(deploy-test): retry once on the FirstBootDone timeout flake, keep its serial console - #165
Merged
Merged
Conversation
…its serial console Master's deploy-test went red again on 2026-08-02 (run 30737611048, ipv6-outputs leg): FirstBootDone "WaitCondition timed out. Received 0 conditions when expecting 1" — the intermittent unprovisioned-boot flake tracked in issue #112, with issue #84 the suspected root cause. Every recurrence costs a red master and, on the ipv6-outputs leg, destroys the only evidence (that leg has no SSM path, so the failed instance's console is gone once the stack is deleted). The Create-stack step now: - retries create ONCE, but only when the failure is specifically the FirstBootDone WaitCondition timeout — template bugs, quota and IAM failures still fail hard rather than being masked by a retry; - dumps the FAILED instance's serial console before deleting the first attempt, so every recurrence leaves issue-84 evidence in the log; - cancels any Spot request on the failed attempt before deletion (the same re-open hazard the Destroy step already guards). Job timeout goes 30 -> 60 minutes: two create waits (the WaitCondition alone is 15 min) + a delete + the post-create asserts need the headroom. Verification: workflow YAML parses; the embedded script passes `sh -n` (with ${{...}} stubbed) and actionlint reports only the same info-level shellcheck classes as the pre-existing steps (unquoted $sirs is deliberate multi-id word-splitting, matching the Destroy step). The retry path itself can only be exercised by a real recurrence — the next one will show the warning, the console dump, and a second attempt instead of a red run. Refs #112, #84. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VoCn5tqBY3hKKwmtCH1kx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Master went red on run 30737611048 (ipv6-outputs leg):
FirstBootDone — WaitCondition timed out, the intermittent unprovisioned-boot flake of #112 / #84.Verification: YAML parses, embedded script passes
sh -n, actionlint shows only the pre-existing info-level shellcheck classes. The retry path is only exercisable by a real recurrence; the next flake will retry instead of reddening master, and will leave console evidence either way.AWS cost impact: a retried run keeps a t4g.small up for up to ~35 extra minutes; no IAM/networking changes.
Refs #112, #84.
🤖 Generated with Claude Code
https://claude.ai/code/session_013VoCn5tqBY3hKKwmtCH1kx