fix: retry engine_data CAS on InnoDB deadlocks instead of dropping writes#2786
Open
chubes4 wants to merge 1 commit into
Open
fix: retry engine_data CAS on InnoDB deadlocks instead of dropping writes#2786chubes4 wants to merge 1 commit into
chubes4 wants to merge 1 commit into
Conversation
…ites InnoDB deadlocks and lock-wait timeouts were returned as a generic 'db_error' and treated as non-retryable, so the optimistic-concurrency loop bailed and silently dropped the engine_data write. Concurrent events ingestion flows (parallel scraper/Ticketmaster jobs CAS-ing the same rows) hit this daily. Classify deadlock (1213) / lock-wait timeout (1205) as a transient, retryable condition and re-read the latest snapshot with a small randomized backoff, mirroring the existing logical-conflict retry path. Genuinely fatal DB errors still fail fast. Closes #2785
Contributor
Homeboy Results —
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2785. InnoDB deadlocks (and lock-wait timeouts) during the engine_data compare-and-swap were returned as a generic
db_errorand treated as non-retryable, so the optimistic-concurrency loop bailed and silently dropped the write. This is exactly the case MySQL recommends retrying ("try restarting transaction").Observed in production on events.extrachill.com — parallel scraper/Ticketmaster ingestion flows CAS the same engine_data rows at the same scheduled tick (~15:02–15:03 UTC) and step on each other's locks, dropping tool-run-state / step-progress writes.
Changes
Jobs::compare_and_swap_engine_data()— on$wpdb->updatefailure, classify$wpdb->last_errorvia a newis_retryable_db_error()helper (matches deadlock 1213 / lock-wait timeout 1205). Return aretryableflag anderror: 'deadlock', and log atwarning(noterror) when transient.EngineData::mutate()— treatretryablelike a logicalconflict: re-read the latest snapshot and retry within the existingmax_attemptsbudget, with a small randomized backoff (5–25ms) to let the winning transaction commit. Genuinely fatal DB errors still fail fast.RuntimeToolRunStateStore::mutate_engine_data()— mirror the same classification in the fallback CAS loop.Behavior
max_attempts).warning, withreason: deadlock|conflict) from fatal failure (error).Verification
php -lclean on all three files.phpcsclean (no warnings/errors) on all three files.