perf(dex_solana.trades test): bound accepted_range scan to a recent window#9779
Open
a-monteiro wants to merge 1 commit into
Open
perf(dex_solana.trades test): bound accepted_range scan to a recent window#9779a-monteiro wants to merge 1 commit into
a-monteiro wants to merge 1 commit into
Conversation
…indow The dbt_utils.accepted_range test on amount_usd full-scanned all history (17.2B rows / 236 GB) on every run. Bound it to the last 7 days via a where config: the test exists to flag outlier amount_usd as data lands, so a recent window matches intent. Cuts the test scan ~17x IO and ~16x CPU.
Member
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
PR SummaryLow Risk Overview The $1b cap is unchanged; only test scope and cost change. Outliers in Reviewed by Cursor Bugbot for commit e723d21. Configure here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

The
dbt_utils.accepted_rangetest ondex_solana.trades.amount_usdran with nowhereconfig, so every run full-scanned the entire table — 17.2B rows / 236.8 GB, ~1.34M cpu_ms, ~62s wall — just to flag trades above $1b. It runs 4 heavy times/day = ~0.95 TB IO and ~1.49 CPU-hrs/day, purely re-validating immutable history.The test exists to catch outlier
amount_usdas data lands, so bounding it to a recent window matches its intent. This adds aconfig.whereofblock_time >= now() - interval '7' day.block_timeis a regular column with Delta min/max stats andblock_monthis the partition key, so the bound prunes to recent files (pushdown confirmed live).Measured on prod data (spellbook-hourly, faithful full-column scan, 3 warm runs, medians):
Per day (4 heavy runs): ~947 → ~55 GB IO and ~1.49 → ~0.10 CPU-hrs.
No equivalence proof here on purpose: this changes the test's coverage from all-history to the recent window. An outlier landing more than 7 days ago (e.g. a late backfill) would no longer be flagged — wanting reviewer sign-off on the window length (happy to bump to 30d for a wider safety net; ~30d still scans far less than full history).
Compiled test confirms the bound wraps the relation:
Fixes CUR2-2800