fix: TopK aggregation drops groups whose MIN/MAX value is NULL - #23684
fix: TopK aggregation drops groups whose MIN/MAX value is NULL#23684u70b3 wants to merge 1 commit into
Conversation
9238ac1 to
a7f380c
Compare
a7f380c to
3ee7c2f
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #23684 +/- ##
==========================================
+ Coverage 80.77% 80.84% +0.07%
==========================================
Files 1096 1096
Lines 373769 374207 +438
Branches 373769 374207 +438
==========================================
+ Hits 301909 302536 +627
+ Misses 53850 53588 -262
- Partials 18010 18083 +73 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
3ee7c2f to
e0370ea
Compare
|
Hi @kosiew — this PR has been open for ~2 weeks with CI fully green (40/40). Since you've reviewed the recent TopK changes (#23091, #23096), would you mind taking a look when you have a moment? It's a correctness fix for TopK aggregation silently dropping groups whose MIN/MAX value is NULL (#23440, #22190). Happy to address any feedback. Thanks! |
e0370ea to
c5bb62e
Compare
|
run benchmark topk_aggregate |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing fix/grouped-topk-null-aggregate-values (c5bb62e) to 2f25454 (merge-base) diff using: topk_aggregate File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetopk_aggregate — base (merge-base)
topk_aggregate — branch
File an issue against this benchmark runner |
|
run benchmark topk_aggregate |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing fix/grouped-topk-null-aggregate-values (c5bb62e) to 2f25454 (merge-base) diff using: topk_aggregate File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetopk_aggregate — base (merge-base)
topk_aggregate — branch
File an issue against this benchmark runner |
|
|
||
| # issue #22190: single all-NULL group, MIN ASC NULLS LAST LIMIT 20 | ||
| query R | ||
| SELECT min_y FROM (SELECT s, MIN(y) AS min_y FROM t0 GROUP BY s) ORDER BY min_y ASC NULLS LAST LIMIT 20; |
There was a problem hiding this comment.
The NULLS LAST all-NULL regression should still exercise bounded TopK aggregation, but this test currently checks only the final result. Could we add a nearby EXPLAIN assertion that verifies AggregateExec ... lim=[...] for one NULLS LAST all-NULL or mixed case? That would help ensure this coverage does not accidentally pass through the regular aggregation fallback.
There was a problem hiding this comment.
Added a nearby EXPLAIN assertion for the NULLS LAST all-NULL case, verifying that the physical plan retains AggregateExec with lim=[20]. Thanks!
Grouped TopK aggregation previously dropped groups whose MIN/MAX inputs were all NULL. Track bounded all-NULL candidates alongside valued groups and emit them for the final sort to rank and truncate. Handle NULL-to-value transitions, fully reuse freed hash-table slots, and keep NULL bookkeeping off the common no-NULL hot path. Nullable MIN/MAX with NULLS FIRST is not monotonic for bounded aggregation: a group can move from NULL to a worse non-NULL rank. Skip the TopK pushdown in that case to preserve exact results while retaining it for NULLS LAST and non-nullable inputs. Closes apache#23440, closes apache#22190.
c5bb62e to
99df683
Compare
Which issue does this PR close?
Rationale for this change
When
TopKAggregationpushes aLIMITinto a MIN/MAX aggregate, a group whose aggregate inputs are all NULL can disappear instead of being returned with a NULL aggregate value.The stream previously skipped NULL aggregate inputs without registering the group. This is correct for an individual MIN/MAX input, but not for a group whose inputs are all NULL.
There is also an important correctness boundary: nullable MIN/MAX with
NULLS FIRSTis not monotonic for a bounded aggregation. A group can start at NULL, later become non-NULL, and thereby move to a worse rank. Keeping onlylimitNULL candidates can therefore discard a group that belongs in the final result.What changes are included?
limitall-NULL candidates alongside valued TopK groups and emit them for the parent sort to rank and truncate.NULLS FIRST; regular aggregation is used for exact results. TopK remains enabled forNULLS LAST, non-nullable MIN/MAX inputs, and GROUP BY-only/DISTINCT queries.Are these changes tested?
Yes. The following passed on the final commit:
cargo fmt --all -- --checkcargo clippy --all-targets --all-features -- -D warningsaggregates_topk.sltand affectedgroup_by.slttestsavro,json,backtrace,extended_tests,recursive_protection,parquet_encryption, including all 495 sqllogic files and extended/fuzz suitesPerformance
The
topk_aggregate10-million-row time-series benchmark exposed an initial ~5.5% regression from checking NULL state on every row. Moving NULL handling to a batch-selected slow path removed the measurable regression.Final 30-sample 95% intervals on the same machine and settings:
main: 25.630–26.436 msAre there any user-facing changes?
Queries that previously dropped all-NULL groups under
ORDER BY <min/max> ... LIMITnow return correct SQL results. Nullable MIN/MAX queries usingNULLS FIRSTmay use regular aggregation rather than the bounded TopK optimization to guarantee correctness. There are no API or configuration changes.