Allow constant expressions in date_part - #23997
Draft
Adam-Alani wants to merge 2 commits into
Draft
Conversation
`date_part` previously returned `Int32` for most parts (year, month, day, hour, minute, second, etc.), `Int64` for `nanosecond`, and `Float64` for `epoch`. This is inconsistent and disagrees with PostgreSQL, which is defined to return `double precision` for `date_part` regardless of the part being extracted: https://www.postgresql.org/docs/current/functions-datetime.html Change `DatePartFunc::return_field_from_args` to always advertise `Float64`, and cast the kernel result to `Float64` at the end of `invoke_with_args` (the underlying arrow kernels still produce `Int32`/`Int64` for non-epoch parts; this is a no-op cast for `epoch`). `preimage` is updated to also accept `Float64` literals so `date_part('year', col) = 2024` still gets pushed down as a date range after type coercion turns the `2024` literal into `Float64`. The Spark `date_part` wrapper now wraps the simplified DataFusion call in a cast back to `Int32` so that Spark-side behavior (and its existing sqllogictests) are preserved. Tests and docs updated: - `datafusion/sqllogictest/test_files/datetime/date_part.slt`, `datetime/timestamps.slt`, `clickbench.slt`, `group_by.slt` switched from `query I[I...]` to `query R[R...]` for `date_part` outputs (and `arrow_typeof` results updated from `Int32`/`Int64` to `Float64`). - `docs/source/user-guide/sql/scalar_functions.md` and `docs/source/user-guide/expressions.md` regenerated.
Defer first-argument value validation until execution now that the return type is value-independent, allowing casts and other foldable constants.
github-actions
Bot
added
documentation
Adam-Alani
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
date_partcurrently determines its return field before constant folding runs, so a valid constant expression such asCAST('epoch' AS VARCHAR)is rejected because it is not a direct literal. This change is stacked on #22851, which makes the function's return type value-independent and PostgreSQL-compatible.What
This allows foldable constant expressions in the first argument of
date_partwhile preserving runtime validation that the evaluated value is a non-null scalar string. It adds regression coverage for a castedepochargument and updates type expectations added since #22851 was opened.How
Return-field inference now uses only the temporal argument's nullability and always reports
Float64; it no longer inspects the unsimplified first argument. The normal expression simplifier can therefore fold casts and other constant expressions before execution, where the existing scalar-value validation still applies.Validated with:
cargo test --test sqllogictests -p datafusion-sqllogictest -- datetime/date_part.sltDepends on #22851.