refactor(proto): migrate scalar subquery serde by Phoenix500526 · Pull Request #23915 · apache/datafusion

Phoenix500526 · 2026-07-27T05:10:27Z

Which issue does this PR close?

Closes Proto: migrate ScalarSubqueryExec (adds a subquery-results scope to the decode ctx) #23515.

Rationale for this change

ScalarSubqueryExpr nodes must share the results container populated by their enclosing ScalarSubqueryExec. A normal child decode loses that scope when serialization moves into the plan implementation.

Add a scoped child decode operation so the new plan hooks preserve the shared container while removing the central serialization path.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

ScalarSubqueryExpr nodes must share the results container populated by their enclosing ScalarSubqueryExec. A normal child decode loses that scope when serialization moves into the plan implementation. Add a scoped child decode operation so the new plan hooks preserve the shared container while removing the central serialization path. Closes apache#23515 Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

github-actions · 2026-07-27T05:14:46Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  37.668s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.140s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  38.173s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.141s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.650s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure trait_method_added: pub trait method added ---

Description:
A non-sealed public trait added a new method without a default implementation, which breaks downstream implementations of the trait
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-new-item-no-default
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/trait_method_added.ron

Failed in:
  trait method datafusion_physical_plan::proto::ExecutionPlanDecode::decode_plan_with_scalar_subquery_results in file /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/proto.rs:108

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  78.025s] datafusion-physical-plan
    Building datafusion-proto v54.1.0 (current)
       Built [  59.774s] (current)
     Parsing datafusion-proto v54.1.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-proto v54.1.0 (baseline)
       Built [  59.824s] (baseline)
     Parsing datafusion-proto v54.1.0 (baseline)
      Parsed [   0.020s] (baseline)
    Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.229s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure trait_method_marked_deprecated: trait method #[deprecated] added ---

Description:
A trait method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/trait_method_marked_deprecated.ron

Failed in:
  method try_into_scalar_subquery_physical_plan in trait datafusion_proto::physical_plan::PhysicalPlanNodeExt in /home/runner/work/datafusion/datafusion/datafusion/proto/src/physical_plan/mod.rs:664
  method try_from_scalar_subquery_exec in trait datafusion_proto::physical_plan::PhysicalPlanNodeExt in /home/runner/work/datafusion/datafusion/datafusion/proto/src/physical_plan/mod.rs:664

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [ 121.351s] datafusion-proto

codecov-commenter · 2026-07-27T05:37:36Z

Codecov Report

❌ Patch coverage is 67.56757% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.65%. Comparing base (e8a65f2) to head (6f12a16).

Files with missing lines	Patch %	Lines
datafusion/proto/src/physical_plan/mod.rs	38.46%	16 Missing ⚠️
datafusion/physical-plan/src/scalar_subquery.rs	80.00%	2 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23915      +/-   ##
==========================================
- Coverage   80.66%   80.65%   -0.02%     
==========================================
  Files        1095     1095              
  Lines      372294   372319      +25     
  Branches   372294   372319      +25     
==========================================
- Hits       300324   300296      -28     
- Misses      54055    54101      +46     
- Partials    17915    17922       +7

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

adriangb · 2026-07-30T12:33:39Z


+    /// Deserialize a child plan with `results` active for scalar subquery
+    /// expressions in that plan's subtree.
+    fn decode_plan_with_scalar_subquery_results(


It'd be nice to seal this trait. I'll make a PR.

We can't seal it, but I opened #24001. Should not block this PR.

…API (apache#24001) ## Which issue does this PR close? - Part of apache#23494. Precursor to apache#23915 (and to the remaining `DataSource` / `DataSink` items, apache#23497 / apache#23498). ## Rationale for this change `ExecutionPlanEncode` / `ExecutionPlanDecode` (added in apache#23495) are dispatch details of the `try_to_proto` / `try_from_proto` hooks, not extension points. They are defined in `datafusion-physical-plan`, implemented only by `ConverterPlanEncoder` / `ConverterPlanDecoder` in `datafusion-proto`, and plans reach them exclusively through `ExecutionPlanEncodeCtx` / `ExecutionPlanDecodeCtx` — their own docs already say so: > Internal dispatch trait backing [`ExecutionPlanEncodeCtx`]. Implemented by `datafusion-proto`. > Plan authors never name this trait. They are `pub` only because those adapters live in another crate. Nothing said that to the tooling, so every capability the epic still has to add to the ctx reads as a major breaking change. apache#23915 hits this first: it needs `decode_plan_with_scalar_subquery_results` so `ScalarSubqueryExec` can decode its input with the subquery-results container in scope, and `cargo-semver-checks` flags the required method as `trait_method_added`. The `DataSource` / `DataSink` families will want their own primitives next. `#[doc(hidden)]` states what was already true, and makes those additions changes to something that was never public API — rather than asking each follow-up PR to explain away a breakage report. ## What changes are included in this PR? Mark `ExecutionPlanEncode` and `ExecutionPlanDecode` `#[doc(hidden)]`, and say why in their docs and in the module overview. One file, no behavior, wire-format, or signature changes. A sealed-trait supertrait was the first thing I tried. It is worse here: sealing across a crate boundary needs a `pub` marker anyway, so it does not actually prevent a downstream impl — it just adds a public item in order to say "this is not public API", plus an `impl` line for every implementor including test doubles. `#[doc(hidden)]` says the same thing by removing API surface instead of adding it, and `cargo-semver-checks` honors both identically (measured below). ## Are these changes tested? The property this PR buys is a `cargo-semver-checks` classification, so it is verified with that tool directly — v0.49.0, the version CI installs, invoked the way CI invokes it: | baseline | change under test | result | |---|---|---| | `main` | add a required method to `ExecutionPlanDecode` | `trait_method_added` — **major** | | this PR | the same required method | 223 checks pass, **no semver update required** | Worth recording, since it drove the shape of this PR: the lint reads `public_api_sealed` from the **baseline**, so this has to land before the PRs that add methods, not alongside them. Also, `cargo-semver-checks` does not re-qualify the traits as public API even though `ExecutionPlanEncodeCtx::new` still names them in a public signature. Also ran, on the pinned 1.97.0 toolchain: `cargo fmt --all -- --check`, `cargo clippy --all-targets --all-features` for both crates, `cargo doc` with `-D warnings` (no broken intra-doc links from the module docs to the now-hidden traits), and the full `datafusion-proto` integration suite (211 passed). ## Are there any user-facing changes? `cargo-semver-checks` will report `trait_now_doc_hidden` on this PR, and that is the intended content of the change. Real-world impact is nil: both traits were introduced by apache#23495, which merged after `54.1.0` was tagged, so no released version of `datafusion-physical-plan` contains them. The load-bearing public API — `ExecutionPlan::try_to_proto`, the two ctx types, `AsExecutionPlan`, `PhysicalExtensionCodec`, `PhysicalProtoConverterExtension` — is untouched, and `datafusion-proto` needs no change at all. Note for contributors on apache#23494: the expression-side equivalents (`PhysicalExprEncode` / `PhysicalExprDecode` in `physical-expr-common`) have the same shape and the same argument, but several test doubles across crates implement them. Left for a follow-up. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

github-actions Bot added proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 27, 2026

github-actions Bot added the auto detected api change Auto detected API change label Jul 27, 2026

adriangb approved these changes Jul 30, 2026

View reviewed changes

adriangb mentioned this pull request Jul 30, 2026

refactor: mark the ExecutionPlan proto dispatch traits as non-public API #24001

Merged

adriangb enabled auto-merge July 30, 2026 14:30

adriangb added this pull request to the merge queue Jul 30, 2026

Merged via the queue into apache:main with commit 3aef16c Jul 30, 2026
64 of 65 checks passed

Phoenix500526 deleted the issue/23515 branch July 31, 2026 00:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor(proto): migrate scalar subquery serde - #23915

refactor(proto): migrate scalar subquery serde#23915
adriangb merged 1 commit into
apache:mainfrom
Phoenix500526:issue/23515

Phoenix500526 commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

codecov-commenter commented Jul 27, 2026

Uh oh!

Uh oh!

adriangb Jul 30, 2026

Uh oh!

adriangb Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Phoenix500526 commented Jul 27, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

codecov-commenter commented Jul 27, 2026

Codecov Report

Uh oh!

Uh oh!

adriangb Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants