refactor: mark the ExecutionPlan proto dispatch traits as non-public API

[adriangb]

Which issue does this PR close?

• Part of [EPIC] Port ExecutionPlan serialization to try_to_proto / try_from_proto hooks #23494. Precursor to refactor(proto): migrate scalar subquery serde #23915 (and to the remaining DataSource / DataSink items, Proto: add DataSource/FileSource try_to_proto hook + port FileScanConfig serde #23497 / Proto: add DataSink try_to_proto hook + port FileSinkConfig serde #23498).

Rationale for this change

ExecutionPlanEncode / ExecutionPlanDecode (added in #23495) are dispatch details of the
try_to_proto / try_from_proto hooks, not extension points. They are defined in
datafusion-physical-plan, implemented only by ConverterPlanEncoder / ConverterPlanDecoder
in datafusion-proto, and plans reach them exclusively through ExecutionPlanEncodeCtx /
ExecutionPlanDecodeCtx — their own docs already say so:

Internal dispatch trait backing [ExecutionPlanEncodeCtx]. Implemented by datafusion-proto.
Plan authors never name this trait.

They are pub only because those adapters live in another crate. Nothing said that to the tooling,
so every capability the epic still has to add to the ctx reads as a major breaking change. #23915
hits this first: it needs decode_plan_with_scalar_subquery_results so ScalarSubqueryExec can
decode its input with the subquery-results container in scope, and cargo-semver-checks flags the
required method as trait_method_added. The DataSource / DataSink families will want their own
primitives next.

#[doc(hidden)] states what was already true, and makes those additions changes to something that
was never public API — rather than asking each follow-up PR to explain away a breakage report.

What changes are included in this PR?

Mark ExecutionPlanEncode and ExecutionPlanDecode #[doc(hidden)], and say why in their docs
and in the module overview. One file, no behavior, wire-format, or signature changes.

A sealed-trait supertrait was the first thing I tried. It is worse here: sealing across a crate
boundary needs a pub marker anyway, so it does not actually prevent a downstream impl — it just
adds a public item in order to say "this is not public API", plus an impl line for every
implementor including test doubles. #[doc(hidden)] says the same thing by removing API surface
instead of adding it, and cargo-semver-checks honors both identically (measured below).

Are these changes tested?

The property this PR buys is a cargo-semver-checks classification, so it is verified with that
tool directly — v0.49.0, the version CI installs, invoked the way CI invokes it:

baseline	change under test	result
main	add a required method to ExecutionPlanDecode	trait_method_added — major
this PR	the same required method	223 checks pass, no semver update required

Worth recording, since it drove the shape of this PR: the lint reads public_api_sealed from the
baseline, so this has to land before the PRs that add methods, not alongside them. Also,
cargo-semver-checks does not re-qualify the traits as public API even though
ExecutionPlanEncodeCtx::new still names them in a public signature.

Also ran, on the pinned 1.97.0 toolchain: cargo fmt --all -- --check, cargo clippy --all-targets --all-features for both crates, cargo doc with -D warnings (no broken intra-doc links from the
module docs to the now-hidden traits), and the full datafusion-proto integration suite
(211 passed).

Are there any user-facing changes?

cargo-semver-checks will report trait_now_doc_hidden on this PR, and that is the intended
content of the change.

Real-world impact is nil: both traits were introduced by #23495, which merged after 54.1.0 was
tagged, so no released version of datafusion-physical-plan contains them. The load-bearing public
API — ExecutionPlan::try_to_proto, the two ctx types, AsExecutionPlan, PhysicalExtensionCodec,
PhysicalProtoConverterExtension — is untouched, and datafusion-proto needs no change at all.

Note for contributors on #23494: the expression-side equivalents (PhysicalExprEncode /
PhysicalExprDecode in physical-expr-common) have the same shape and the same argument, but
several test doubles across crates implement them. Left for a follow-up.

[adriangb]

@kumarUjjawal small change I found while working on our EPIC, please take a look when you can 🙏🏻

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  38.634s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.140s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  38.003s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.140s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.608s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure trait_now_doc_hidden: pub trait is now #[doc(hidden)] ---

Description:
A pub trait is now #[doc(hidden)], removing it from the crate's public API.
        ref: https://doc.rust-lang.org/rustdoc/write-documentation/the-doc-attribute.html#hidden
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/trait_now_doc_hidden.ron

Failed in:
  trait ExecutionPlanEncode in file /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/proto.rs:81
  trait ExecutionPlanDecode in file /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/proto.rs:111

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  78.979s] datafusion-physical-plan

[kumarUjjawal]

Thanks @adriangb

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.85%. Comparing base (2f25454) to head (e36e9b6).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24001      +/-   ##
==========================================
+ Coverage   80.75%   80.85%   +0.09%     
==========================================
  Files        1096     1096              
  Lines      373588   373911     +323     
  Branches   373588   373911     +323     
==========================================
+ Hits       301687   302311     +624     
+ Misses      53898    53557     -341     
- Partials    18003    18043      +40     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.