fix: make EnsembleByKey transformSchema match output by fallintoplace · Pull Request #2575 · microsoft/SynapseML

fallintoplace · 2026-07-26T02:05:54Z

Problem

EnsembleByKey.transformSchema describes the opposite shape from transform:

with collapseGroup=true, transform returns grouping keys and aggregate columns, while transformSchema keeps every input column
with collapseGroup=false, transform joins aggregates back to the input, while transformSchema keeps only fields selected from the aggregation column set

This can break pipeline schema validation because the declared schema differs from the DataFrame produced at runtime. The old implementation also fails when transformSchema is called before transform without explicit colNames, and declares FloatType for scalar means that Spark produces as DoubleType.

Changes

initialize default output names consistently from both transform and transformSchema
derive grouping fields from keys
return grouping fields followed by aggregate fields for collapsed groups
match Spark join ordering for non-collapsed groups: keys, remaining non-overwritten input fields, then aggregates
declare scalar mean outputs as DoubleType for both float and double inputs
assert exact schema equality for double and float inputs in both collapse modes, with distinct key and aggregation columns and no explicit output name

Testing

core/testOnly com.microsoft.azure.synapse.ml.stages.EnsembleByKeySuite (9 tests passed)
core/Test/scalastyle (0 findings)
core/Compile/scalastyle (0 findings)

azure-pipelines · 2026-07-26T02:06:02Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

github-actions · 2026-07-26T02:06:04Z

Hey @fallintoplace 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

fix: Fix LightGBM crashes with empty partitions
feat: Make HTTP on Spark back-offs configurable
docs: Update Spark Serving usage
build: Add codecov support
perf: improve LightGBM memory usage
refactor: make python code generation rely on classes
style: Remove nulls from CNTKModel
test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

Copilot

Pull request overview

This PR updates EnsembleByKey’s transformSchema logic so it matches the actual output schema produced by transform, across both collapseGroup=true and collapseGroup=false modes, and adds a regression test to validate schema consistency.

Changes:

Rebuild transformSchema output fields using keys for grouping fields and consistent output ordering for both collapse modes.
In non-collapsing mode, model Spark join column ordering and drop/replace conflicting output column names in the predicted schema.
Add a test asserting transformSchema(schema) equals transform(df).schema for both collapse modes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
core/src/main/scala/com/microsoft/azure/synapse/ml/stages/EnsembleByKey.scala	Adjusts `transformSchema` field selection and ordering to align with `transform` behavior.
core/src/test/scala/com/microsoft/azure/synapse/ml/stages/EnsembleByKeySuite.scala	Adds schema-equality coverage for both `collapseGroup` modes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ranadeepsingh · 2026-07-31T05:55:02Z

/azp run

azure-pipelines · 2026-07-31T05:55:13Z

Azure Pipelines: Successfully started running 1 pipeline(s).

codecov-commenter · 2026-07-31T06:11:22Z

Codecov Report

❌ Patch coverage is 98.20789% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.00%. Comparing base (617ad0f) to head (a758b11).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...rosoft/azure/synapse/ml/stages/EnsembleByKey.scala	98.20%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2575      +/-   ##
==========================================
+ Coverage   84.78%   85.00%   +0.22%     
==========================================
  Files         334      334              
  Lines       17806    18049     +243     
  Branches     1623     1631       +8     
==========================================
+ Hits        15097    15343     +246     
+ Misses       2709     2706       -3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ranadeepsingh · 2026-07-31T06:31:17Z

/azp run

azure-pipelines · 2026-07-31T06:31:26Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ranadeepsingh · 2026-07-31T08:14:50Z

/azp run

azure-pipelines · 2026-07-31T08:15:03Z

Azure Pipelines: Successfully started running 1 pipeline(s).

ranadeepsingh · 2026-07-31T08:59:11Z

CI follow-up for f4bc2c9: Azure build 229036776 completed with 67 successful checks. The only failing job was Databricks E2E databricks-gpu, which aborted before running tests (Tests: succeeded 0, failed 0) because cluster startup terminated with INSTANCE_POOL_MAX_CAPACITY_REACHED / Instance pool is full. All CPU Databricks jobs, core/stages unit tests, Python/R tests, style, publish, and Fabric E2E passed. This is infrastructure-only, so I am not issuing another /azp run.

ranadeepsingh · 2026-07-31T18:16:04Z

/azp run

azure-pipelines · 2026-07-31T18:16:15Z

Azure Pipelines: Successfully started running 1 pipeline(s).

ranadeepsingh · 2026-07-31T23:11:20Z

/azp run

azure-pipelines · 2026-07-31T23:11:30Z

Azure Pipelines: Successfully started running 1 pipeline(s).

AB#2575 ## Summary Add Round 3 code review artifact for SynapseML PR microsoft#2575 focusing on edge cases and robustness. ## Prompting Intent Perform Round 3 of the DIRECT sequential six-round review gauntlet for SynapseML PR microsoft#2575. Focus on edge cases, robustness, error handling, boundary conditions, concurrency/session behavior, and failure modes. ## Linked Sources - Prompt: C:\Users\singhrana\.copilot\session-state\c2ea157b-c36a-4cbc-896d-e8caa3ea05bd\files\pr-2575\prompts\review-round-3.md - Review artifact: C:\Users\singhrana\Documents\SynapseML-pr-2575\reviews\pr-2575\task-2575-attempt-1-review-3-gemini-3.6-flash.md ## Rationale Recorded findings for edge cases, null join behavior, session case sensitivity, and vector aggregate nullability in the Round 3 review artifact as instructed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Make EnsembleByKey schema resolution match Spark output for collapsed and non-collapsed transforms, including scalar and vector means, nested and qualified keys, duplicate attributes, null keys, overwrite behavior, output naming, metadata, nullability, and unsupported inputs. Add generated-Python compatibility coverage and preserve the complete six-round review audit trail. ## Prompting Intent Take end-to-end ownership of GitHub PR microsoft#2575 in an isolated worktree. Independently verify and harden EnsembleByKey schema behavior, run the required six-round DIRECT sequential multi-model review, fix and rerun every finding until clean, validate Scala and generated-language surfaces, avoid generated target edits and history rewrites, and prepare the existing PR branch for normal push and Azure validation. ## Linked Sources - Pull request: microsoft#2575 - Round 1 review: reviews/pr-2575/task-2575-attempt-1-review-1-claude-opus-5.md - Round 2 review: reviews/pr-2575/task-2575-attempt-1-review-2-gpt-5.6-sol.md - Round 3 review: reviews/pr-2575/task-2575-attempt-1-review-3-gemini-3.6-flash.md - Round 4 review: reviews/pr-2575/task-2575-attempt-1-review-4-claude-opus-5.md - Round 5 review: reviews/pr-2575/task-2575-attempt-1-review-5-gpt-5.6-sol.md - Round 6 review: reviews/pr-2575/task-2575-attempt-1-review-6-gemini-3.6-flash.md ## Rationale Keep business logic Scala-first and derive the Python surface through SynapseML's internal-wrapper override pattern. Resolve configured fields against schema ordinals and analyzed Spark attributes so transformSchema remains optimistic where qualifier or ExprId identity is unavailable while transform rejects genuine runtime ambiguity. Use positional normalization and public DataFrame APIs for Spark 3.5/4.1 compatibility, and use null-safe joins to preserve non-collapsed rows with null grouping keys. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ranadeepsingh · 2026-08-02T09:51:26Z

/azp run

azure-pipelines · 2026-08-02T09:51:37Z

Azure Pipelines: Successfully started running 1 pipeline(s).

## Summary Remove the six generated review audit files from the committed PR contents while leaving all reviewed implementation, documentation, and regression tests unchanged. ## Prompting Intent The engineer asked to prevent the generated review files from being merged. Preserve the existing PR history, avoid force-pushing, and remove only those committed artifacts. ## Linked Sources - Pull request: microsoft#2575 - User request: remove the review files from the committed files ## Rationale Use a follow-up deletion commit rather than rewriting the existing commit so the shared PR branch remains safe and auditable. The review process remains reflected in the conversation and prior commit history, but the final merged tree contains only the product changes and their tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Merge upstream master at `617ad0f8fc8932d93fc6185cb2547e5ab2c2719a` into the existing EnsembleByKey PR branch. ## Prompting Intent The engineer asked to update the published PR branch with the latest master changes. Preserve branch history and use a normal push rather than rebasing or force-pushing. ## Linked Sources - Pull request: microsoft#2575 - Upstream repository: https://github.com/microsoft/SynapseML ## Rationale Use a merge commit because the branch is already published and the engineer selected the history-preserving option. This incorporates current upstream changes without rewriting commits reviewed and validated on the shared PR branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ranadeepsingh · 2026-08-04T07:33:17Z

/azp run

azure-pipelines · 2026-08-04T07:33:28Z

Azure Pipelines: Successfully started running 1 pipeline(s).

ranadeepsingh · 2026-08-04T07:38:26Z

/azp run

azure-pipelines · 2026-08-04T07:38:37Z

Azure Pipelines: Successfully started running 1 pipeline(s).

ranadeepsingh · 2026-08-04T09:19:27Z

/azp run

azure-pipelines · 2026-08-04T09:19:41Z

Azure Pipelines: Successfully started running 1 pipeline(s).

ranadeepsingh · 2026-08-04T10:52:58Z

/azp run

azure-pipelines · 2026-08-04T10:53:09Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot AI review requested due to automatic review settings July 26, 2026 02:05

Copilot started reviewing on behalf of fallintoplace July 26, 2026 02:06 View session

Copilot AI reviewed Jul 26, 2026

View reviewed changes

Comment thread core/src/main/scala/com/microsoft/azure/synapse/ml/stages/EnsembleByKey.scala Outdated

Comment thread core/src/test/scala/com/microsoft/azure/synapse/ml/stages/EnsembleByKeySuite.scala Outdated

fallintoplace force-pushed the fix/ensemble-by-key-schema branch from 32aee3b to 0f15e8c Compare July 26, 2026 02:09

fallintoplace changed the title ~~fix: align EnsembleByKey transform schema~~ fix: make EnsembleByKey transformSchema match output Jul 26, 2026

fix: align EnsembleByKey transform schema

bdc39f9

fallintoplace force-pushed the fix/ensemble-by-key-schema branch from 0f15e8c to bdc39f9 Compare July 26, 2026 02:11

ranadeepsingh and others added 2 commits July 30, 2026 21:47

Merge branch 'master' into fix/ensemble-by-key-schema

5e6f164

fix: align EnsembleByKey vector schema

222d99e

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

test: cover unsupported EnsembleByKey inputs

f4bc2c9

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge branch 'master' into fix/ensemble-by-key-schema

e49df02

ranadeepsingh and others added 3 commits July 31, 2026 20:58

Merge branch 'master' into fix/ensemble-by-key-schema

3098665

ranadeepsingh and others added 2 commits August 4, 2026 00:27

Uh oh!

Conversation

fallintoplace commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

Testing

Uh oh!

azure-pipelines Bot commented Jul 26, 2026

Uh oh!

github-actions Bot commented Jul 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

ranadeepsingh commented Jul 31, 2026

Uh oh!

azure-pipelines Bot commented Jul 31, 2026

Uh oh!

codecov-commenter commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ranadeepsingh commented Jul 31, 2026

Uh oh!

azure-pipelines Bot commented Jul 31, 2026

Uh oh!

ranadeepsingh commented Jul 31, 2026

Uh oh!

azure-pipelines Bot commented Jul 31, 2026

Uh oh!

ranadeepsingh commented Jul 31, 2026

Uh oh!

ranadeepsingh commented Jul 31, 2026

Uh oh!

azure-pipelines Bot commented Jul 31, 2026

Uh oh!

ranadeepsingh commented Jul 31, 2026

Uh oh!

azure-pipelines Bot commented Jul 31, 2026

Uh oh!

ranadeepsingh commented Aug 2, 2026

Uh oh!

azure-pipelines Bot commented Aug 2, 2026

Uh oh!

ranadeepsingh commented Aug 4, 2026

Uh oh!

azure-pipelines Bot commented Aug 4, 2026

Uh oh!

ranadeepsingh commented Aug 4, 2026

Uh oh!

azure-pipelines Bot commented Aug 4, 2026

Uh oh!

ranadeepsingh commented Aug 4, 2026

Uh oh!

azure-pipelines Bot commented Aug 4, 2026

Uh oh!

ranadeepsingh commented Aug 4, 2026

Uh oh!

azure-pipelines Bot commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fallintoplace commented Jul 26, 2026 •

edited

Loading

codecov-commenter commented Jul 31, 2026 •

edited

Loading