fix: skip VW barrier execution for single-partition training - #2592
fix: skip VW barrier execution for single-partition training#2592ranadeepsingh wants to merge 1 commit into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Hey @ranadeepsingh 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
This PR adjusts Vowpal Wabbit training to avoid Spark barrier execution when training runs with a single partition, while preserving the existing behavior for multi-partition training (both barrier-enabled and barrier-disabled). This improves compatibility with single-partition workloads by preventing unnecessary barrier scheduling.
Changes:
- Guard barrier execution in
VowpalWabbitBaseLearnerso it only runs when barrier mode is enabled and the training DataFrame has more than one partition. - Add execution-path tests that verify barrier usage (enabled multi-partition), non-barrier usage (disabled multi-partition), and skipping barrier usage (enabled single-partition) via a tracking subclass and accumulator.
Show a summary per file
| File | Description |
|---|---|
| vw/src/main/scala/com/microsoft/azure/synapse/ml/vw/VowpalWabbitBaseLearner.scala | Adds a partition-count guard to skip barrier execution for single-partition training. |
| vw/src/test/scala/com/microsoft/azure/synapse/ml/vw/VerifyVowpalWabbitClassifier.scala | Adds tests that assert which execution path was used (barrier vs non-barrier) across partition and configuration combinations. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Lite
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## master #2592 +/- ##
==========================================
- Coverage 84.79% 78.33% -6.46%
==========================================
Files 334 334
Lines 17806 17806
Branches 1623 1608 -15
==========================================
- Hits 15099 13949 -1150
- Misses 2707 3857 +1150 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
## Summary Use barrier execution only when VowpalWabbit training enables it and the prepared dataset has more than one partition. Add Spark-stage execution-path assertions for enabled multi-partition, disabled multi-partition, and enabled single-partition training. ## Prompting Intent Revalidate the current VowpalWabbitBaseLearner behavior, restore the minimal source-compatible rule proposed by the ancient PR, prove it through TDD and targeted VW validation, and keep the change isolated from LightGBM. ## Linked Sources - Original proposal: microsoft#1912 - Initial CI failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229206650 - Repository review guidance: .github/skills/code-review/SKILL.md - Local validation guidance: .github/skills/synapseml-local-setup/SKILL.md ## Rationale Keep the public API and multi-partition synchronization behavior unchanged with a short-circuit partition-count guard. Observe Spark stage metadata rather than adding a test-only PipelineStage subclass, and use a marker job to drain asynchronous listener events deterministically; this proves the selected execution path without entering global stage discovery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
91c7c96 to
71a0815
Compare
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Summary
Validation
BarrierTaskContextbefore the source guardvw/testOnly com.microsoft.azure.synapse.ml.vw.VerifyVowpalWabbitClassifier -- -z skipsvw/testOnly com.microsoft.azure.synapse.ml.vw.VerifyVowpalWabbitClassifier -- -z libsvmvw/compileandvw/Test/compilescalastyle test:scalastyleCompatibility
The production change is a source- and binary-compatible execution guard; no public API changes are introduced. This change is scoped only to Vowpal Wabbit and does not alter LightGBM.
Related
Recreates the still-relevant proposal from #1912 on current
masterwithout closing or modifying that original PR.