ci: gate Databricks CPU and GPU tests independently - #2582
Merged
ranadeepsingh merged 2 commits intoAug 4, 2026
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Hey @ranadeepsingh 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
ranadeepsingh
changed the title
This was referenced Aug 1, 2026
## Summary Classify changed paths against the actual Databricks CPU and GPU runtime surfaces, emit separate fail-open decisions, and gate each matrix leg independently. ## Prompting Intent The engineer asked to determine exactly when Databricks tests should run, lock down the path rules, and deliver the work as a stacked pull request above PR #2581. ## Linked Sources - Base CI hardening PR: #2581 - GitHub stacked PR documentation: https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-stacked-pull-requests - ADO timing audit: build 229176406 ## Rationale CPU and GPU decisions are separated because most module changes cannot affect the expensive GPU notebooks. Unknown paths and shared build or test infrastructure remain fail-open, while explicit test-only and unrelated tooling paths skip safely. This preserves coverage while avoiding unrelated GPU capacity waits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ranadeepsingh
force-pushed
the
ci/databricks-impact-gating
branch
from
August 1, 2026 10:50
ee4b3f1 to
6a23382
Compare
This was referenced Aug 1, 2026
Collaborator
Author
|
Status update: #2588 merged as 27bfeefd. It adds the T4 local RAG notebook to the active Databricks GPU suite and updates the shared GPU library pins. This does not supersede the independent CPU/GPU gating work here. When this stack is rebased onto master, please ensure the impact classifier continues treating changes to the new GPU notebook, its suite registration, and GPU dependency pins as GPU-affecting paths. |
* Fix and streamline release branch compatibility checks ## Summary Run release compatibility checks for both GitHub target-branch formats and replace redundant compile, setup, credential, and per-package SBT tasks with one cached, project-scoped validation process. ## Prompting Intent The engineer asked to fix the silently skipped ReleaseBranchCompat job and simplify it before enabling it so the check is both reliable and efficient. ## Linked Sources - Base CI hardening PR: #2581 - Evidence build with skipped phase: ADO build 229176406 - Parent stack layer: ci/databricks-impact-gating ## Rationale The target condition accepts both values observed across Azure Repos and GitHub PR providers. A single SBT process retains full test compilation and the intended core, VW, and OpenCV compatibility suites while removing repeated build loading, root-wide IntelliJ setup, unnecessary Key Vault access, and Azure CLI authentication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: use Entra authentication for ACR cleanup (#2584) * Use Entra authentication for ACR cleanup Make the weekly ACR cleanup schedule-only, switch it to the dedicated cleanup service connection, replace storage connection-string authentication with Azure CLI Entra authentication, and add fail-safe cleanup tests. The engineer asked to repair the weekly cleanup failures caused by disabled key-based storage authentication, use the declared least-privileged identity, and prevent accidental CI or PR execution. - Failed scheduled build: ADO build 228250033 - Base CI hardening PR: #2581 - Azure CLI pipeline-run reference: https://learn.microsoft.com/en-us/cli/azure/acr/pipeline-run - Parent stack layer: ci/release-branch-compat Using az storage blob exists with auth-mode login keeps all operations inside the AzureCLI task identity and removes runtime SDK installation, Key Vault access, and storage keys. Images are deleted only after the archive is confirmed, and subprocess argument lists avoid shell interpolation of registry-controlled names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant CI authentication and Conda work (#2585) ## Summary Run non-Azure setup and coverage commands as Bash steps, install pinned Black without restoring the 8.6 GB Conda environment, and remove the ineffective standalone Conda cache consumer. ## Prompting Intent The engineer asked for additional improvements that should ship with the requested CI fixes to make builds faster and more reliable without broad behavioral changes. ## Linked Sources - CI efficiency audit from ADO build 229176406 - Base CI hardening PR: #2581 - Parent stack layer: ci/fix-acr-cleanup-auth ## Rationale AzureCLI tasks create an isolated login for every invocation, so setup and coverage steps that never call az gain no authentication benefit. The Style job only needs pinned Black, not the full cached environment. The standalone Conda job was not a dependency and therefore could not prewarm consumers or prevent cold-cache fan-out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ranadeepsingh
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Aug 4, 2026
## Summary Merge the reviewed microsoft#2582 squash commit back into microsoft#2581 after a concurrent master refresh displaced the child merge, restoring the cumulative microsoft#2582- microsoft#2585 pipeline changes on top of current master. ## Prompting Intent The engineer asked to diagnose the failing tests on microsoft#2581 while completing, reviewing, and fully validating the stacked CI pull requests without losing work already merged through the stack. ## Linked Sources - Parent PR: microsoft#2581 - Databricks gating PR: microsoft#2582 - Release compatibility PR: microsoft#2583 - ACR authentication PR: microsoft#2584 - CI setup cleanup PR: microsoft#2585 - Failed Fabric build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229191951 ## Rationale The old red build was caused by a transient DNS failure resolving the Fabric service endpoint, not by the CI changes. During the investigation, a branch refresh raced with the stacked merges and moved microsoft#2581 past microsoft#2582's squash commit. Merging the official squash commit preserves the exact reviewed child tree and its history instead of reconstructing or duplicating those changes, while retaining the latest master updates already present on the parent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues/PRs
rana/harden-sbt-maven-429)PR #2581 is cross-fork, so GitHub native stacks cannot include it directly. This PR is based on an upstream mirror of its exact head commit.
What changes are proposed?
runDatabricksCpuE2EandrunDatabricksGpuE2Edecisions.This prevents CPU-only changes such as LightGBM work from waiting for scarce GPU capacity or executing unrelated GPU fine-tuning notebooks.
ADO validation
true.pipeline.yamlchanges are fail-open. Future CPU-only/GPU-only PRs use the independent path decisions.How is this patch tested?
Dependencies / feature surface