ci: land sbt bootstrap and efficient CI fixes by ranadeepsingh · Pull Request #2581 · microsoft/SynapseML

ranadeepsingh · 2026-07-31T23:37:34Z

Integration PR for the completed CI stack

This is the final PR that lands the completed stacked work on master.
The child PRs were merged into their parent feature branches, not directly into
master, so their changes do not ship unless #2581 is merged.

PR	Change now included in this branch	Squash commit
#2582	Gate Databricks CPU and GPU E2E independently	`124d5210af`
#2583	Fix and streamline release-branch compatibility validation	`796af25657`
#2584	Use Entra authentication and safer archive-before-delete ACR cleanup	`121a444e3b`
#2585	Remove redundant Azure authentication and Conda setup	`01f00f6411`

The cumulative child tree is restored in
ac80bf32b6
after a concurrent branch refresh briefly displaced #2582's squash commit.

Intent

This PR consolidates the CI reliability and efficiency work into one merge:

Prevent the hosted-agent sbt/Maven thundering herd that caused HTTP 429
failures before tests started.
Restore and share the sbt launcher, Ivy, and Coursier caches through a
mandatory prewarm gate.
Run Databricks CPU and GPU suites only for PR changes that can affect them,
while remaining fail-open for unknown or shared-impact changes.
Validate release branches with one focused rebase/compile/test process.
Remove AzureCLI and Conda setup from jobs that do not need credentials or
the full SynapseML environment.
Move scheduled ACR cleanup to Entra authentication and defer deletion until
an exported backup is confirmed.
Allow the two mandatory Docker image builds enough time to finish under
current hosted-agent and package-download latency.

Does this actually save time?

Yes for clearly non-impacting and suite-specific PRs. It is primarily an
agent-capacity saving, with a smaller developer wall-clock saving because the
matrix jobs run in parallel.

Measured from the final green Azure build
229602106:

Work avoided when safely gated	Measured agent time
Five Databricks CPU jobs	142m 31s
Databricks GPU job	69m 37s
Both CPU and GPU suites	212m 8s (3h 32m)

For that run:

Total pipeline wall time was 75m 31s.
Databricks GPU was the critical path and finished at 69m 37s.
The next-longest required job, BuildDocker, finished at 58m 41s.
A clearly non-impacting PR that skips both Databricks suites would therefore
have finished about 11 minutes earlier, while consuming about
3.5 fewer hosted-agent hours.
A CPU-only change saves the 69m 37s GPU job and about 11 minutes of wall time.
A GPU-only change saves 142m 31s of CPU agent time, although wall time remains
GPU-bound.

The mandatory sbt prewarm took 3m 20s in the same run. It is intentionally
a reliability trade-off rather than a standalone speed claim: one controlled
dependency resolution happens before fan-out instead of roughly 30 fresh
agents concurrently hitting Maven Central and failing with rate limits.

This PR itself touches pipeline.yaml and templates, so the detector correctly
runs both Databricks suites. The savings apply after merge to PRs whose paths
are demonstrably non-impacting or affect only one suite.

Conservative gating

The detector skips only known-safe changes such as GitHub metadata, website
files, .pipelines/, ACR/CI tooling, non-notebook documentation, and ordinary
module test sources outside shared Databricks infrastructure.

It remains fail-open for runtime code, build definitions, pipeline.yaml,
templates, shared Databricks utilities, unknown paths, empty diffs, and
detection/fetch failures. Scheduled, master, tag, and manual builds continue
to run both suites.

Reliability fixes

BuildAndCacheSbt must succeed before sbt-running jobs fan out.
Cache-service failures fall back safely, but failed prewarming blocks the
fan-out rather than recreating the rate-limit stampede.
tools/ci/sbt_retry.sh provides bounded attempts, timeouts, jitter, backoff,
argument forwarding, and visible final failures.
BuildDocker retains both image validations and now has a 120-minute job
budget. Three recent master builds and the first integrated PR run exceeded
Azure Pipelines' default 60-minute limit; no Docker validation was removed.
ACR cleanup is schedule-only and uses the synapseml-clean-acr service
connection. That identity must retain blob read/exists, ACR list/delete, and
export pipeline-run creation permissions.

Validation

Final head: fda789bad0

Azure build
229602106
passed in full, including:

sbt prewarm and cache restoration
both Docker image builds
Fabric E2E
all five Databricks CPU jobs and Databricks GPU
Spark 3.5 and Spark 4.1 release compatibility
publish, website, style, unit, Python, and R jobs

Targeted integration validation also passed with 36 CI/ACR tests, Black 22.3.0,
Scala style checks, YAML parsing, and git diff --check.

Scope

CI, release validation, and scheduled ACR maintenance only. No SynapseML
runtime, model, notebook, or public API behavior is changed by this PR.

## Summary Require lower-is-better validation metrics to improve by more than improvementTolerance before resetting the early-stopping counter. Clarify the parameter documentation and add focused regression coverage for both metric directions and zero tolerance. ## Prompting Intent Investigate GitHub issue microsoft#2565 from a new branch based on master, determine whether the report is valid, and implement a complete fix suitable for an upstream SynapseML pull request. ## Linked Sources - GitHub issue: microsoft#2565 ## Rationale The existing higher-is-better comparison already treats improvementTolerance as a minimum delta, while lower-is-better metrics accepted small regressions. A package-internal comparison helper makes the intended symmetric behavior directly testable without adding a slow native LightGBM fixture or changing public APIs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Expand improvement-tolerance coverage across representative LightGBM metrics and tolerance values. Preserve disabled early stopping when earlyStoppingRound is zero, validate both early-stopping parameters, and document their accepted ranges. ## Prompting Intent The engineer requested broader parameter testing to ensure the issue microsoft#2565 fix does not introduce downstream regressions. Cover related defaults, boundaries, metric families, invalid values, and early-stopping-round interactions before updating the pull request. ## Linked Sources - GitHub issue: microsoft#2565 - Pull request: microsoft#2578 - LightGBM 3.3.5 parameters: https://lightgbm.readthedocs.io/en/v3.3.5/Parameters.html#early-stopping-round ## Rationale Correct tolerance semantics classify more rounds as non-improving, so the wrapper must explicitly preserve LightGBM's zero-means-disabled behavior. Shared Spark parameter validators reject values that LightGBM does not support, while deterministic matrix tests cover the decision logic without depending on platform-specific native binaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

SynapseML's Azure Pipelines fans out ~30 hosted-agent matrix jobs that each cold-bootstrap the sbt launcher (org.scala-sbt:sbt:1.10.11, pinned in project/build.properties) and resolve Ivy dependencies from public Maven Central. When many fresh agents -- and several overlapping PR builds -- do this simultaneously, Maven Central returns HTTP 429 (rate limit) and "Setup repo" fails before any test runs (e.g. ADO build 229124511, UnitTests flaky). The pre-existing jittered retries only widened the window against a sustained throttle; they did not remove the thundering herd. Durable fix (cache-first, stagger as supplement): * templates/sbt_cache.yml (primary): Azure Cache@2 for the sbt launcher boot dir (~/.sbt/boot -- the artifact that 429s) and the Ivy cache (~/.ivy2/cache). In steady state, jobs restore these from Azure's cache service and never touch Maven Central. Keys derive from the bootstrap inputs (project/build.properties, project/plugins.sbt, build.sbt) so they invalidate exactly when those change; restoreKeys give a safe partial fallback and continueOnError keeps a cache miss/corruption non-fatal. * BuildAndCacheSbt prewarm job: warms those caches once per run, mirroring the existing BuildAndCacheCondaEnv job. * tools/ci/sbt_retry.sh: single tested helper replacing the duplicated inline retry blocks. Smooths only the cold-cache path with a bounded random start stagger (desynchronises concurrent cold bootstraps) plus bounded jittered exponential-backoff retries. Fails visibly on exhaustion -- no success fallback masking. Wired the shared cache template into every sbt-running job (Style, Publish, Databricks/Fabric E2E, BuildDocker, PythonTests, RTests, WebsiteSamplesTests, UnitTests, ReleaseBranchCompat) by reviving the dormant ivy_cache placeholders, and routed all `sbt setup` bootstraps through the helper. Tests (python -m pytest tools/ci/tests/): deterministically exercise the retry/backoff/stagger + visible-failure behaviour with a fake sbt, and assert pipeline.yaml parses, the cache keys invalidate on bootstrap inputs, and every sbt job is wired to the cache template + prewarm job. No LightGBM, Isolation Forest, GPU, or application changes. TLS verification, job coverage, and all tests are preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-07-31T23:37:44Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

github-actions · 2026-07-31T23:37:45Z

Hey @ranadeepsingh 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

fix: Fix LightGBM crashes with empty partitions
feat: Make HTTP on Spark back-offs configurable
docs: Update Spark Serving usage
build: Add codecov support
perf: improve LightGBM memory usage
refactor: make python code generation rely on classes
style: Remove nulls from CNTKModel
test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

Copilot

Pull request overview

This PR hardens SynapseML’s Azure Pipelines sbt bootstrap path to reduce Maven Central HTTP 429 failures by restoring sbt/ivy artifacts from Azure Cache and centralizing the cold-cache retry/stagger logic. The current diff also includes stacked LightGBM early-stopping tolerance fixes from #2578 (per PR description).

Changes:

Introduces a shared Cache@2 template for ~/.sbt/boot and ~/.ivy2/cache, and wires it into sbt-running Azure Pipeline jobs.
Adds tools/ci/sbt_retry.sh to provide bounded start stagger + bounded jittered exponential-backoff retries, and replaces duplicated inline retry snippets in the pipeline.
Adds deterministic CI tests validating both sbt_retry.sh behavior and that pipeline.yaml is wired to the cache template + prewarm job.

Show a summary per file

File	Description
tools/ci/tests/test_sbt_retry.py	Deterministic pytest coverage for `sbt_retry.sh` (stagger/backoff/retry/arg forwarding).
tools/ci/tests/test_pipeline_yaml.py	YAML-based wiring assertions for `pipeline.yaml` + templates (presence, parsing, cache usage, retry helper usage).
tools/ci/sbt_retry.sh	New retry/stagger wrapper for sbt bootstrap in CI.
tools/ci/README.md	Documents the CI bootstrap caching + retry approach and how to run tests.
templates/sbt_cache.yml	New reusable Cache@2 template for sbt launcher boot + Ivy cache.
pipeline.yaml	Adds sbt cache template usage, adds `BuildAndCacheSbt` prewarm job, and switches sbt setup calls to the retry helper.
lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split1/TrainUtilsSuite.scala	Tests for LightGBM improvement tolerance symmetry and early stopping parameter validation.
lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/TrainUtils.scala	Refactors early-stopping logic into helpers and fixes improvement comparison semantics.
lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/params/LightGBMParams.scala	Tightens param docs and adds non-negative validators for early stopping round and improvement tolerance.

Review details

Files reviewed: 9/9 changed files
Comments generated: 1
Review effort level: Lite

## Summary Make the sbt bootstrap prewarm a mandatory gate before Azure Pipeline matrix jobs start. Add Coursier caching, require exact hits on the boot, Ivy, and Coursier caches before disabling the cold-cache stagger, wire the conditional release job, and strengthen pipeline tests around the dependency graph and cache lifecycle. ## Prompting Intent The engineer asked to fix Maven Central HTTP 429 setup failures in a new stacked PR. The solution must prevent fresh hosted agents from cold-bootstrapping sbt concurrently, allow at least the existing job fan-out after bootstrap is safe, retain bounded retry behavior for cache-service failures, and keep bootstrap failures visible rather than masking them. ## Linked Sources - Failing Azure job: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229124511&view=logs&jobId=e97036a1-bcdb-5cd5-905e-b0cf2c8f33cf - Parent PR investigation: microsoft#2578 (comment) - Stacked PR: microsoft#2581 - Prewarm concurrency review: microsoft#2581 (comment) ## Rationale A best-effort prewarm running beside the matrix does not protect the first run for a new dependency key, so every sbt-running job now waits for one successful warm job. Cache-service errors remain non-fatal and fall back to staggered retries, but a failed warm blocks fan-out to avoid recreating the thundering herd. Coursier is cached alongside sbt boot and Ivy because modern resolution uses all three stores, and the stagger is suppressed only when every cache is an exact hit so dependency-only changes remain desynchronized. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ranadeepsingh · 2026-08-01T00:07:10Z

/azp run

azure-pipelines · 2026-08-01T00:07:21Z

Azure Pipelines: Successfully started running 1 pipeline(s).

codecov-commenter · 2026-08-01T00:30:34Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.71%. Comparing base (04897ba) to head (fda789b).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2581      +/-   ##
==========================================
+ Coverage   84.03%   84.71%   +0.67%     
==========================================
  Files         335      335              
  Lines       17882    17882              
  Branches     1639     1639              
==========================================
+ Hits        15028    15148     +120     
+ Misses       2854     2734     -120

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ranadeepsingh · 2026-08-01T03:56:33Z

/azp run

azure-pipelines · 2026-08-01T03:56:40Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

ranadeepsingh · 2026-08-01T03:57:37Z

/azp run

azure-pipelines · 2026-08-01T03:57:47Z

Azure Pipelines: Successfully started running 1 pipeline(s).

ranadeepsingh · 2026-08-01T04:27:27Z

/azp run

azure-pipelines · 2026-08-01T04:27:39Z

Azure Pipelines: Successfully started running 1 pipeline(s).

## Summary Add conservative pull-request impact detection for the six-leg Databricks E2E matrix. Clearly non-impacting documentation, website, GitHub metadata, CI helper, and isolated test-source changes skip Databricks, while all uncertain or runtime-affecting changes continue to run it. ## Prompting Intent The engineer asked to extend PR microsoft#2581 so expensive Databricks Azure Pipeline jobs are skipped when the pull request cannot affect notebook execution. The gate must preserve scheduled and branch coverage, avoid brittle CPU-shard mapping, and default to running whenever impact detection is incomplete or uncertain. ## Linked Sources - Stacked CI PR: microsoft#2581 - Full green baseline build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229164855 - Azure multi-job output variables: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops#job-output-variables-used-in-other-job-conditions ## Rationale Use one fail-open decision for the complete Databricks matrix because the five CPU partitions mix notebooks across modules and are not stable ownership boundaries. The detector skips only a narrow allowlist of clearly inert paths; runtime code, notebooks, build and pipeline files, Databricks test utilities, shared TestBase infrastructure, unknown paths, empty diffs, and fetch or classifier failures all keep E2E enabled. Non-PR builds always run to preserve scheduled and release coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ranadeepsingh · 2026-08-01T07:03:52Z

/azp run

azure-pipelines · 2026-08-01T07:04:04Z

Azure Pipelines: Successfully started running 1 pipeline(s).

ranadeepsingh · 2026-08-01T08:10:49Z

Validated the Databricks impact gate in Azure build 229178168:

Detector compared the PR merge ref with master and listed the effective changed paths.
pipeline.yaml and templates/sbt_cache.yml were correctly classified as impacting, so runDatabricksE2E=true.
The mandatory sbt prewarm passed before fan-out.
All five Databricks CPU legs and the Databricks GPU leg started and passed.
The complete Azure pipeline passed with no Maven 429 failures.

Local deterministic coverage for the skip branch also passes: docs/website/GitHub metadata/CI-helper/non-Databricks test-only changes return false; runtime, notebooks, builds, pipeline/templates, shared test infrastructure, unknown paths, empty diffs, and detection failures remain fail-open (true).

ranadeepsingh · 2026-08-01T09:38:57Z

/azp run

azure-pipelines · 2026-08-01T09:39:08Z

Azure Pipelines: Successfully started running 1 pipeline(s).

## Summary Classify changed paths against the actual Databricks CPU and GPU runtime surfaces, emit separate fail-open decisions, and gate each matrix leg independently. ## Prompting Intent The engineer asked to determine exactly when Databricks tests should run, lock down the path rules, and deliver the work as a stacked pull request above PR #2581. ## Linked Sources - Base CI hardening PR: #2581 - GitHub stacked PR documentation: https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-stacked-pull-requests - ADO timing audit: build 229176406 ## Rationale CPU and GPU decisions are separated because most module changes cannot affect the expensive GPU notebooks. Unknown paths and shared build or test infrastructure remain fail-open, while explicit test-only and unrelated tooling paths skip safely. This preserves coverage while avoiding unrelated GPU capacity waits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Run release compatibility checks for both GitHub target-branch formats and replace redundant compile, setup, credential, and per-package SBT tasks with one cached, project-scoped validation process. ## Prompting Intent The engineer asked to fix the silently skipped ReleaseBranchCompat job and simplify it before enabling it so the check is both reliable and efficient. ## Linked Sources - Base CI hardening PR: #2581 - Evidence build with skipped phase: ADO build 229176406 - Parent stack layer: ci/databricks-impact-gating ## Rationale The target condition accepts both values observed across Azure Repos and GitHub PR providers. A single SBT process retains full test compilation and the intended core, VW, and OpenCV compatibility suites while removing repeated build loading, root-wide IntelliJ setup, unnecessary Key Vault access, and Azure CLI authentication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Make the weekly ACR cleanup schedule-only, switch it to the dedicated cleanup service connection, replace storage connection-string authentication with Azure CLI Entra authentication, and add fail-safe cleanup tests. The engineer asked to repair the weekly cleanup failures caused by disabled key-based storage authentication, use the declared least-privileged identity, and prevent accidental CI or PR execution. - Failed scheduled build: ADO build 228250033 - Base CI hardening PR: #2581 - Azure CLI pipeline-run reference: https://learn.microsoft.com/en-us/cli/azure/acr/pipeline-run - Parent stack layer: ci/release-branch-compat Using az storage blob exists with auth-mode login keeps all operations inside the AzureCLI task identity and removes runtime SDK installation, Key Vault access, and storage keys. Images are deleted only after the archive is confirmed, and subprocess argument lists avoid shell interpolation of registry-controlled names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Run non-Azure setup and coverage commands as Bash steps, install pinned Black without restoring the 8.6 GB Conda environment, and remove the ineffective standalone Conda cache consumer. ## Prompting Intent The engineer asked for additional improvements that should ship with the requested CI fixes to make builds faster and more reliable without broad behavioral changes. ## Linked Sources - CI efficiency audit from ADO build 229176406 - Base CI hardening PR: #2581 - Parent stack layer: ci/fix-acr-cleanup-auth ## Rationale AzureCLI tasks create an isolated login for every invocation, so setup and coverage steps that never call az gain no authentication benefit. The Style job only needs pinned Black, not the full cached environment. The standalone Conda job was not a dependency and therefore could not prewarm consumers or prevent cold-cache fan-out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Use Entra authentication for ACR cleanup Make the weekly ACR cleanup schedule-only, switch it to the dedicated cleanup service connection, replace storage connection-string authentication with Azure CLI Entra authentication, and add fail-safe cleanup tests. The engineer asked to repair the weekly cleanup failures caused by disabled key-based storage authentication, use the declared least-privileged identity, and prevent accidental CI or PR execution. - Failed scheduled build: ADO build 228250033 - Base CI hardening PR: #2581 - Azure CLI pipeline-run reference: https://learn.microsoft.com/en-us/cli/azure/acr/pipeline-run - Parent stack layer: ci/release-branch-compat Using az storage blob exists with auth-mode login keeps all operations inside the AzureCLI task identity and removes runtime SDK installation, Key Vault access, and storage keys. Images are deleted only after the archive is confirmed, and subprocess argument lists avoid shell interpolation of registry-controlled names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant CI authentication and Conda work (#2585) ## Summary Run non-Azure setup and coverage commands as Bash steps, install pinned Black without restoring the 8.6 GB Conda environment, and remove the ineffective standalone Conda cache consumer. ## Prompting Intent The engineer asked for additional improvements that should ship with the requested CI fixes to make builds faster and more reliable without broad behavioral changes. ## Linked Sources - CI efficiency audit from ADO build 229176406 - Base CI hardening PR: #2581 - Parent stack layer: ci/fix-acr-cleanup-auth ## Rationale AzureCLI tasks create an isolated login for every invocation, so setup and coverage steps that never call az gain no authentication benefit. The Style job only needs pinned Black, not the full cached environment. The standalone Conda job was not a dependency and therefore could not prewarm consumers or prevent cold-cache fan-out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix and streamline release branch compatibility checks ## Summary Run release compatibility checks for both GitHub target-branch formats and replace redundant compile, setup, credential, and per-package SBT tasks with one cached, project-scoped validation process. ## Prompting Intent The engineer asked to fix the silently skipped ReleaseBranchCompat job and simplify it before enabling it so the check is both reliable and efficient. ## Linked Sources - Base CI hardening PR: #2581 - Evidence build with skipped phase: ADO build 229176406 - Parent stack layer: ci/databricks-impact-gating ## Rationale The target condition accepts both values observed across Azure Repos and GitHub PR providers. A single SBT process retains full test compilation and the intended core, VW, and OpenCV compatibility suites while removing repeated build loading, root-wide IntelliJ setup, unnecessary Key Vault access, and Azure CLI authentication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: use Entra authentication for ACR cleanup (#2584) * Use Entra authentication for ACR cleanup Make the weekly ACR cleanup schedule-only, switch it to the dedicated cleanup service connection, replace storage connection-string authentication with Azure CLI Entra authentication, and add fail-safe cleanup tests. The engineer asked to repair the weekly cleanup failures caused by disabled key-based storage authentication, use the declared least-privileged identity, and prevent accidental CI or PR execution. - Failed scheduled build: ADO build 228250033 - Base CI hardening PR: #2581 - Azure CLI pipeline-run reference: https://learn.microsoft.com/en-us/cli/azure/acr/pipeline-run - Parent stack layer: ci/release-branch-compat Using az storage blob exists with auth-mode login keeps all operations inside the AzureCLI task identity and removes runtime SDK installation, Key Vault access, and storage keys. Images are deleted only after the archive is confirmed, and subprocess argument lists avoid shell interpolation of registry-controlled names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant CI authentication and Conda work (#2585) ## Summary Run non-Azure setup and coverage commands as Bash steps, install pinned Black without restoring the 8.6 GB Conda environment, and remove the ineffective standalone Conda cache consumer. ## Prompting Intent The engineer asked for additional improvements that should ship with the requested CI fixes to make builds faster and more reliable without broad behavioral changes. ## Linked Sources - CI efficiency audit from ADO build 229176406 - Base CI hardening PR: #2581 - Parent stack layer: ci/fix-acr-cleanup-auth ## Rationale AzureCLI tasks create an isolated login for every invocation, so setup and coverage steps that never call az gain no authentication benefit. The Style job only needs pinned Black, not the full cached environment. The standalone Conda job was not a dependency and therefore could not prewarm consumers or prevent cold-cache fan-out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Gate Databricks CPU and GPU tests independently ## Summary Classify changed paths against the actual Databricks CPU and GPU runtime surfaces, emit separate fail-open decisions, and gate each matrix leg independently. ## Prompting Intent The engineer asked to determine exactly when Databricks tests should run, lock down the path rules, and deliver the work as a stacked pull request above PR microsoft#2581. ## Linked Sources - Base CI hardening PR: microsoft#2581 - GitHub stacked PR documentation: https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-stacked-pull-requests - ADO timing audit: build 229176406 ## Rationale CPU and GPU decisions are separated because most module changes cannot affect the expensive GPU notebooks. Unknown paths and shared build or test infrastructure remain fail-open, while explicit test-only and unrelated tooling paths skip safely. This preserves coverage while avoiding unrelated GPU capacity waits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: fix and streamline release branch compatibility (microsoft#2583) * Fix and streamline release branch compatibility checks ## Summary Run release compatibility checks for both GitHub target-branch formats and replace redundant compile, setup, credential, and per-package SBT tasks with one cached, project-scoped validation process. ## Prompting Intent The engineer asked to fix the silently skipped ReleaseBranchCompat job and simplify it before enabling it so the check is both reliable and efficient. ## Linked Sources - Base CI hardening PR: microsoft#2581 - Evidence build with skipped phase: ADO build 229176406 - Parent stack layer: ci/databricks-impact-gating ## Rationale The target condition accepts both values observed across Azure Repos and GitHub PR providers. A single SBT process retains full test compilation and the intended core, VW, and OpenCV compatibility suites while removing repeated build loading, root-wide IntelliJ setup, unnecessary Key Vault access, and Azure CLI authentication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: use Entra authentication for ACR cleanup (microsoft#2584) * Use Entra authentication for ACR cleanup Make the weekly ACR cleanup schedule-only, switch it to the dedicated cleanup service connection, replace storage connection-string authentication with Azure CLI Entra authentication, and add fail-safe cleanup tests. The engineer asked to repair the weekly cleanup failures caused by disabled key-based storage authentication, use the declared least-privileged identity, and prevent accidental CI or PR execution. - Failed scheduled build: ADO build 228250033 - Base CI hardening PR: microsoft#2581 - Azure CLI pipeline-run reference: https://learn.microsoft.com/en-us/cli/azure/acr/pipeline-run - Parent stack layer: ci/release-branch-compat Using az storage blob exists with auth-mode login keeps all operations inside the AzureCLI task identity and removes runtime SDK installation, Key Vault access, and storage keys. Images are deleted only after the archive is confirmed, and subprocess argument lists avoid shell interpolation of registry-controlled names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant CI authentication and Conda work (microsoft#2585) ## Summary Run non-Azure setup and coverage commands as Bash steps, install pinned Black without restoring the 8.6 GB Conda environment, and remove the ineffective standalone Conda cache consumer. ## Prompting Intent The engineer asked for additional improvements that should ship with the requested CI fixes to make builds faster and more reliable without broad behavioral changes. ## Linked Sources - CI efficiency audit from ADO build 229176406 - Base CI hardening PR: microsoft#2581 - Parent stack layer: ci/fix-acr-cleanup-auth ## Rationale AzureCLI tasks create an isolated login for every invocation, so setup and coverage steps that never call az gain no authentication benefit. The Style job only needs pinned Black, not the full cached environment. The standalone Conda job was not a dependency and therefore could not prewarm consumers or prevent cold-cache fan-out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ranadeepsingh · 2026-08-04T07:06:05Z

/azp run

azure-pipelines · 2026-08-04T07:06:17Z

Azure Pipelines: Successfully started running 1 pipeline(s).

ranadeepsingh · 2026-08-04T07:07:13Z

/azp run

azure-pipelines · 2026-08-04T07:07:25Z

Azure Pipelines: Successfully started running 1 pipeline(s).

## Summary Merge the reviewed microsoft#2582 squash commit back into microsoft#2581 after a concurrent master refresh displaced the child merge, restoring the cumulative microsoft#2582- microsoft#2585 pipeline changes on top of current master. ## Prompting Intent The engineer asked to diagnose the failing tests on microsoft#2581 while completing, reviewing, and fully validating the stacked CI pull requests without losing work already merged through the stack. ## Linked Sources - Parent PR: microsoft#2581 - Databricks gating PR: microsoft#2582 - Release compatibility PR: microsoft#2583 - ACR authentication PR: microsoft#2584 - CI setup cleanup PR: microsoft#2585 - Failed Fabric build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229191951 ## Rationale The old red build was caused by a transient DNS failure resolving the Fabric service endpoint, not by the CI changes. During the investigation, a branch refresh raced with the stacked merges and moved microsoft#2581 past microsoft#2582's squash commit. Merging the official squash commit preserves the exact reviewed child tree and its history instead of reconstructing or duplicating those changes, while retaining the latest master updates already present on the parent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ranadeepsingh · 2026-08-04T07:22:08Z

The previous Azure failure in build 229191951 was an external Fabric DNS failure, not a regression in this PR. All non-Fabric jobs passed; the three reported notebook failures shared UnknownHostException: WABI-US-CENTRAL-B-PRIMARY-redirect.analysis.windows.net, and later tests failed immediately after that first endpoint-resolution failure. Recent scheduled master builds succeeded.

While refreshing the stack, the parent branch update raced with the squash merge of #2582, temporarily dropping the cumulative #2582-#2585 changes from this PR even though those child PRs were marked merged. Commit ac80bf32b6 restores the official #2582 squash tree on top of current master; the PR diff now includes the Databricks gating, release compatibility, ACR Entra cleanup, and redundant setup removal changes.

The integrated CI/ACR test set passes (35 tests), along with pinned Black 22.3.0 and Scala style checks. Requesting a fresh full Azure validation on the restored head.

ranadeepsingh · 2026-08-04T07:22:09Z

/azp run

azure-pipelines · 2026-08-04T07:22:22Z

Azure Pipelines: Successfully started running 1 pipeline(s).

## Summary Raise the BuildDocker job timeout from 60 to 120 minutes and add a pipeline regression test that preserves enough time for both sequential image builds. ## Prompting Intent The engineer asked to diagnose and fix the remaining failure on microsoft#2581 and to continue full validation until the parent PR is ready, without hiding genuine test failures. ## Linked Sources - Parent PR: microsoft#2581 - Failed PR build: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229579403 - Matching master failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229578121 - Matching master failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229580525 - Matching master failure: https://msdata.visualstudio.com/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=229576499 ## Rationale The Dockerfiles and image behavior were unchanged, but recent hosted-agent builds required roughly 51 minutes when successful and exceeded the default one-hour job cap in multiple master and PR runs. A 120-minute job budget keeps both image validations mandatory while tolerating current registry and package download latency. This is safer and more targeted than skipping an image or doubling agent usage by splitting the builds into parallel jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Merge the latest master commits into microsoft#2581 after restoring the stacked CI changes and extending the Docker validation budget. ## Prompting Intent The engineer asked to refresh the parent PR against master, preserve all merged stack changes, and continue full validation until the current merge candidate is ready. ## Linked Sources - Parent PR: microsoft#2581 - CDN migration merged to master: microsoft#2589 - Azure Search AAD authentication merged to master: microsoft#2591 ## Rationale Merging master preserves the existing reviewed branch history and the official stack recovery commit without rewriting the fork. The merge is conflict-free, keeps the Docker timeout regression fix intact, and ensures the next Azure run tests the actual current integration candidate rather than a stale base. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ranadeepsingh · 2026-08-04T10:11:13Z

Fresh build 229579403 confirmed the earlier Fabric DNS issue was transient: Fabric E2E passed, as did all Databricks CPU/GPU legs, both release-branch compatibility jobs, every unit/Python/R suite, publish, style, and website validation.

The sole failure was BuildDocker reaching Azure Pipelines' default 60-minute job cap. The demo image completed after 41.7 minutes; the minimal image was still building when the job was canceled. This is reproducible on current master: builds 229578121, 229580525, and 229576499 all hit the same cap, while the successful neighboring run needed about 51 minutes for the two sequential image builds.

Commit 3f0f238554 raises only BuildDocker to 120 minutes and adds a regression assertion; it does not skip or weaken either image build. Commit fda789bad0 then merges current master so this rerun validates the latest integration candidate.

ranadeepsingh · 2026-08-04T10:11:14Z

/azp run

azure-pipelines · 2026-08-04T10:11:27Z

Azure Pipelines: Successfully started running 1 pipeline(s).

ranadeepsingh · 2026-08-04T11:42:35Z

Final validation is green on the current head. Azure build 229602106 succeeded, including BuildDocker (58m41s), Fabric E2E, Databricks CPU/GPU, release compatibility, publish, unit, Python, R, style, and website jobs. The PR is ready for review/merge.

ranadeepsingh and others added 4 commits July 30, 2026 05:35

Merge branch 'master' into fix/issue-2565-lightgbm-improvement-tolerance

0f5761d

ranadeepsingh requested a review from svotaw as a code owner July 31, 2026 23:37

Copilot AI review requested due to automatic review settings July 31, 2026 23:37

Copilot started reviewing on behalf of ranadeepsingh July 31, 2026 23:38 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

Comment thread pipeline.yaml Outdated

ranadeepsingh added the area/build label Aug 1, 2026

Merge branch 'master' into rana/harden-sbt-maven-429

e44797b

Merge branch 'master' into rana/harden-sbt-maven-429

b495cac

ranadeepsingh mentioned this pull request Aug 1, 2026

ci: gate Databricks CPU and GPU tests independently #2582

Merged

5 tasks

This was referenced Aug 1, 2026

ci: fix and streamline release branch compatibility #2583

Merged

ci: use Entra authentication for ACR cleanup #2584

Merged

ci: remove redundant authentication and Conda setup #2585

Merged

ranadeepsingh and others added 2 commits August 4, 2026 00:05

Merge branch 'master' into rana/harden-sbt-maven-429

b82cb4b

Merge branch 'master' into rana/harden-sbt-maven-429

b07ddac

ranadeepsingh and others added 2 commits August 4, 2026 03:06

ranadeepsingh changed the title ~~ci: prevent sbt bootstrap Maven rate limits~~ ci: land sbt bootstrap and stacked CI fixes (#2582-#2585) Aug 4, 2026

ranadeepsingh requested a review from BrendanWalsh August 4, 2026 23:08

ranadeepsingh changed the title ~~ci: land sbt bootstrap and stacked CI fixes (#2582-#2585)~~ ci: land sbt bootstrap and efficient CI fixes Aug 4, 2026

Uh oh!

Conversation

ranadeepsingh commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration PR for the completed CI stack

Intent

Does this actually save time?

Conservative gating

Reliability fixes

Validation

Scope

Uh oh!

azure-pipelines Bot commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

ranadeepsingh commented Aug 1, 2026

Uh oh!

azure-pipelines Bot commented Aug 1, 2026

Uh oh!

codecov-commenter commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ranadeepsingh commented Aug 1, 2026

Uh oh!

azure-pipelines Bot commented Aug 1, 2026

Uh oh!

ranadeepsingh commented Aug 1, 2026

Uh oh!

azure-pipelines Bot commented Aug 1, 2026

Uh oh!

ranadeepsingh commented Aug 1, 2026

Uh oh!

azure-pipelines Bot commented Aug 1, 2026

Uh oh!

ranadeepsingh commented Aug 1, 2026

Uh oh!

azure-pipelines Bot commented Aug 1, 2026

Uh oh!

ranadeepsingh commented Aug 1, 2026

Uh oh!

ranadeepsingh commented Aug 1, 2026

Uh oh!

azure-pipelines Bot commented Aug 1, 2026

Uh oh!

ranadeepsingh commented Aug 4, 2026

Uh oh!

azure-pipelines Bot commented Aug 4, 2026

Uh oh!

ranadeepsingh commented Aug 4, 2026

Uh oh!

azure-pipelines Bot commented Aug 4, 2026

Uh oh!

ranadeepsingh commented Aug 4, 2026

Uh oh!

ranadeepsingh commented Aug 4, 2026

Uh oh!

azure-pipelines Bot commented Aug 4, 2026

Uh oh!

ranadeepsingh commented Aug 4, 2026

Uh oh!

ranadeepsingh commented Aug 4, 2026

Uh oh!

azure-pipelines Bot commented Aug 4, 2026

Uh oh!

ranadeepsingh commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

ranadeepsingh commented Jul 31, 2026 •

edited

Loading

codecov-commenter commented Aug 1, 2026 •

edited

Loading