chore: migrate artifact links off retiring Azure CDN - #2589
Conversation
## Summary Replace all 400 current-master references to mmlspark.azureedge.net with the repository-owned mmlspark Blob Storage origin across runtime package configuration, release output, examples, documentation, notebooks, and every published documentation version. ## Prompting Intent Recreate the intent of the stale CDN-removal PR on current master only after verifying the supported artifact destination and Azure CDN retirement path. Audit each endpoint use by semantics, preserve package and content paths, validate live artifacts and package resolution, and avoid changing or closing the original PR. ## Linked Sources - Original proposal: microsoft#2326 - Azure CDN retirement FAQ: https://learn.microsoft.com/en-us/azure/cdn/classic-cdn-retirement-faq - Azure CDN migration guidance: https://learn.microsoft.com/en-us/azure/cdn/migrate-tier - Azure Front Door/CDN comparison: https://learn.microsoft.com/en-us/azure/frontdoor/front-door-cdn-comparison ## Rationale SynapseML's release pipeline publishes artifacts directly to the mmlspark storage account, the repository already uses that public Blob Storage origin extensively, and byte-for-byte URL checks confirmed the CDN currently proxies the same content. Using the verified origin removes the retiring CDN hostname without inventing an unverified Front Door name, while preserving Maven, documentation, R-package, model, dataset, and icon path semantics. Historical links that already return 404 retain the same status and are not broadened into unrelated artifact-repair work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Hey @ranadeepsingh 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
Migrates SynapseML artifact/documentation links away from the retiring Azure CDN hostname (mmlspark.azureedge.net) to the direct, repository-owned Blob Storage origin (mmlspark.blob.core.windows.net) across runtime configuration, documentation, and examples.
Changes:
- Replaced CDN URLs with Blob Storage URLs across docs/website content (including multiple versioned-doc sets and notebooks).
- Updated runtime defaults and helper scripts to resolve Maven artifacts and datasets from the Blob origin.
- Updated build/release messaging to advertise the Blob-based Maven resolver.
Show a summary per file
| File | Description |
|---|---|
| website/versioned_docs/version-1.1.3/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.1.3/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.1.3/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.3/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.3/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.3/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.2/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.1.2/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.1.2/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.2/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.2/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.2/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.1/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.1.1/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.1.1/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.1/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.1/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.1/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.0/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.1.0/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.1.0/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.0/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.0/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.1.0/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.9/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.9/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.9/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.9/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.9/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.9/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.8/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.8/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.8/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.8/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.8/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.8/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.7/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.7/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.7/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.7/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.7/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.7/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.6/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.6/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.6/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.6/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.6/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.6/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.5/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.5/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.5/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.5/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.5/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.5/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.4/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.4/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.4/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.4/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.4/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.3/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.3/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.3/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.3/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.3/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.2/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.2/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.2/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.2/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.2/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.15/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.15/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.15/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.15/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.15/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.15/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.14/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.14/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.14/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.14/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.14/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.14/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.13/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.13/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.13/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.13/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.13/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.13/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.12/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.12/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.12/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.12/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.12/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.12/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.11/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.11/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.11/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.11/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.11/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.11/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.10/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.10/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.10/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.10/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.10/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.10/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.1/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-1.0.1/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-1.0.1/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.1/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-1.0.1/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-0.11.4/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-0.11.4/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-0.11.4/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-0.11.4/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-0.11.4/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-0.11.3/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| website/versioned_docs/version-0.11.3/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| website/versioned_docs/version-0.11.3/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-0.11.3/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.md | Update Maven resolver URL to Blob origin |
| website/versioned_docs/version-0.11.3/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| website/src/pages/index.js | Update displayed resolver URLs to Blob origin |
| website/doctest.py | Update embedded resolver URL used in doctest injection |
| tools/helm/zeppelin/zeppelin-env.sh | Update commented Spark submit repo URL to Blob origin |
| tools/helm/zeppelin/mmlsparkExamples/submitjob_2DZ7DHX6E.zpln | Update repo URL inside Zeppelin notebook JSON |
| tools/helm/zeppelin/mmlsparkExamples/simplification_mmlspark.zpln | Update dataset download URL inside Zeppelin notebook JSON |
| tools/helm/zeppelin/mmlsparkExamples/classification_mmlspark_2E3REACQR.zpln | Update dataset/default URL references inside Zeppelin notebook JSON |
| tools/docker/demo/init_notebook.py | Update docker demo resolver URL to Blob origin |
| README.md | Update badge/icon and resolver URLs to Blob origin |
| project/BlobMavenPlugin.scala | Update printed Maven resolver to Blob origin |
| docs/Reference/R Setup.md | Update R artifact install URLs to Blob origin |
| docs/Reference/Contributor Guide.md | Update PySpark API docs link to Blob origin |
| docs/Get Started/Install SynapseML.md | Update Maven resolver URL to Blob origin |
| docs/Explore Algorithms/Other Algorithms/Quickstart - Anomalous Access Detection.ipynb | Update Maven resolver URL in notebook JSON |
| docs/Explore Algorithms/Deep Learning/Getting Started.md | Update Maven resolver URL to Blob origin |
| docs/Explore Algorithms/Anomaly Detection/Quickstart - Isolation Forests.ipynb | Update Maven resolver URL in notebook JSON |
| core/src/main/scala/com/microsoft/azure/synapse/ml/core/env/PackageUtils.scala | Update central repo constant to Blob origin |
| core/src/main/python/synapse/ml/downloader/ModelDownloader.py | Update default model download base URL to Blob origin |
| core/src/main/python/synapse/ml/core/init_spark.py | Update Spark session resolver config to Blob origin |
| CONTRIBUTING.md | Update PySpark API docs link to Blob origin |
Review details
- Files reviewed: 139/139 changed files
- Comments generated: 1
- Review effort level: Lite
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2589 +/- ##
=======================================
Coverage 84.79% 84.79%
=======================================
Files 334 334
Lines 17806 17806
Branches 1623 1623
=======================================
Hits 15099 15099
Misses 2707 2707 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
## Summary Repair current and versioned R setup guidance so each release installs its six published, version-matched component archives and resolves SynapseML JVM artifacts through Blob Storage. Document the compatibility bypass required by already-published wrappers, correct the Databricks setup and LightGBM example, remove invalid HTML-page Maven repositories from the Docker demo, and add generator/docs regressions. ## Prompting Intent Investigate the review finding that published R archives still register the retired Azure CDN resolver. Make repository-controlled R installation work with that hostname unavailable, avoid claiming that externally published archives were rewritten, validate local and Databricks-oriented resolution paths, and state the exact external publishing prerequisite for a full artifact migration. ## Linked Sources - Original migration PR: microsoft#2326 - Current migration PR: microsoft#2589 - Maven repository review: microsoft#2589 (comment) - Azure CDN retirement FAQ: https://learn.microsoft.com/en-us/azure/cdn/classic-cdn-retirement-faq - Azure Front Door migration guidance: https://learn.microsoft.com/en-us/azure/cdn/migrate-tier - Apache Spark package repository configuration: https://spark.apache.org/docs/3.5.0/configuration.html#runtime-environment ## Rationale Existing release archives cannot be repaired by a source-only change because their generated sparklyr metadata is already published. Version-matched component downloads plus an explicit Blob resolver and `extensions = character()` provide a tested repository-controlled path without racing or misrepresenting external publication. Future generated archives inherit the corrected resolver from PackageUtils; fully repairing historical metadata still requires an authorized regeneration and publish to the `mmlspark/rrr` container (or a replacement release). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
R completeness follow-up is ready for re-review in 9f1ddcf. The guides now use version-matched component archives, explicitly route Maven resolution to Blob, suppress the published wrappers' embedded retired-resolver registration, and configure Databricks before cluster start. I validated all 126 archive URLs and ran real R/Spark core + LightGBM examples with the legacy hostname made unresolvable. The PR description now explicitly scopes the remaining external prerequisite: authorized regeneration/republication of historical wrapper archives. The prior Maven HTML-page review is fixed and resolved. Please re-review the updated diff. |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Review details
Suppressed comments (1)
website/src/pages/index.js:410
- The inline comment in the Python install snippet says to use SynapseML 1.1.3 for both Spark 3.5 and Spark 3.4, but elsewhere (including the SBT snippet just below) the Spark 3.4 guidance is 1.0.15. This is misleading for users copying the snippet.
snippet={`import pyspark
spark = (pyspark.sql.SparkSession.builder.appName("MyApp")
.config("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:1.1.3") # Please use 1.1.3 version for Spark3.5 and 1.1.3 version for Spark3.4
.config("spark.jars.repositories", "https://mmlspark.blob.core.windows.net/maven")
.getOrCreate())
- Files reviewed: 141/141 changed files
- Comments generated: 0 new
- Review effort level: Lite
## Summary Correct the Spark Packages and Python installation snippets so both identify SynapseML 1.0.15 as the compatible release for Spark 3.4 while retaining SynapseML 1.1.3 for Spark 3.5. ## Prompting Intent Address the remaining actionable review feedback on PR microsoft#2589 in the existing branch, verify the surrounding compatibility guidance stays consistent, run targeted website validation and code review, and rerun the full PR checks. ## Linked Sources - Pull request and review feedback: microsoft#2589 - Original migration context: microsoft#2326 ## Rationale The Databricks, Fabric, and SBT guidance already distinguishes SynapseML 1.1.3 for Spark 3.5 from 1.0.15 for Spark 3.4. Updating only the two stale explanatory references restores consistency without changing the Spark 3.5 commands that the snippets demonstrate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
## Summary Merge the latest master commits into microsoft#2581 after restoring the stacked CI changes and extending the Docker validation budget. ## Prompting Intent The engineer asked to refresh the parent PR against master, preserve all merged stack changes, and continue full validation until the current merge candidate is ready. ## Linked Sources - Parent PR: microsoft#2581 - CDN migration merged to master: microsoft#2589 - Azure Search AAD authentication merged to master: microsoft#2591 ## Rationale Merging master preserves the existing reviewed branch history and the official stack recovery commit without rewriting the fork. The merge is conflict-free, keeps the Docker timeout regression fix intact, and ensures the next Azure run tests the actual current integration candidate rather than a stale base. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Related work and retirement guidance
What changed
mmlspark.azureedge.netin 139 files with the repository-ownedmmlspark.blob.core.windows.netorigin.extensions = character();mvnrepository.com/artifact/...HTML pages from the Docker demo's Maven repository list; Maven Central remains Spark/Ivy's default for those dependencies.Why direct Blob Storage
mmlsparkstorage account.Published R archive limitation (scope clarification)
This is a source-and-documentation migration; it does not claim that already-published R ZIP contents were rewritten. The component archives for the documented releases still contain generated sparklyr metadata naming the retired resolver. Current source generation now emits the Blob resolver, but repairing historical artifact metadata requires an external publishing action.
The repository-controlled installation path in this PR was tested with the legacy hostname made unresolvable: install the Blob-hosted, version-matched component archives; pass the aggregate Maven coordinate and Blob repository explicitly; and use
extensions = character()so sparklyr ignores the embedded registration.Exact prerequisite for full artifact migration: a maintainer with authorized write access to the
mmlspark/rrrBlob container must regenerate and publish the six supported component archives with the updatedRCodegen/PackageUtilsoutput (or publish replacement release versions), then verify the publishedpackage_register.Rmetadata. This PR intentionally does not overwrite release artifacts.Validation
ml_clean_missing_datatransformed all 272 rows;com.microsoft.azure:synapseml_2.12:1.1.3and all module dependencies through the Blob resolver.sbt -batch "core/Test/testOnly com.microsoft.azure.synapse.ml.codegen.VerifyRCodegen"— 1 passed.sbt -batch scalastyle test:scalastyle core/Test/compile— passed.black==22.3.0 --check --extend-exclude 'docs/' .— 179 files unchanged.npm test— 24 passed;npm run build— production build succeeded.git diff --check; zero tracked legacy-host references; zeromvnrepository.com/artifactrepository entries.Pre-existing dead paths
The former combined R archive URLs and two directory-style dataset/model URLs were already 404 through both the CDN and Blob origin. The combined R links are now removed from the setup guides in favor of the published component archives. No dead artifact is presented as newly repaired by a hostname substitution.