feat(search): migrate Azure AI Search to the 2026-04-01 API and profile-based vector schema

[ranadeepsingh]

Related Issues/PRs

• Follow-up to feat: add backward-compatible AAD auth to Azure Search #2591 (AAD auth for Azure Search), which is now merged. This PR is the second half of that work: the API version and schema modernization, split out to keep feat: add backward-compatible AAD auth to Azure Search #2591 reviewable.

What changes are proposed in this pull request?

SynapseML pinned the Azure AI Search data-plane API to 2023-07-01-Preview. That is the one preview version Microsoft explicitly flags as deprecated — deprecated 2024-04-08, no longer supported since 2024-07-08 — and the migration guide says outright: "2023-07-01-preview was the first REST API for vector support. Do not use this API version."

This moves the default to 2026-04-01, the current stable data-plane version.

The reason this wasn't just a one-line version bump: 2023-11-01 introduced a breaking change to the vector index schema that SynapseML never adopted, so bumping the version alone would break every vector index.

Legacy (≤ 2023-10-01-preview)	Modern (≥ 2023-11-01)
vectorSearch.algorithmConfigurations	vectorSearch.algorithms + vectorSearch.profiles
field vectorSearchConfiguration	field vectorSearchProfile
searchable not permitted on a vector field	searchable: true required on vector fields

Rather than a hard cutover, this adds VectorSchema.align, which translates a parsed index definition in both directions into whichever shape the requested api-version expects, and runs before validation so what gets validated is exactly what gets sent. Consequences:

• Users who pass a hand-written legacy indexJson keep working — it's upgraded on the way out.
• Users who deliberately pin an older apiVersion keep working — modern definitions are downgraded to match.
• When upgrading legacy JSON, each algorithm config is mirrored as a same-named profile, so pre-existing field references such as vectorSearchConfiguration: "vectorConfig" resolve unchanged as vectorSearchProfile: "vectorConfig".
• getVectorColConf reads back either spelling when inspecting an existing index.

Also fixed here, because the version change made it user-visible: AddDocuments hard-coded the default api-version into its document endpoint, so an explicit apiVersion option applied to index creation but silently not to document upload. Both now use the resolved version.

Public class names (AzureSearch, AzureSearchWriter, AddDocuments) are deliberately unchanged — renaming them would be a breaking API change. Only prose, comments, and error messages adopt the current service name.

Docs

The service was renamed Azure Cognitive Search → Azure AI Search; the docs still used the old name and shipped stale sample api-versions (2019-05-06, 2023-07-01-Preview). Updated across 4 notebooks and _AzureSearch.md. website/versioned_docs/** is intentionally untouched, since those are frozen release snapshots.

How is this patch tested?

• New secret-free VectorSchemaMigrationSuite (10 tests): default-version sanity, the api-version gate, legacy→modern upgrade, modern→legacy downgrade, idempotency, round-trip stability, dual parsing of both shapes, non-vector no-op, serialized-key assertions, and nested-field translation.
• Existing SearchWriterSuitePart1/Part2 assertions updated to the version-agnostic vectorReference helper. Their legacy indexJson fixtures were deliberately kept as-is, so they now double as backward-compatibility coverage.
• cognitive/Test/compile
• cognitive/scalastyle and cognitive/Test/scalastyle — 0 errors
• cognitive/testOnly ...search.VectorSchemaMigrationSuite ...AzureSearchAuthSuite ...AzureSearchGenericParamPersistenceSuite ...AddDocumentsHeaderPersistenceSuite — 65 tests, all passing

The live-service suites remain behind the repo's existing credential gates and were not run locally.

Note for reviewers

One thing worth a maintainer's eye: indexes already created under 2023-07-01-Preview may need a genuine service-side migration, since Microsoft's guidance prescribes a manual Get Index → modify → Create-or-Update. Accepting both field spellings on read mitigates this, but I couldn't confirm without a live service whether the service auto-translates old vector indexes when read at a newer api-version.

Does this PR change any dependencies?

• No.

Does this PR add a new feature? If so, have you added samples on website?

• No new user-facing component. Existing docs and samples are updated in place to the current service name and api-version.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey @ranadeepsingh 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Copilot]

Pull request overview

Updates SynapseML’s Azure AI Search integration to default to the 2026-04-01 data-plane API while preserving compatibility across the legacy (pre-2023-11-01) and modern vector index schema shapes by translating index definitions to the schema required by the selected api-version.

Changes:

• Switch default Azure AI Search API version to 2026-04-01 and add an api-version gate for profile-based vector schemas.
• Introduce vector schema alignment (legacy ⇄ modern), including dual parsing and a unified vectorReference accessor.
• Update writer behavior so document upload uses the same resolved api-version as index operations; refresh docs/notebooks to “Azure AI Search” and current sample API versions.

Show a summary per file

File	Description
docs/Quick Examples/transformers/cognitive/_AzureSearch.md	Renames service heading to Azure AI Search.
docs/Explore Algorithms/OpenAI/Quickstart - Understand and Search Forms.ipynb	Updates narrative + sample REST api-version to 2026-04-01.
docs/Explore Algorithms/AI Services/Quickstart - Document Question and Answering with PDFs.ipynb	Updates naming + vector-store REST api-version to 2026-04-01.
docs/Explore Algorithms/AI Services/Quickstart - Create a Visual Search Engine.ipynb	Updates naming + sample REST api-version to 2026-04-01.
docs/Explore Algorithms/AI Services/Overview.ipynb	Updates Azure AI Search naming in overview and comments.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/VectorSchemaMigrationSuite.scala	Adds secret-free tests validating schema migration + version gate behavior.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/split2/SearchWriterSuitePart2.scala	Uses version-agnostic vectorReference assertion for vectors.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/split1/SearchWriterSuitePart1.scala	Uses version-agnostic vectorReference assertions for vectors.
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchSchemas.scala	Adds modern vector schema types (profiles/algorithms), vectorReference, and VectorSchema.align.
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchAuth.scala	Updates user-facing auth error strings to “Azure AI Search”.
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchAPI.scala	Defaults api-version to 2026-04-01; aligns schema pre-validation; updates vector validation to use vectorReference.
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearch.scala	Emits modern vector schema by default; pins AddDocuments endpoint to the selected api-version; switches vector detection to isVectorField.

Review details

Suppressed comments (1)

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchAPI.scala:160

• validateIndexInfo aligns vector schemas but still allows vector fields to be sent without any vectorSearch.algorithms / legacy vectorSearch.algorithmConfigurations being defined (e.g., if a user supplies dimensions + vectorSearchProfile/config on fields but omits the vectorSearch section). That will pass current validation and fail later at request time with a less actionable service error. Consider rejecting this early with an explicit validation check after alignment.

  private def validateIndexInfo(indexJson: String, apiVersion: String): Try[IndexInfo] = {
    // Align first so the definition that is validated is exactly the one that gets sent.
    val schema = VectorSchema.align(parseIndexJson(indexJson), apiVersion)
    for {
      _ <- validName(schema.name.get)
      _ <- validIndexFields(schema.fields)
    } yield schema
  }

• Files reviewed: 12/12 changed files
• Comments generated: 0
• Review effort level: Lite

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[codecov-commenter]

Codecov Report

❌ Patch coverage is 77.14286% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.67%. Comparing base (04897ba) to head (311a67b).

Files with missing lines	Patch %	Lines
...ynapse/ml/services/search/AzureSearchSchemas.scala	63.15%	14 Missing ⚠️
...e/synapse/ml/services/search/AzureSearchAuth.scala	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2604      +/-   ##
==========================================
+ Coverage   84.03%   84.67%   +0.63%     
==========================================
  Files         335      335              
  Lines       17882    17933      +51     
  Branches     1639     1644       +5     
==========================================
+ Hits        15028    15185     +157     
+ Misses       2854     2748     -106     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).