feat: Support lossless string identifiers in SAR

[ranadeepsingh]

Context

Closes #2275.

This is a current-master replacement for the intent of #2283. The original PR is intentionally left untouched; its direct string-to-integer casts were not reused because nonnumeric and wide identifiers could not round-trip safely.

What changed

• Build deterministic, contiguous user and item indices while fitting SAR.
• Persist reversible, typed originalID/index mappings as model-owned DataFrameParams.
• Keep matrix calculations on internal indices and decode caller-visible scores and recommendations safely.
• Reject null identifiers during training; drop null or unseen scoring/subset identifiers.
• Preserve string and wide/fractional numeric IDs in recommendation outputs while retaining the established integer schema for numeric IDs that round-trip through Int.
• Retain safe identity-mapping fallback behavior for models saved before mapping params existed.
• Add typed all-user, user-subset, all-item, and item-subset paths and Python wrappers.
• Correct item-to-user matrix multiplication and deterministic ranking.

Independent review fixes

The first follow-up review findings are addressed in 15d0be60c0:

1. RankingTrainValidationSplit.splitDF uses typed Spark structs, arrays, slicing, and exploding in both rated and unrated branches; string IDs are no longer coerced through Double UDFs.
2. Numeric scoring schemas may differ from training when each value safely casts to the mapping type and round-trips back unchanged. Fractional aliases are dropped as unknown; string schemas still require exact type compatibility.
3. Existing recommendation APIs retain integer ID schemas when all numeric IDs safely round-trip through Int; strings and non-round-tripping numeric IDs remain lossless.
4. Top-K ranking considers only real destination indices, so zero-filled gaps in legacy matrices cannot consume recommendation slots.

The second follow-up review findings are addressed in 622fa446d3:

1. Numeric compatibility and legacy identity mappings use Spark SQL try_cast in both directions, so wide values are filtered rather than throwing CAST_OVERFLOW when spark.sql.ansi.enabled=true.
2. Whether user/item IDs round-trip through IntegerType is computed once during fitting, persisted with the model, and given conservative defaults for legacy models. Recommendation planning no longer scans mapped-model mappings or collects their destination indices; index collection remains only for mapping-less legacy models.

The reviews also prompted removal of an unmanaged mapping cache/materialization, replacement of an unused grouped count with distinct, and alignment of the Python recommendForAllItems(numItems=...) keyword.

API and design notes

SAR owns the mappings rather than requiring RecommendationIndexer. Composing that stage would stringify numeric values, expose index columns to pipeline users, and still be unable to recover every original Spark type. Existing Scala method signatures are retained, including the legacy numItems parameter name on recommendForAllItems.

The deterministic global mapping sort and serialized mapping data are deliberate costs for reproducible save/load behavior. SAR already materializes its interaction structures on the driver, so this does not introduce a new distributed-to-driver boundary in the core algorithm.

Validation

• Five targeted recommendation suites: 34/34 passed (SARIdentifierSpec, SARSpec, RankingTrainValidationSpec, RankingEvaluatorSpec, and RecommendationIndexerSpec)
• Final focused SARIdentifierSpec: 13/13 passed
• Repository production and test Scalastyle checks passed
• core / codegen
• core / Compile / packageBin
• black==22.3.0 --check --extend-exclude 'docs/' .: 179 files passed
• Packaged-JAR Python/JVM smoke passed for string IDs, ANSI-enabled compatible numeric scoring, legacy flag defaults, integer/wide output schemas, all/subset APIs, keyword wrappers, and save/load
• Final repository code-review checklist: no concrete findings

Regression coverage

• Rated and unrated ranking splits preserve string user/item schemas and all rows
• Numeric scoring accepts safe widening/narrowing and rejects fractional or out-of-range aliases, including under ANSI mode
• Existing integer recommendation schemas remain compatible where lossless
• Gapped legacy factor IDs cannot consume top-K slots
• Mapped recommendation planning does not execute mapping scans or collect mapped destination indices
• Compatibility flags and typed mappings survive model save/load
• Direct string and wide numeric training/scoring/recommendations
• Deterministic mappings, unknown/null semantics, transformSchema, and Python wrappers

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey @ranadeepsingh 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Adds lossless support for string and wide numeric user/item identifiers in the SAR recommender by introducing deterministic, model-owned ID↔index mappings, and ensuring all caller-visible outputs decode back to the original identifier types.

Changes:

• Build and persist deterministic (contiguous) ID mappings during SAR fitting; run all internal computations on indices and decode outputs back to original identifier types.
• Extend SARModel recommendation APIs (including item-subset recommendations) and add corresponding Python wrappers.
• Add Scala/Python test coverage for direct string IDs, wide numeric IDs, mapping determinism, and save/load behavior.

Show a summary per file

File	Description
core/src/main/scala/com/microsoft/azure/synapse/ml/recommendation/SAR.scala	Implements deterministic ID mapping and updates SAR fit/schema validation to support string/numeric identifiers.
core/src/main/scala/com/microsoft/azure/synapse/ml/recommendation/SARModel.scala	Adds model-owned mappings, decoding, subset recommendation support, and deterministic ranking.
core/src/main/python/synapse/ml/recommendation/SARModel.py	Exposes additional SARModel recommendation methods to Python.
core/src/test/scala/com/microsoft/azure/synapse/ml/recommendation/SARSpec.scala	Adjusts existing SAR tests for deterministic/sorted DataFrame equality and updated identifier handling.
core/src/test/scala/com/microsoft/azure/synapse/ml/recommendation/SARIdentifierSpec.scala	New Scala test suite validating identifier type preservation, determinism, and save/load.
core/src/test/python/synapsemltest/recommendation/test_ranking.py	Adds Python tests for direct string identifier training/scoring and save/load.

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 4
• Review effort level: Lite

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.27793% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.95%. Comparing base (7d9fabc) to head (622fa44).

Files with missing lines	Patch %	Lines
...icrosoft/azure/synapse/ml/recommendation/SAR.scala	86.23%	19 Missing ⚠️
...oft/azure/synapse/ml/recommendation/SARModel.scala	98.97%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2594      +/-   ##
==========================================
+ Coverage   84.79%   84.95%   +0.15%     
==========================================
  Files         334      334              
  Lines       17806    18056     +250     
  Branches     1623     1609      -14     
==========================================
+ Hits        15099    15339     +240     
- Misses       2707     2717      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ranadeepsingh]

Review fixes are ready in 15d0be60c0 and the PR description now documents each resolution.

• typed rated/unrated ranking splits preserve string IDs
• numeric scoring casts require value-level round trips
• established integer recommendation schemas remain compatible when lossless
• mapped candidates are filtered before top-K

Focused regressions were added; 35/35 targeted Scala tests, codegen/package, repository style, Black, and the final Python/JVM smoke pass. A separate read-only correctness review found no remaining issues. Re-review requested.

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[ranadeepsingh]

Second follow-up review fixes are now in 622fa44: ANSI-safe try_cast handling prevents CAST_OVERFLOW, and persisted compatibility flags remove mapped-model mapping scans/index collection while preserving legacy gap filtering. Local validation passed (34/34 targeted Scala tests, 13/13 final SAR identifier tests, codegen/package, both Scalastyle checks, repository-wide Black, and packaged-JAR Python/JVM smoke). @eisber, please re-review when available.

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).