fix: preserve hand-written Python package initializers

[ranadeepsingh]

Related PRs

• Recreates the still-relevant init-file preservation intent from Fix init file codegen #2286 on current master.
• Preserves the OpenAI compatibility hook introduced by feat: Add v1 OpenAI endpoint support and remove legacy completions API #2560.

Problem

PyCodegen copies hand-written Python overrides and then rewrites nested __init__.py files, discarding required manual content. It also deletes every namespace-root initializer and the prepend approach proposed in #2286 would duplicate generated headers when init generation is repeated.

Before the fix, targeted regression tests reproduced three failures: nested manual UTF-8 content was missing, a non-empty manual namespace root was deleted, and OpenAI manual content was lost around the #2560 hook.

Changes

• Read manual initializer content from the source override tree, never from generated output.
• Emit generated imports in deterministic filename order using explicit UTF-8.
• Compose generated package metadata/imports, initFileExtra, then manual content.
• Keep empty/absent namespace roots as namespace packages while retaining non-empty manual roots.
• Keep the cognitive compatibility initializer entirely hand-written.
• Add focused tests for roots, nested packages, UTF-8, manual presence/absence, OpenAI extras, and repeat-run idempotency.

No files under target/ or other generated output are committed.

Validation

• core/testOnly com.microsoft.azure.synapse.ml.codegen.PyCodegenSuite: 5 passed.
• core/Test/compile and cognitive/Test/compile: passed.
• Canonical core/codegen, cognitive/codegen, and deepLearning/codegen: passed.
• core/scalastyle and core/Test/scalastyle: passed with zero findings.
• black==22.3.0 --check --extend-exclude docs/ .: 179 files unchanged.
• Strict UTF-8 generated-output assertions passed for namespace roots, manual suffixes, OpenAI hook/import ordering, and the cognitive compatibility package.
• Repeated fixed-version core/codegen was byte-idempotent across all 30 generated __init__.py files.

Review

The repository code-review checklist was run before commit; no concrete findings remained.

CI iteration

The first Azure Pipelines run exposed a stale source override at core/src/main/python/synapse/ml/automl/__init__.py: it imported _FindBestModel, but that generated module does not exist. The file had previously been harmless only because PyCodegen overwrote it. The override is now removed so deterministic codegen supplies the valid FindBestModel import. Local targeted tests and canonical core codegen passed after the fix.

Packaging review follow-up

Independent review found that find_namespace_packages(include=["synapse.ml.*"]) omitted a preserved synapse/ml/__init__.py from wheels. Component and aggregate setup templates now include both the exact synapse.ml package and synapse.ml.*. Regression tests build and inspect real fixture wheels for both templates, including UTF-8 root initializer content.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey @ranadeepsingh 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR updates SynapseML’s Python code generation (PyCodegen) to preserve hand-written __init__.py content (including UTF-8 text) while still emitting deterministic generated imports and required OpenAI compatibility glue. It also adds targeted regression tests to validate manual-content retention, namespace-root behavior, ordering, and idempotency across repeated runs.

Changes:

• Refactors PyCodegen.makeInitFiles to read manual initializer content from the source override tree (src/main/python) and compose it after generated metadata/imports (and any initFileExtra).
• Makes generated import emission deterministic (sorted) and explicitly UTF-8 encoded for init-file writes/reads.
• Adds PyCodegenSuite covering UTF-8 preservation, namespace-root deletion/retention rules, OpenAI hook ordering, cognitive manual-only behavior, and repeat-run byte idempotency.

Show a summary per file

File	Description
core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegen.scala	Refactors init-file generation to preserve manual content from the override tree, enforce deterministic ordering, and use explicit UTF-8.
core/src/test/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegenSuite.scala	Adds focused regression tests for manual __init__.py preservation, namespace-root handling, UTF-8, OpenAI hook behavior, and idempotency.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Lite

[codecov-commenter]

Codecov Report

❌ Patch coverage is 96.87500% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 84.80%. Comparing base (7d9fabc) to head (484c24f).

Files with missing lines	Patch %	Lines
...microsoft/azure/synapse/ml/codegen/PyCodegen.scala	96.87%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2590      +/-   ##
==========================================
+ Coverage   84.79%   84.80%   +0.01%     
==========================================
  Files         334      334              
  Lines       17806    17826      +20     
  Branches     1623     1643      +20     
==========================================
+ Hits        15099    15118      +19     
- Misses       2707     2708       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ranadeepsingh]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[ranadeepsingh]

/azp run

[ranadeepsingh]

@imatiach-msft Re-review requested after addressing the namespace-root wheel packaging finding.

Commit 484c24f5ea now includes both synapse.ml and synapse.ml.* in component and aggregate discovery. The new regression coverage builds and inspects both wheel variants and verifies the preserved UTF-8 synapse/ml/__init__.py content.

Local validation: 7/7 targeted PyCodegen tests passed, component/test scalastyle passed with zero findings, Black passed, and canonical core codegen emitted the corrected discovery expression.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).