[tests] Skip macOS/Mac Catalyst tests on mismatched beta macOS versions#25829
[tests] Skip macOS/Mac Catalyst tests on mismatched beta macOS versions#25829rolfbjarne wants to merge 2 commits into
Conversation
…cOS beta version. Add EXPECTED_MACOS_BUILD_VERSION to Make.config (currently set to 26A5368g for macOS 27 beta 1). The run-packaged-macos-tests script checks the current macOS build version via kern.osversion sysctl, and exits early with success if it doesn't match the expected version. This prevents spurious test failures when CI machines are running a different beta than what the tests target. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… match. Stable macOS versions (build version ending in a digit, like '24G720') should still run tests even if they don't match the expected beta build version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
✅ [PR Build #3939be2] Build passed (Detect API changes) ✅Pipeline on Agent |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
✅ API diff for current PR / commitNET (empty diffs)✅ API diff vs stableNET (empty diffs)ℹ️ Generator diffGenerator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes) Pipeline on Agent |
✅ [PR Build #3939be2] Build passed (Build macOS tests) ✅Pipeline on Agent |
This comment has been minimized.
This comment has been minimized.
🚀 [CI Build #3939be2] Test results 🚀Test results✅ All tests passed on VSTS: test results. 🎉 All 200 tests passed 🎉 Tests counts✅ assembly-processing: All 1 tests passed. Html Report (VSDrops) Download macOS tests✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download Linux Build VerificationPipeline on Agent |
There was a problem hiding this comment.
Pull request overview
This PR adds a CI-friendly mechanism to avoid spurious macOS/Mac Catalyst test failures when build agents are running a different macOS beta than the branch is targeting, by propagating an expected macOS build version into the packaged test runner and conditionally skipping test execution on mismatched beta builds.
Changes:
- Add
EXPECTED_MACOS_BUILD_VERSIONtoMake.configas the branch’s targeted macOS build version. - Teach
run-packaged-macos-testsa new--expected-macos-build-versionoption and skip tests when running on a mismatched beta build. - Update the mac CI PowerShell runner to read the value from
Make.configand pass it through to the test runner.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tools/devops/automation/scripts/run_mac_tests.ps1 | Reads EXPECTED_MACOS_BUILD_VERSION from packaged Make.config and forwards it to the test runner. |
| scripts/run-packaged-macos-tests/run-packaged-macos-tests.cs | Adds CLI arg + beta build-version check to skip running packaged mac tests on mismatched beta OS builds. |
| Make.config | Introduces EXPECTED_MACOS_BUILD_VERSION and documents how to obtain the value. |
| if (isBeta && currentBuildVersion != expectedMacOSBuildVersion) { | ||
| Console.WriteLine ($"Current macOS build version '{currentBuildVersion}' is a beta that does not match expected '{expectedMacOSBuildVersion}'. Skipping tests."); | ||
| return 0; | ||
| } |
| // [--timeout <seconds>] Default timeout per test in seconds (default: 300) | ||
| // [--timeout-longer <seconds>] Longer timeout for heavy tests (default: 600) | ||
| // [--launch-arguments <args>] Extra arguments passed to test executables | ||
| // [--expected-macos-build-version <v>] Expected macOS build version (skip tests if mismatched) |
|
|
||
| # The expected macOS build version for running tests. | ||
| # To get the build version for the current macOS, run: sysctl -n kern.osversion | ||
| # If the running macOS build version doesn't match, macOS/Mac Catalyst tests will be skipped. |
✅ [PR Build #3939be2] Build passed (Build packages) ✅Pipeline on Agent |
When the xcode27.0 branch targets a specific macOS beta (e.g. beta 1), CI machines may be running a different beta version, causing spurious test failures. This PR adds a mechanism to skip macOS and Mac Catalyst tests when the running macOS is a beta that doesn't match the expected one.
Approach:
EXPECTED_MACOS_BUILD_VERSIONvariable inMake.configstores the build version we target (currently26A5368gfor macOS 27 beta 1). A comment explains how to get the value (sysctl -n kern.osversion).run-packaged-macos-testsscript accepts--expected-macos-build-versionand checkskern.osversionat startup. If the running OS is a beta (build version ends with a lowercase letter) and doesn't match, it exits 0 (skipping tests gracefully).Make.configand passes it through.🤖 Pull request created by Copilot