Fix the failing CI integration tests#810
Conversation
InvalidDiagnosticException from the solver was not caught by the per-case handler, so one unsolvable case (ilamb amoc-RAPID) aborted the whole fetch loop and left later diagnostics without catalogs and paths sidecars.
The float32 weighted mean differs by ~1e-6 relative between platforms, so baselines minted on macOS never matched Linux CI. Bumps the diagnostic version to 2 and re-mints the baselines.
Platform-level floating-point noise sits around 1e-6 relative, right at the old tolerance. Committed bundles keep 7 significant figures, now two digits finer than the tolerance.
- Fail populate-cache when the fetch crashes instead of masking it - Give the pmp test-cases job 120 minutes - Run the ESMValTool dask cluster as one 8 GiB worker so climate-drivers-for-fire stops killing 4 GiB workers
|
Warning Review limit reached
Next review available in: 24 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe changes adjust CI resource and timeout settings, make test-case fetching continue after per-case failures, relax regression comparison tolerance, and regenerate global SST diagnostic outputs after switching its reduction to float64. ChangesCI and regression updates
Sequence Diagram(s)sequenceDiagram
participant FetchCommand as ref test-cases fetch
participant CatalogBuilding as catalog building
participant WarningLogging as warning logging
FetchCommand->>CatalogBuilding: process each test case
CatalogBuilding-->>FetchCommand: return success or per-case exception
FetchCommand->>WarningLogging: log individual failures
FetchCommand->>WarningLogging: log aggregated failed_cases warning
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 88b62141-f713-401d-9c83-014c20e1047e
📒 Files selected for processing (16)
.github/actions/setup/esmvaltool-config/config.yml.github/workflows/ci-integration.yamlchangelog/810.fix.mddocs/background/regression-baselines.mdpackages/climate-ref-core/src/climate_ref_core/regression/_quantise.pypackages/climate-ref-core/src/climate_ref_core/regression/compare.pypackages/climate-ref-core/tests/unit/regression/test_compare.pypackages/climate-ref-core/tests/unit/regression/test_quantise.pypackages/climate-ref-example/src/climate_ref_example/surface_temperature.pypackages/climate-ref-example/tests/test-data/global-sst-bias/cmip7/manifest.jsonpackages/climate-ref-example/tests/test-data/global-sst-bias/cmip7/regression/diagnostic.jsonpackages/climate-ref-example/tests/test-data/global-sst-bias/cmip7/regression/series.jsonpackages/climate-ref-example/tests/test-data/global-sst-bias/default/manifest.jsonpackages/climate-ref-example/tests/test-data/global-sst-bias/default/regression/series.jsonpackages/climate-ref/src/climate_ref/cli/test_cases/discovery.pypackages/climate-ref/tests/unit/cli/test_test_cases.py
The coupling gate requires a test_case_version bump to authorise a committed bundle change. The diagnostic version bump alone does not. Re-authored via mint --from-replay --bump-version, no blobs changed.
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
A 5e-7 difference also passed under the old 1e-6 default. Use 5e-6, which sits between the old and new tolerance, so the test proves the new contract.
1 participant
Description
The nightly CI Integration Tests workflow has been red for several unrelated reasons (run 29109525132).
ref test-cases fetchaborting the whole loop when the solver finds no matching datasets for one case (ilambamoc-RAPIDraisesInvalidDiagnosticException). Later diagnostics were left without paths sidecars, which cascaded intoDatasetResolutionErroracross all the test-cases jobs. The workflow no longer masks a non-zero exit from the fetch step.global-sst-biasexample diagnostic to accumulate its global means in float64. The float32 reduction differs by ~1e-6 between platforms, so baselines minted on macOS never matched Linux CI.Diagnostic.versionbumps to 2 and the baselines were re-minted, the series values move at the 6th decimal place.rtolfrom 1e-6 to 1e-5 so platform floating-point noise cannot trip the gate. Committed bundles keep 7 significant figures, which is now two digits finer than the tolerance.climate-drivers-for-firestops killing 4 GiB workers.Checklist
Please confirm that this pull request has done the following:
changelog/Summary by CodeRabbit
Bug Fixes
CI Improvements