Skip to content

Add requirements-reference.txt for reproducible Python deps#749

Closed
PranjalManhgaye wants to merge 95 commits into
precice:developfrom
PranjalManhgaye:fix-610-python-dependency-versions
Closed

Add requirements-reference.txt for reproducible Python deps#749
PranjalManhgaye wants to merge 95 commits into
precice:developfrom
PranjalManhgaye:fix-610-python-dependency-versions

Conversation

@PranjalManhgaye

@PranjalManhgaye PranjalManhgaye commented Mar 8, 2026

Copy link
Copy Markdown
Collaborator

Description

Problem
For each distribution release we need reproducible Python dependency versions, but pinning in every tutorial requirements.txt would require a lot of merge back-and-forth. There is also no explicit list of versions known to work for a given release.

Approach
Introduce a separate requirements-reference.txt (lockfile-style) with pinned versions, without changing the existing tutorial requirements.txt or run scripts. The reference file is only a manifest of versions that are known to work for a distribution.

What changed

  • Add tools/tests/requirements-reference.txt with pinned versions (pyprecice, numpy, matplotlib, nutils, setuptools)
  • Add update_requirements_reference.py to regenerate it from reference_versions.yaml (pyprecice from PYTHON_BINDINGS_REF)
  • Add validate_requirements_reference.py to ensure the pyprecice version matches reference_versions.yaml
  • Add GitHub Action check-requirements-reference.yml to run this validation on PR/push
  • Document the mechanism in tools/tests/README.md and add a reminder to the release PR template

How to update at release

  • Preferred: pip freeze from the systemtest Docker image, then pip freeze | python3 update_requirements_reference.py --from-freeze
  • Or: python3 update_requirements_reference.py (uses reference_versions.yaml for pyprecice and script defaults for other packages)

Fixes #610


Checklist:

  • I added a summary of any user-facing changes (compared to the last release) in the changelog-entries/<PRnumber>.md.
  • I will remember to squash-and-merge, providing a useful summary of the changes of this PR.

@MakisH MakisH added the GSoC Contributed in the context of the Google Summer of Code label Mar 13, 2026
@MakisH MakisH changed the title Add requirements-reference.txt for reproducible Python deps (fixes #610) Add requirements-reference.txt for reproducible Python deps May 9, 2026
…ecice#610)

- Add tools/tests/requirements-reference.txt with pinned versions
- Add update_requirements_reference.py to regenerate from reference_versions.yaml
- Add validate_requirements_reference.py to check pyprecice matches
- Add GitHub Action check-requirements-reference.yml
- Document in tools/tests/README.md and release PR template
marinlauber and others added 24 commits May 25, 2026 22:22
It is installed automatically and we do not specify such versions for other solvers.
…ve error message (precice#794)

Co-authored-by: preCICE Tests VM <tests@precice.org>
…e_test (precice#798)

Co-authored-by: preCICE Tests VM <tests@precice.org>
* Add partitioned-pipe to the system tests
* Add partitioned-pipe-two-phase to the system tests
* Add partitioned-pipe-multiscale to the system tests
* Add missing venv checks in partitioned-pipe-multiscale
MakisH and others added 23 commits June 8, 2026 08:50

@MakisH MakisH left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for contributing. I think the idea of having a script to generate requirements-reference.txt is indeed one possible solution to #610. However, I don't see the connection to reference_versions.yaml, which is about other dependencies than the requirements.txt.

Note that #610 mainly concerns the various Python-based tutorials that have requirements.txt. The system tests scripts are not that interesting for reproducibility: we mainly need the latest version to work. But when someone needs a older version of a Python-based tutorial, it is important that they can reproduce the original dependencies.

I think we need such a script to report the installed requirements.txt at a given time, on demand, plus some more version restrictions in the each requirements.txt. These version restrictions should state the requirements, without enforcing specific versions, so that the latest supported can generally be installed.

See how many tutorials require numpy >1 <2, or precice~=3.0.

Note that we should be able to get the latest available packages on PyPI, not just the ones that PIP might have cached on the local system.

Would you still like to continue working on this PR, or would you prefer to start a new one?

@PranjalManhgaye

PranjalManhgaye commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator Author

@MakisH for now i am contnuing on this pr only and work with your given direction ,
i am reworking it to focus on tutorial requirements.txt (looser constraints) plus an on-demand tools/report_tutorial_requirements.py that resolves versions from PyPI into root-level requirements-reference.txt , without tying it to reference_versions.yaml ,,

Add loose constraints to tutorial requirements.txt files, PyPI-based
report_tutorial_requirements.py, and root requirements-reference.txt.
@PranjalManhgaye

Copy link
Copy Markdown
Collaborator Author

Closing in favor of a new PR with the reworked implementation (rebased onto latest develop, tutorial-centric approach per review) ,,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

GSoC Contributed in the context of the Google Summer of Code

Projects

Development

Successfully merging this pull request may close these issues.

Fix versions of Python dependencies for each distribution

6 participants