Add requirements-reference.txt for reproducible Python deps#749
Add requirements-reference.txt for reproducible Python deps#749PranjalManhgaye wants to merge 95 commits into
Conversation
…ecice#610) - Add tools/tests/requirements-reference.txt with pinned versions - Add update_requirements_reference.py to regenerate from reference_versions.yaml - Add validate_requirements_reference.py to check pyprecice matches - Add GitHub Action check-requirements-reference.yml - Document in tools/tests/README.md and release PR template
b88bfcc to
19bd1cb
Compare
It is installed automatically and we do not specify such versions for other solvers.
…ve error message (precice#794) Co-authored-by: preCICE Tests VM <tests@precice.org>
…e_test (precice#798) Co-authored-by: preCICE Tests VM <tests@precice.org>
* Add partitioned-pipe to the system tests * Add partitioned-pipe-two-phase to the system tests * Add partitioned-pipe-multiscale to the system tests * Add missing venv checks in partitioned-pipe-multiscale
There was a problem hiding this comment.
Thank you for contributing. I think the idea of having a script to generate requirements-reference.txt is indeed one possible solution to #610. However, I don't see the connection to reference_versions.yaml, which is about other dependencies than the requirements.txt.
Note that #610 mainly concerns the various Python-based tutorials that have requirements.txt. The system tests scripts are not that interesting for reproducibility: we mainly need the latest version to work. But when someone needs a older version of a Python-based tutorial, it is important that they can reproduce the original dependencies.
I think we need such a script to report the installed requirements.txt at a given time, on demand, plus some more version restrictions in the each requirements.txt. These version restrictions should state the requirements, without enforcing specific versions, so that the latest supported can generally be installed.
See how many tutorials require numpy >1 <2, or precice~=3.0.
Note that we should be able to get the latest available packages on PyPI, not just the ones that PIP might have cached on the local system.
Would you still like to continue working on this PR, or would you prefer to start a new one?
|
@MakisH for now i am contnuing on this pr only and work with your given direction , |
Add loose constraints to tutorial requirements.txt files, PyPI-based report_tutorial_requirements.py, and root requirements-reference.txt.
|
Closing in favor of a new PR with the reworked implementation (rebased onto latest develop, tutorial-centric approach per review) ,, |
Description
Problem
For each distribution release we need reproducible Python dependency versions, but pinning in every tutorial
requirements.txtwould require a lot of merge back-and-forth. There is also no explicit list of versions known to work for a given release.Approach
Introduce a separate
requirements-reference.txt(lockfile-style) with pinned versions, without changing the existing tutorialrequirements.txtor run scripts. The reference file is only a manifest of versions that are known to work for a distribution.What changed
tools/tests/requirements-reference.txtwith pinned versions (pyprecice, numpy, matplotlib, nutils, setuptools)update_requirements_reference.pyto regenerate it fromreference_versions.yaml(pyprecice from PYTHON_BINDINGS_REF)validate_requirements_reference.pyto ensure the pyprecice version matchesreference_versions.yamlcheck-requirements-reference.ymlto run this validation on PR/pushtools/tests/README.mdand add a reminder to the release PR templateHow to update at release
pip freezefrom the systemtest Docker image, thenpip freeze | python3 update_requirements_reference.py --from-freezepython3 update_requirements_reference.py(usesreference_versions.yamlfor pyprecice and script defaults for other packages)Fixes #610
Checklist:
changelog-entries/<PRnumber>.md.