Introduce a consistent version management for all simtools dependencies#2376
Merged
Conversation
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
43 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a centralized “dependency version catalog” in pyproject.toml and adds tooling to (1) export those declared versions for CI/Docker builds and (2) capture a runtime “dependency manifest” for provenance. It then rewires Dockerfiles and GitHub Actions workflows to consume these exported values, and extends sim_telarray metadata to embed provenance pointers.
Changes:
- Added
simtools.dependency_versions+simtools-dependency-versionsto validate/export build matrices and version scalars frompyproject.toml. - Added dependency provenance manifest generation (
simtools-dependency-manifest, manifest schema, digest sidecar) and wired it into container builds + metadata export. - Updated Dockerfiles, workflows, and documentation to use the catalog as the single source of truth.
Reviewed changes
Copilot reviewed 41 out of 41 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit_tests/test_dependency_versions.py | Adds unit tests for catalog loading/validation, matrix generation, and exports. |
| tests/unit_tests/test_dependencies.py | Adds unit tests for dependency manifest reading/building/digesting and build-info export changes. |
| tests/unit_tests/configuration/test_argument_helpers.py | Updates build-info CLI action test to assert logging-based output. |
| tests/unit_tests/applications/test_dependency_versions_application.py | Tests the new dependency-versions CLI application behavior. |
| tests/unit_tests/applications/test_dependency_manifest.py | Tests the new dependency-manifest CLI application behavior. |
| tests/unit_tests/application/test_control.py | Adjusts version-info test behavior when build options are missing. |
| src/simtools/simtel/simtel_config_writer.py | Adds dependency provenance metadata and filters which build options become sim_telarray metadata. |
| src/simtools/schemas/sim_telarray_meta_parameters.schema.yml | Registers new provenance-related metadata keys. |
| src/simtools/schemas/dependency_versions.schema.yml | Introduces a schema for the dependency version catalog. |
| src/simtools/schemas/dependency_manifest.schema.yml | Introduces a schema for the dependency provenance manifest. |
| src/simtools/dependency_versions.py | Implements catalog discovery/validation/export, matrices, and requirements export. |
| src/simtools/dependencies.py | Implements manifest building/reading/digesting + richer build-info export. |
| src/simtools/configuration/argument_helpers.py | Switches build-info action output from print to logger + dependency summary. |
| src/simtools/applications/dependency_versions.py | Adds simtools-dependency-versions application wrapper with standalone mode. |
| src/simtools/applications/dependency_manifest.py | Adds simtools-dependency-manifest application to write provenance manifests. |
| src/simtools/application/control.py | Continues version-info logging even if build options file is missing. |
| pyproject.toml | Adds scripts, adds packaging, and adds [tool.gammasimtools.dependency-versions] catalog content. |
| MANIFEST.in | Includes *.yml under src so schemas ship in sdists. |
| docs/source/user-guide/applications/simtools-dependency-versions.md | Documents the new dependency-versions application. |
| docs/source/user-guide/applications/simtools-dependency-manifest.md | Documents the new dependency-manifest application. |
| docs/source/user-guide/applications.md | Adds both new applications to the user-guide toctree. |
| docs/source/developer-guide/simtools_build_images.md | Updates image-build guidance to use the catalog exporter. |
| docs/source/developer-guide/release.md | Adds release-candidate guidance around validating the catalog/immutability. |
| docs/source/developer-guide/index.md | Adds developer-guide entry for dependency versions/provenance docs. |
| docs/source/developer-guide/dependency_versions.md | New developer documentation on catalog vs manifest and workflows. |
| docs/source/data-model/sim_telarray_metadata.md | Documents the new software-provenance metadata fields and intent. |
| docs/source/api-reference/dependencies.md | Adds API reference entry for simtools.dependency_versions. |
| docker/README.md | Updates documentation links for container build + dependency version/provenance docs. |
| docker/Dockerfile-simtools-prod | Removes embedded version defaults; installs from checkout + writes manifest during build. |
| docker/Dockerfile-simtools-dev | Uses catalog-exported requirements; generates dev manifest in-image. |
| docker/Dockerfile-simtel_array | Parameterizes versions/URLs/revisions/checksums; records build provenance in build_opts. |
| docker/Dockerfile-corsika7 | Parameterizes versions/URLs/revisions/checksums; records provenance in build_opts. |
| .github/workflows/CI-software-update.yml | Renames automated update branch/title/commit metadata. |
| .github/workflows/CI-linter.yml | Validates dependency catalog summary output; adds pip check. |
| .github/workflows/CI-integrationtests.yml | Reads interaction-tables version from the catalog export. |
| .github/workflows/build-simtools-prod.yml | Introduces dependency-config job and drives build args/matrices from catalog export; adds provenance artifact. |
| .github/workflows/build-simtools-dev.yml | Introduces dependency-config job and drives dev image build args from catalog export. |
| .github/workflows/build-sim_telarray.yml | Introduces dependency-config job and drives sim_telarray builds from catalog export. |
| .github/workflows/build-corsika7.yml | Introduces dependency-config job; downloads/validates auxiliary artifacts; separates CORSIKA source acquisition. |
| .env_template | Aligns default model version formatting with catalog conventions (no leading v). |
| .dockerignore | Adds repository-level dockerignore for leaner build contexts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
GernotMaier
marked this pull request as ready for review
July 24, 2026 12:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce a consistent version management for all simtools dependencies, meaning all dependency versions are read from the pyrpoject.toml file (relevant part below).
This makes changes easier - as right now we have defined these dependencies in Dockerfiles, github Workflows, Code, environment files, pyproject.toml, etc... - essentially uniformly distributed across the code.
Extract from the pyproject.toml:
There is a new application to retrieve the dependency - used e.g., but the Github runners, but can also be used to dump all the dependencies to the screen: