Change-aware GHDL/cocotb CI: run only affected cocotb tests on feature-branch pushes#1445
Closed
ruck314 wants to merge 1 commit into
Closed
Change-aware GHDL/cocotb CI: run only affected cocotb tests on feature-branch pushes#1445ruck314 wants to merge 1 commit into
ruck314 wants to merge 1 commit into
Conversation
ruck314
force-pushed
the
ci-test-cherry-pick-using-GHDL-cf-file
branch
from
b065526 to
178687e
Compare
ruck314
force-pushed
the
ci-test-cherry-pick-using-GHDL-cf-file
branch
from
18c0927 to
40058ca
Compare
There was a problem hiding this comment.
Pull request overview
Adds a dependency-based resolver to make the GHDL/cocotb regression CI change-aware: feature-branch pushes can run only the cocotb tests affected by RTL (and certain test Python) changes, while main/tags/PRs-to-main continue to run the full suite. The resolver is derived from the actual imported build graph (.cf + ghdl --gen-depends) and fails open to full runs on indeterminate cases.
Changes:
- Introduces a dependency resolver (
tests/common/dep_map.py) plus CLI entrypoint (python -m tests.common) to select affected cocotb tests or emit aFORCE_FULLsentinel. - Adds comprehensive unit tests for the resolver behavior and fail-open invariants (
tests/common/test_dep_map.py). - Updates CI workflow to choose between full vs selective execution and to install an LLVM-backend GHDL for dependency resolution.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
tests/common/dep_map.py |
Implements discovery, dependency extraction, wrapper attribution, and changed-file → affected-test selection logic. |
tests/common/__main__.py |
CLI wrapper around the resolver used by CI to emit targets or FORCE_FULL. |
tests/common/test_dep_map.py |
Unit tests covering resolver correctness, fail-open rules, and wrapper/ip_integrator handling. |
.github/workflows/surf_ci.yml |
Adds PR trigger + selective/full mode selection, installs LLVM GHDL for --gen-depends, and runs selected tests on feature-branch pushes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Run only the cocotb tests whose RTL dependencies actually changed on feature-branch pushes, while tags, main, and PRs into main still run the complete suite. The dependency graph is derived from the actual build (build/*.cf + `ghdl --gen-depends`) at CI time, so it cannot drift from the RTL, and every indeterminate case fails safe toward running more tests. - tests/common/dep_map.py (importable core) + `python -m tests.common` CLI: map each cocotb test (by its toplevel) to its transitive RTL sources via build/*.cf and `ghdl --gen-depends`, diff against the merge-base with origin/main, and print the affected pytest targets (or a FORCE_FULL sentinel). Any indeterminate case falls back to always-run or a full run. - Wrapper / IP-integrator resolution: most cocotb tests wrap their DUT in a **/wrappers/ or **/ip_integrator/ entity (cocotb cannot drive VHDL records), and `make import` never compiles those sources -- so their toplevels are absent from the surf library and `ghdl --gen-depends` cannot resolve them. Import those sources into the resolver's library first (with absolute paths, so the .cf entries stay REPO_ROOT-resolvable) so each such test resolves to its real transitive dependency set instead of falling back to always-run. Subsystems whose core RTL is family-gated out of the generic GHDL build (pgp2b, pgp2fc, coaxpress) remain always-run by fail-safe. - Changed-wrapper attribution: a changed **/wrappers/ file selects exactly the test(s) whose toplevel instantiates it (or FORCE_FULL if unattributable) and never fans out as another test's production dependency. - Shared base-package changes (e.g. StdRtlPkg, AxiLitePkg, AxiStreamPkg, or any high-fan-out hub) force a full run, detected by a data-driven fan-out threshold over the built dependency map. - Strict-subset guarantee: a runtime guard intersects the selection with the discovered test universe before emitting it, and a randomized property test asserts the selected set is always a subset of the full suite. - .github/workflows/surf_ci.yml: trigger on push and pull_request into main (preserving concurrency cancel-in-progress); selective on feature-branch pushes, full on tag/main/PR->main; fail open to the full suite on resolver error. The regression job installs GHDL's official LLVM build via ghdl/setup-ghdl (v6.0.0) for --gen-depends only, and runs the cocotb simulations on the apt mcode ghdl. The lint/syntax job is unchanged. - tests/common/test_dep_map.py: unit and property tests for the parser, the wrapper index, wrapper/ip_integrator source discovery and import, selection routing, base-package force-full, and the strict-subset invariant.
ruck314
force-pushed
the
ci-test-cherry-pick-using-GHDL-cf-file
branch
from
94373fe to
70559db
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Make the GHDL/cocotb regression job change-aware: on ordinary feature-branch pushes it runs only the cocotb tests whose RTL dependencies actually changed, while the release path (tags,
main, and PRs intomain) still runs the complete suite. Previously every push re-ran the entire suite.The dependency graph is derived from the actual build (
build/*.cf+ghdl --gen-depends) at CI time — never a hand-maintained list — so it cannot drift from the RTL. Every indeterminate case fails safe toward running more tests, so selective mode can never skip a test the full suite would have run. Because cocotb cannot drive VHDL records, most tests wrap their DUT in a**/wrappers/or**/ip_integrator/entity thatmake importnever compiles; those sources are imported into the resolver's library first so their toplevels resolve to real dependency sets instead of falling back to always-run. Subsystems whose core RTL is family-gated out of the generic GHDL build (pgp2b, pgp2fc, coaxpress) remain always-run by fail-safe.CI trigger behavior
When the full suite runs vs. only the affected tests:
mainand not a tag)mainrefs/tags/*)mainmainbranch (e.g.pre-release)pull_requesttrigger is scoped tomain; the branch's own push already ran the selective job