Skip to content

feat: make Python module discovery cwd-aware#9

Merged
grouville merged 3 commits into
dagger:mainfrom
grouville:feat/cwd-aware-discovery
Jul 21, 2026
Merged

feat: make Python module discovery cwd-aware#9
grouville merged 3 commits into
dagger:mainfrom
grouville:feat/cwd-aware-discovery

Conversation

@grouville

@grouville grouville commented Jul 21, 2026

Copy link
Copy Markdown
Member

Context

The Python SDK helper discovers modules by globbing every dagger.json from the workspace root. That predates workspaces and creates three problems:

  1. discovery ignores the caller's current working directory;
  2. modern dagger-module.toml modules are invisible;
  3. the filesystem scan can disagree with the workspace's modules.python-sdk.as-sdk.modules list.

This PR makes Python SDK discovery follow the cwd-aware convention from dagger/dagger#13688.

Behavior after this change

modules(ws) asks polyfill for config directories visible from the caller cwd, converts those cwd-relative paths to workspace paths, and intersects them with ws.sdk(name: currentModule.name).modules.

Caller location Modules visible to modules / generateAll
Workspace root Managed Python modules at or below the root
A module root That module and managed descendants
Inside a module The nearest enclosing managed module plus managed descendants
A sibling directory Not included

Both dagger-module.toml and legacy dagger.json participate in discovery. Virtual environments and installed packages are excluded.

mod(ws, path) also learns both config formats. It compares both findUp results and validates the nearest module's runtime, so filename order cannot select a farther config.

Internally, Mod.rootPath is workspace-root-relative and is used for generation and configuration. The public Mod.path is relative to the caller cwd. Keeping the two coordinate systems explicit prevents clients from resolving a cwd-relative path as though it were rooted at the workspace.

Workspace ownership

The injected Workspace! argument is authoritative throughout discovery. Reading the managed module list through ws.sdk(...) is intentional: current engines no longer inherit an ambient workspace into module dependencies (dagger/dagger#13229), so currentModule.asSDK fails from @check execution with workspace not loaded.

This preserves the previous data model: WorkspaceModule.source is the same workspace-root-relative module path used for the intersection.

Scope

  • Changes module discovery and generateAll scoping.
  • Adds modern TOML support to mod lookup.
  • Pins fix: select the nearest enclosing config, not the first filename's polyfill#4, which selects the nearest enclosing config across mixed filenames.
  • Does not change template generation or Python build configuration behavior.
  • Keeps the workspace-managed SDK list authoritative; finding a config file alone does not make it a managed Python module.
  • Removes the stale initClientCheck left after PythonSdk.initClient was removed, allowing the rebased e2e module to load.

Reviewer guide

The implementation is concentrated in:

  • python-sdk.dang: cwd discovery, path normalization, explicit workspace SDK lookup, and generateAll;
  • mod.dang: separation of rootPath from cwd-relative path;
  • .dagger/modules/e2e/fixtures/toml: explicit lookup where a nearer TOML must beat the repository's outer JSON;
  • .dagger/modules/e2e/fixtures/mixed-discovery: shared discovery where a nearer JSON must beat an ancestor TOML;
  • .dagger/modules/e2e/main.dang: focused lookup and discovery assertions.

The reverse mixed-format fixture is:

ancestor/
├── dagger-module.toml             # farther; searched first
└── work/app/
    ├── dagger.json                # nearer managed Python module
    └── nested/deeper/             # discovery starts here

The expected module is app, with public path ../.. from nested/deeper.

Reproduce locally

Run the focused checks from the repository root:

dagger check e-2-e:mixed-config-lookup-check e-2-e:module-discovery-check e-2-e:generate-all-check

Inspect cwd-relative output from inside a checked-in legacy module:

cd .dagger/modules/e2e/fixtures/lookup/app/nested
dagger call python-sdk modules path

Expected output:

..

Verification

  • dagger check: 11/11 against a locally rebuilt main engine.
  • dagger --x-release=v1.0.0-beta.7 check: 11/11.
  • dagger-dogfood CI: load and all 11 checks pass.

@TomChv TomChv left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, I fixed dagger/polyfill#2 so just update your polyfill dep and merge the PR.

I tested it and it worked as expected.

@grouville
grouville force-pushed the feat/cwd-aware-discovery branch from 41b296b to 94548db Compare July 21, 2026 21:56
@grouville

grouville commented Jul 21, 2026

Copy link
Copy Markdown
Member Author

Resolved. The failing checks were exposing a workspace-context compatibility change, not a problem with the mixed-config fixture.

Current dogfood/main engines include dagger/dagger#13229, which stops inheriting the caller's ambient workspace into module dependencies. currentModule.asSDK reads that ambient workspace, so modules and generateAll failed under @check with:

get current workspace: no current workspace: workspace not loaded

The injected ws: Workspace! was loaded correctly. Commit 1cabf0f now resolves the managed SDK through that explicit value:

ws.sdk(name: currentModule.name).modules

This is the same migration used by dagger/typescript-sdk#9. WorkspaceModule.source remains workspace-root-relative, so the discovery behavior does not change.

The local A/B reproduced the CI boundary:

Engine / code Result
Published beta.7, before fix Pass
Locally rebuilt main, before fix workspace not loaded
Locally rebuilt main, after fix 11/11 pass
Published beta.7, after fix 11/11 pass
Dogfood CI, after fix load + 11/11 pass

The earlier pin-only isolation still failed on dogfood, proving the new fixture was not the cause. The mixed-format regression is therefore restored in d644f30; it covers a nearer dagger.json below a farther dagger-module.toml and expects ../.. from two levels inside the managed module.

Focused dogfood results: module discovery, generate all, mixed config lookup.

@grouville
grouville force-pushed the feat/cwd-aware-discovery branch 2 times, most recently from 739bd70 to 10a793c Compare July 21, 2026 22:10
Workspace inheritance into module dependencies is disabled on current engines.
Reading currentModule.asSDK therefore depends on ambient state that is absent
when this SDK runs from an @check function.

Resolve the installed SDK from the explicit Workspace argument instead. Its
module source is the same workspace-root-relative path previously exposed as
the current module SDK path, so discovery semantics remain unchanged.

Signed-off-by: Guillaume de Rouville <guillaume@dagger.io>
Polyfill previously selected the first config filename found while walking up,
so mixed dagger.json and dagger-module.toml layouts could resolve to a farther
module based on filename order.

Pin the dagger/polyfill#4 fix and cover a managed Python module whose nearby
dagger.json is nested below an ancestor dagger-module.toml. Discovery from two
levels inside the module must select the nearby module and return "../..".

Signed-off-by: Guillaume de Rouville <guillaume@dagger.io>
@grouville
grouville force-pushed the feat/cwd-aware-discovery branch from 279e14f to d644f30 Compare July 21, 2026 22:40
@grouville
grouville merged commit 5ee93fe into dagger:main Jul 21, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants