Skip to content

Add GENERATED package lock mode (pnpm/npm style, commit-pinned)#707

Open
jamierpond wants to merge 2 commits into
cpm-cmake:masterfrom
jamierpond:feat/generated-package-lock
Open

Add GENERATED package lock mode (pnpm/npm style, commit-pinned)#707
jamierpond wants to merge 2 commits into
cpm-cmake:masterfrom
jamierpond:feat/generated-package-lock

Conversation

@jamierpond

Copy link
Copy Markdown

Motivation

The current package lock records each dependency's declaration verbatim and only updates when you build the cpm-update-package-lock target. For projects that declare deps against a moving ref (e.g. GIT_TAG main), this has two rough edges compared to pnpm-lock.yaml / package-lock.json:

  1. The lock isn't reproducible — it records GIT_TAG main, not the commit that was actually used.
  2. It doesn't regenerate on its own. Delete it and reconfigure and it stays gone until you remember the separate target step.

What this adds

A GENERATED keyword on CPMUsePackageLock (and an equivalent CPM_GENERATE_PACKAGE_LOCK option / env var):

CPMUsePackageLock(package-lock.cmake GENERATED)

In this mode:

  • Auto-maintained: the lock at the given path is rewritten in the source tree on every configure — no cpm-update-package-lock step. Delete it and reconfigure to refresh the pins, just like deleting a pnpm lock.
  • Commit-pinned: git packages are pinned to the exact commit that was checked out, not the declared (possibly moving) GIT_TAG. The pin is then consumed on the next configure (declarations override call-site args), so it's stable/idempotent: main → resolved commit on the first run, identical lock thereafter.

Existing (non-GENERATED) behaviour is unchanged.

Implementation notes

  • The lock entry is now written after the fetch instead of before it, so the package source dir exists and its commit can be resolved. This is the only behavioural move for the existing path; the same set of packages is recorded as before.
  • New helper cpm_get_git_commit_hash() reads the checked-out HEAD (falls back gracefully — no git / not a repo / URL packages → unchanged args).
  • CPMUsePackageLock now cmake_parse_arguments for the GENERATED flag, includes any existing lock first (to apply current pins), then redirects CPM_PACKAGE_LOCK_FILE at the source path and rewrites it.

Tests

Adds test/unit/package-lock-generated.cmake — hermetic (spins up a throwaway local git repo, no network) — asserting that after a plain configure (no target build) the lock is written into the source tree, pins the dependency to the resolved commit (and no longer references the branch), and is byte-identical across a reconfigure.

Formatted with the repo's cmake-format 0.6.11 config; existing package-lock unit tests still pass.

`CPMUsePackageLock(<file> GENERATED)` (or the `CPM_GENERATE_PACKAGE_LOCK`
option) keeps the lock in sync automatically, the way `pnpm-lock.yaml` /
`package-lock.json` do:

- The lock is (re)written into the source tree on every configure, so there is
  no separate `cpm-update-package-lock` step. Delete it and reconfigure to
  refresh, just like deleting a pnpm lock.
- Git packages are pinned to the exact commit that was checked out rather than
  the (possibly moving) GIT_TAG, making the lock reproducible even when deps are
  declared against a branch like `main`.

To resolve the commit, the lock entry is now written after the fetch (so the
source dir exists) and `cpm_get_git_commit_hash` reads the checked-out HEAD.
Non-generated behaviour is unchanged.

Adds a hermetic unit test (local throwaway git repo, no network) covering
auto-generation, commit pinning, and idempotency, plus README docs.

@TheLartians TheLartians left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR, automatically locking the current commit hash is a big step for CPM's supply chain security and reproducibility - especially when moving targets are specified! I have some thoughts regarding using a auto-updating package lock in nested CPM dependencies and documentation, but looks great otherwise!

Comment thread cmake/CPM.cmake
Comment on lines +1088 to +1091
find_package(Git QUIET)
if(NOT GIT_EXECUTABLE OR NOT EXISTS "${repoPath}/.git")
return()
endif()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case of nested CPM dependencies, the value of ${result} may still be set in the parent scope leading to an incorrect resolution, so we should unset ${result} in the parent scope here.

Comment thread cmake/CPM.cmake Outdated
Comment on lines +1125 to +1126
# Includes the package lock file if it exists and creates a target `cpm-update-package-lock` to
# update it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docstring is no longer up-to-date: the cpm-update-package-lock target is no longer created when GENERATED or CPM_GENERATE_PACKAGE_LOCK is set.

Comment thread cmake/CPM.cmake
Comment on lines +1144 to +1156
# Author the lock directly in the source tree and keep it current on every configure.
set(CPM_PACKAGE_LOCK_GENERATED
TRUE
CACHE INTERNAL ""
)
set(CPM_PACKAGE_LOCK_FILE
"${CPM_ABSOLUTE_PACKAGE_LOCK_PATH}"
CACHE INTERNAL ""
)
file(
WRITE ${CPM_PACKAGE_LOCK_FILE}
"# CPM Package Lock\n# This file is generated by CPM.cmake and should be committed to version control\n\n"
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit unclear to me what will happen if a dependency also uses CPM with a package lock file, I think this might overwrite the existing package lock again. I think calling CPMUsePackageLock should be a no-op if it has already been called at any point before.

Comment thread README.md
Comment on lines +280 to +303
### Generated package lock (pnpm/npm style)

Pass `GENERATED` to keep the lock in sync automatically, the way `pnpm-lock.yaml` or
`package-lock.json` work:

```cmake
CPMUsePackageLock(package-lock.cmake GENERATED)
```

In this mode the lock at the given path is rewritten in your source tree on **every configure**,
and git packages are pinned to the **exact commit that was checked out** rather than the (possibly
moving) `GIT_TAG`. This makes the lock reproducible even when dependencies are declared against a
branch like `main`. No separate `cpm-update-package-lock` step is needed.

To refresh the pins (e.g. to pull in newer upstream commits), delete the lock file and reconfigure,
just like deleting a `pnpm-lock.yaml`:

```bash
rm package-lock.cmake
cmake -Bbuild
```

The same behaviour can be enabled globally without editing `CMakeLists.txt` by setting the
`CPM_GENERATE_PACKAGE_LOCK` option (or environment variable).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding documentation! We should add that also changing any additional arguments like the OPTIONS will not work without removing the lockfile or the entry from the lockfile first.

Maybe it would also make sense to mark this as experimental for now, as for a proper npm-style lock one might expect.

  • Changing the version or explicitly setting a commit hash in the source would also update the locked hash accordingly
  • Any other flags from CPMAddPackage would still be respected, the lock should only impact the GIT_TAG and GIT_REPOSITORY entries.

Implementing this in the future could be a breaking change or lead to unexpected updates, so it makes sense to advertise caution for now.

@jamierpond

Copy link
Copy Markdown
Author

@TheLartians thanks so much for looking at this!

Meant for it to be marked as draft initially, am very grateful for your feedback.

Thanks again for giving this serious consideration, will revise with your feedback in mind!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants