Skip to content

(feat): cache external (plugin-generated) file content in the defs cache#10164

Merged
eytan-starkware merged 1 commit into
mainfrom
eytan_graphite/support-external-file-cache
Jun 28, 2026
Merged

(feat): cache external (plugin-generated) file content in the defs cache#10164
eytan-starkware merged 1 commit into
mainfrom
eytan_graphite/support-external-file-cache

Conversation

@eytan-starkware

Copy link
Copy Markdown
Contributor

Serialize each external file's VirtualFile content into the crate cache blob, keyed by a
portable, parse-free identity (ExternalFileContentKey), and serve it from ext_as_virtual
before falling back to module_sub_files. A loaded crate cache can then materialize
plugin-generated files without re-running plugins. The lookup depends only on the blob
(no module-data edge), so it never re-enters cache loading.

@reviewable-StarkWare

Copy link
Copy Markdown

This change is Reviewable

eytan-starkware commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

@eytan-starkware eytan-starkware changed the base branch from main to graphite-base/10164 June 25, 2026 10:48
@eytan-starkware eytan-starkware force-pushed the eytan_graphite/support-external-file-cache branch from 1334cde to 9cb7a3d Compare June 25, 2026 10:48
@eytan-starkware eytan-starkware changed the base branch from graphite-base/10164 to eytan_graphite/cache-blob-sections June 25, 2026 10:48
@eytan-starkware eytan-starkware marked this pull request as ready for review June 25, 2026 10:48
@cursor

cursor Bot commented Jun 25, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches core cache blob layout (new section shifts indices) and ext_as_virtual resolution during cache load, where incorrect keys or misses can panic or serve wrong generated code; behavior is guarded but stale caches may need regeneration.

Overview
Adds a new first blob section (EXTERNAL_CACHE_SECTION) so plugin-generated (external) file bodies are stored in the crate cache and later served without re-running macros. Section order becomes [external, def, semantic, lowering]; existing def/semantic/lowering readers skip the new prefix via section indices.

On save, caching external FileIds also records VirtualFile payload in an ExternalFileKeyExternalFileContentCached table (keys are parse/plugin-free: on-disk path, virtual parent+name+content hash, or generated parent+offset+name). On load, external_file_contents reads only that section from the blob (keyed by blob_id, not module data) so ext_as_virtual does not pull in defs cache and create salsa cycles.

ext_as_virtual_impl now returns cached content when present; for cache-backed crates with a loaded blob, a miss asserts instead of falling back to module_sub_files (that fallback would re-enter loading). Non-cache-backed crates still use the plugin path.

Lowering artifact writing prepends the external table; cache tests cover #[derive(Drop)] at crate root and in a submodule.

Reviewed by Cursor Bugbot for commit 76e3afe. Bugbot is set up for automated code reviews on this repo. Configure here.

@eytan-starkware eytan-starkware force-pushed the eytan_graphite/support-external-file-cache branch from 9cb7a3d to 329a5a8 Compare June 25, 2026 11:09
@eytan-starkware eytan-starkware force-pushed the eytan_graphite/cache-blob-sections branch from f0a8344 to 0f2cd03 Compare June 25, 2026 11:09

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 329a5a8. Configure here.

Comment thread crates/cairo-lang-lowering/src/cache/mod.rs
Comment thread crates/cairo-lang-defs/src/cache/mod.rs
Comment thread crates/cairo-lang-lowering/src/cache/mod.rs Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 329a5a8. Configure here.

Comment thread crates/cairo-lang-lowering/src/cache/mod.rs
Comment thread crates/cairo-lang-lowering/src/cache/mod.rs Outdated
@eytan-starkware eytan-starkware force-pushed the eytan_graphite/support-external-file-cache branch from 329a5a8 to 3f5bf0b Compare June 25, 2026 11:35

@orizi orizi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@orizi partially reviewed 4 files and made 1 comment.
Reviewable status: 4 of 5 files reviewed, 2 unresolved discussions (waiting on eytan-starkware and TomerStarkware).


crates/cairo-lang-lowering/src/cache/test.rs line 90 at r3 (raw file):

/// strings stand in for parent `FileId`s, which are interned per-db and thus not comparable.
#[derive(Clone, Debug, PartialEq)]
struct ExternalFileSnapshot {

re-valitate.

@eytan-starkware eytan-starkware force-pushed the eytan_graphite/cache-blob-sections branch from 0f2cd03 to caad4bd Compare June 28, 2026 10:27
@eytan-starkware eytan-starkware force-pushed the eytan_graphite/support-external-file-cache branch from 3f5bf0b to c9b0603 Compare June 28, 2026 10:27

@orizi orizi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

@orizi reviewed 5 files and all commit messages, made 1 comment, and resolved 1 discussion.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on TomerStarkware).

@eytan-starkware eytan-starkware force-pushed the eytan_graphite/support-external-file-cache branch from c9b0603 to 3ce0717 Compare June 28, 2026 10:56
@eytan-starkware eytan-starkware changed the base branch from eytan_graphite/cache-blob-sections to main June 28, 2026 11:18
Serialize each external file's VirtualFile content into the crate cache blob as its own
(first) section, keyed by a portable, parse-free identity (ExternalFileKey), and serve it
from ext_as_virtual before falling back to module_sub_files. A loaded crate cache can then
materialize plugin-generated files without re-running plugins. The lookup depends only on
the blob (no module-data edge), so it never re-enters cache loading.
@eytan-starkware eytan-starkware force-pushed the eytan_graphite/support-external-file-cache branch from 3ce0717 to 76e3afe Compare June 28, 2026 11:19
@graphite-app

graphite-app Bot commented Jun 28, 2026

Copy link
Copy Markdown

Merge activity

  • Jun 28, 11:19 AM UTC: Graphite rebased this pull request, because this pull request is set to merge when ready.

@orizi orizi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@orizi reviewed 1 file and all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on TomerStarkware).

@eytan-starkware eytan-starkware added this pull request to the merge queue Jun 28, 2026
Merged via the queue into main with commit 3fdb64b Jun 28, 2026
55 checks passed
@eytan-starkware eytan-starkware deleted the eytan_graphite/support-external-file-cache branch June 28, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants