deps(lance): migrate 7.0.0 → 9.0.0-beta.15 — both corruption fixes upstream, vendor pin retired, blob compaction unlocked

[aaltshuler]

Migrates the storage substrate from Lance 7.0.0 to 9.0.0-beta.15 (git-rev-pinned to the tag commit f24e42c1 — the 9.x betas are git-tags only; switch to crates.io at 9.0.0 stable). Decision context: Lance ships fixes in majors, not patches; both active corruption bugs are first fixed in the 9.x line.

What this buys

• lance#7480 (row-id overlap, #7444) upstream since beta.11 → the vendor/lance-table pin is retired per its documented removal condition; its guard passes on stock lance-table and stays as the tripwire.
• lance#7320 (sequential BTREE segment-merge corruption, #7230 — the embed-refresh + optimize table-breaker we reproduced) upstream since beta.1 → no engine-side mitigation needed.
• Blob-v2 compaction (8.0.0 #7017 + #7618): executed the gate's documented removal plan — LANCE_SUPPORTS_BLOB_COMPACTION, the skip branch, and the skip reason deleted; guards inverted to positive pins. Blob+Vector tables rejoin maintenance.

Migration facts (full audit: the 2026-07-05 stanza in docs/dev/lance.md)

• 382 upstream commits reviewed (243 in 7→8, 139 in 8→9-beta.15).
• Zero ecosystem churn: Arrow 58, DataFusion 53, object_store 0.13.2 all unchanged.
• No format/reader-floor movement: data written under our settings round-trips to a 7.0.0 reader. Soft door: FTS v2 index files (new inverted indexes, default since beta.11) — readable by 7.0.0+, rebuildable derived state.
• One compile break in the whole workspace: RowAddrTreeMap moved to the new lance-select crate.
• Test realignments, each pinning improved upstream behavior: literal-coercion guard (v8 coerces width-mismatched literals → BTREE used), namespace decoupling guard (table_version_storage_enabled removed upstream; thesis holds), lazy-fork branch-consistency (Lance 9 loudly rejects cross-branch manifest opens — the invariant our entry-owner resolution always enforced; production opens by owner-resolved location, unaffected).

Verification

• cargo test --workspace: 68/68 suites green, including cost-budget gates (v9 IO improvements landed within slack).
• Failpoint suites green (engine 57, cluster 11). Surface guards 15/15.
• S3 (RustFS) suites run in this PR's CI.

Known-opens carried forward

lance#6666 still pub(crate) (the create_vector_index inline residual + recovery coverage retained), #6914 unshipped, upstream #7508 open (FTS errors under frequent optimize — our reconciler's pattern; watching). Beta-line caveat recorded in the stanza: blob reads regressed at beta.13, fixed beta.15 — re-audit before any beta advance or the stable switch.

---

Greptile Summary

This PR migrates the Lance storage substrate from 7.0.0 to 9.0.0-beta.15 (git-rev pinned at f24e42c1), retiring the vendored lance-table patch (lance#7480 now upstream), removing the blob-compaction skip gate (LANCE_SUPPORTS_BLOB_COMPACTION), and realigning tests to pin the improved upstream behaviors.

• Vendor pin retired: The vendor/lance-table [patch.crates-io] workaround for the rowid-overlap corruption (lance#7480) is removed since the fix ships in 9.0.0-beta.11; filtered_scan_tolerates_merge_update_row_id_overlap remains as the regression tripwire.
• Blob compaction unlocked: LANCE_SUPPORTS_BLOB_COMPACTION, SkipReason::BlobColumnsUnsupportedByLance, and the skip branch in optimize_one_table are deleted; surface guards and integration tests are inverted to assert positive behavior (compaction succeeds and preserves all rows).
• Test realignments: Literal-coercion guard, namespace-decoupling guard, and branch-consistency guard each re-pinned to improved or hardened upstream behavior in Lance 8/9.

Confidence Score: 5/5

Safe to merge — the migration is mechanically sound, every surface guard and integration test is realigned to pin the improved upstream behavior, and the vendor patch is cleanly retired.

The blob-compaction gate removal and the vendor-pin retirement are both well-guarded by inverted surface guards that turn red on regression. The RowAddrTreeMap import relocation is the only compile-time break, correctly applied in all three use sites. Test realignments pin new upstream behaviors rather than relaxing assertions. The git-rev pin is consistent across all lance crates.

AGENTS.md — architecture diagram on line 56 still says "Lance 7.x" after the substrate migration; one-line fix needed.

Important Files Changed

Filename	Overview
AGENTS.md	Header updated to Lance 9.x; architecture diagram on line 56 still reads "Lance 7.x" — stale reference flagged in previous review and not yet corrected.
Cargo.toml	All lance crates consistently re-pinned to git rev f24e42c1 (9.0.0-beta.15); vendor patch entry and exclude clause removed; lance-select added for RowAddrTreeMap relocation.
crates/omnigraph/src/db/omnigraph/optimize.rs	LANCE_SUPPORTS_BLOB_COMPACTION constant, BlobColumnsUnsupportedByLance skip variant, and has_blob parameter all cleanly removed; optimize_one_table signature tightened; DriftNeedsRepair remains the sole skip path.
crates/omnigraph/tests/lance_surface_guards.rs	Guard 10 correctly inverted to assert positive blob-compaction behavior; RowAddrTreeMap imports updated to lance-select in two places; literal-coercion and branch-consistency guards realigned to v9 behavior.
crates/omnigraph/tests/maintenance.rs	optimize_skips_blob_table_and_reports_skip renamed and inverted to assert blob tables compact, publish, and preserve all rows alongside plain tables.
crates/omnigraph/src/db/manifest/tests.rs	table_version_storage_enabled removed (API gone upstream); branch-consistency test correctly split into a mismatched-open error assertion and an owner-branch open success assertion.
docs/dev/lance.md	Comprehensive 2026-07-05 audit stanza added covering all 382 upstream commits; prior patch-pin stanza marked RETIRED; beta-line caveat noted for blob read regression at beta.13 / fixed beta.15.
docs/dev/invariants.md	Vendor-pin and blob-compaction known-gaps updated to CLOSED; tripwire guard names updated to match renamed tests; remaining open items (lance#6666, #7508) accurately retained.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["optimize_all_tables()"] --> B["Snapshot all table keys"]
    B --> C["Collect (table_key, full_path) pairs\n(blob flag removed)"]
    C --> D["buffer_unordered(concurrency)"]
    D --> E["optimize_one_table()"]
    E --> F{"HEAD > manifest?\n(DriftNeedsRepair)"}
    F -- Yes --> G["Skip: DriftNeedsRepair"]
    F -- No --> H["compact_files()\n(blob tables now included)"]
    H --> I["optimize_indices()"]
    I --> J["ManifestBatchPublisher::publish\n(manifest tracks Lance HEAD)"]
    J --> K["TableOptimizeStats\ncommitted=true"]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["optimize_all_tables()"] --> B["Snapshot all table keys"]
    B --> C["Collect (table_key, full_path) pairs\n(blob flag removed)"]
    C --> D["buffer_unordered(concurrency)"]
    D --> E["optimize_one_table()"]
    E --> F{"HEAD > manifest?\n(DriftNeedsRepair)"}
    F -- Yes --> G["Skip: DriftNeedsRepair"]
    F -- No --> H["compact_files()\n(blob tables now included)"]
    H --> I["optimize_indices()"]
    I --> J["ManifestBatchPublisher::publish\n(manifest tracks Lance HEAD)"]
    J --> K["TableOptimizeStats\ncommitted=true"]

Loading

Comments Outside Diff (1)

1. AGENTS.md, line 56 (link)

   The architecture diagram still references "Lance 7.x" after the substrate migration to 9.x. Since AGENTS.md is always loaded into agent context, this stale line will cause any agent reading the diagram to believe the substrate is still on the 7.x line — contradicting the correctly-updated header on line 21. The AGENTS.md maintenance contract rule Implement aggregate functions with GROUP BY support #6 ("Don't lie") calls this out explicitly.

   

_{Reviews (3): Last reviewed commit: "docs(lance): 9.0.0-beta.15 alignment sta..." | Re-trigger Greptile}

[devin-ai-integration]

Devin Review found 1 potential issue.