fix(wal): correct slot-range endianness so pruning and lookups work

[scarmuega]

Problem

WAL keys store slot bytes big-endian (ChainPoint::into_bytes, crates/core/src/point.rs:137), so redb's lexicographic key comparison makes byte order equal to slot order. But DbChainPoint::slot_range built its scan bounds little-endian, so for any realistic slot (e.g. preprod ~127,000,000 = 0x0799…) the computed range excluded every real key. Every slot-range scan over the WAL returned empty.

Consequences:

• WAL pruning was permanently broken. prune_history → remove_before → approximate_slot_with_retry always returned None → SlotNotFound, which prune_history caught and logged "pruning target slot not found, skipping", returning Ok. Housekeeping believed it had pruned. On long-running instances the WAL grew without bound (observed ~15× over the configured max_rollback).
• dolos data dump-wal --from <slot> and dolos data copy-wal --since/--until reported "slot not found".

Sync/rollback correctness was not affected (those paths build keys big-endian via DbChainPoint::from). The archive store is also unaffected — it keys on native BlockSlot (u64), which redb orders numerically.

Changes (crates/redb3/src/wal/mod.rs)

1. Endianness fix. Added min_bound/max_bound helpers on DbChainPoint that encode the slot with to_be_bytes() (min padded with the zero hash, max with 0xFF); slot_range now uses them. These were the only two to_le_bytes() calls in the crate.
2. Hardened remove_before. The endianness fix turns remove_before into a live deletion path for the first time. It previously located the cutoff via an approximate_slot(±180) search that silently no-op'd in sparse chain regions. It now derives the cutoff bound directly from the target slot via min_bound(slot) and extract_from_if(..bound) — deterministically removing everything with slot < target while retaining real entries at the cutoff slot (nonzero hash). The now-dead SlotNotFound catch in prune_history is removed. locate_point/approximate_slot are unchanged (correct for the dump/copy tooling).
3. Regression tests (5, using the existing TestStore/TestDelta scaffolding): byte-boundary round-trip (2⁸/2¹⁶/2³² + preprod-scale slot), full prune shrinking to exactly max_slots, batched prune convergence without pruning into the protected window, sparse-WAL remove_before, and cutoff-slot retention.

Verification

• cargo test -p dolos-redb3 wal:: → 10/10 pass (the round-trip test fails without the byte-order fix).
• cargo test -p dolos-core point:: → BE key-order proptest holds.
• cargo build (full workspace) and cargo clippy -p dolos-redb3 → clean.

Rollout note

This activates WAL pruning for the first time. On long-running instances the first housekeeping prunes a large backlog in 10,000-slot batches (MAX_PRUNE_SLOTS_PER_HOUSEKEEPING); the data housekeeping CLI loops until done and the periodic sync-loop path catches up over intervals. Consider running dolos data housekeeping once explicitly on affected instances after upgrade.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Corrected slot-based WAL scan key ordering to consistently match numeric slot order across byte boundaries.
  • Updated WAL pruning to deterministically delete only entries strictly older than the cutoff slot, preserving the cutoff record itself.
  • Improved prune error handling by surfacing underlying failures instead of masking missing-slot cases.

• Tests

  • Added regression tests for slot-range ordering across boundaries, extreme/exclusive bounds handling, prune_history window shrinking, batched pruning convergence, and sparse WAL cleanup.

[coderabbitai]

Warning

Review limit reached

@scarmuega, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 29 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: cf1e7d11-8f28-4e1b-98e5-476835d288bc

📥 Commits

Reviewing files that changed from the base of the PR and between 24f2d72 and 4539ff3.

📒 Files selected for processing (1)

• crates/redb3/src/wal/mod.rs

📝 Walkthrough

Walkthrough

This PR changes WAL slot-bound encoding to big-endian bytes, makes pruning use deterministic cutoff-bound deletion, propagates prune errors directly, and adds regression tests for ordering and pruning behavior.

Changes

WAL key ordering and pruning

Layer / File(s)	Summary
Slot bound encoding and range construction crates/redb3/src/wal/mod.rs	DbChainPoint slot bounds now use big-endian bytes, and slot_range maps RangeBounds<BlockSlot> directly through min_bound/max_bound.
Deterministic remove_before and prune_history propagation crates/redb3/src/wal/mod.rs	remove_before switches to an exclusive cutoff bound from min_bound(slot) and removes entries below it; prune_history now propagates remove_before errors directly.
Regression tests crates/redb3/src/wal/mod.rs	Tests cover key ordering across byte boundaries, exclusive bound extremes, prune convergence, and sparse-region cutoff retention.

Estimated code review effort: 4 (Complex) | ~40 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PruneHistory
  participant RemoveBefore
  participant DbChainPoint
  participant WalStore

  PruneHistory->>RemoveBefore: remove_before(prune_before)
  RemoveBefore->>DbChainPoint: min_bound(target_slot)
  DbChainPoint-->>RemoveBefore: exclusive cutoff bound
  RemoveBefore->>WalStore: extract entries < bound
  WalStore-->>RemoveBefore: removed entries
  RemoveBefore-->>PruneHistory: result

Loading

Related PRs: None identified from the provided information.

Suggested labels: wal, redb3, bugfix

Suggested reviewers: None identified from the provided information.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures the main WAL fix: slot-range endianness corrections that restore pruning and lookup behavior.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/wal-slot-range-endianness

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/redb3/src/wal/mod.rs`:
- Around line 543-545: The rustdoc for prune_history is stale and still
describes warning/Ok behavior for a missing target slot, but the implementation
now always prunes by bound via remove_before and only propagates its errors.
Update the doc comment above prune_history to match the current behavior, using
the prune_history and remove_before symbols so it’s clear the old missing-slot
path no longer applies.
- Around line 688-690: In remove_before, the extract_from_if iterator error is
being swallowed by matching only Some(Ok(...)), which can allow wx.commit() to
persist a partial prune after an Err. Update the remove_before flow around
wal.extract_from_if and the to_remove iteration to explicitly process each
drained item and propagate any iterator error with ?, so the cleanup only
commits when all entries are handled successfully.
- Around line 88-104: The slot_range helper is adjusting BlockSlot bounds with
+1/-1, which can overflow for Excluded(BlockSlot::MAX) or underflow for
Excluded(BlockSlot::MIN). Update slot_range to construct the DbChainPoint range
directly from the incoming RangeBounds<BlockSlot> without shifting the slot
values, and keep the logic centered in slot_range with
DbChainPoint::min_bound/max_bound handling the bound conversion.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b5942144-baca-49ff-953a-bddce79a8882

📥 Commits

Reviewing files that changed from the base of the PR and between deb312c and 2394603.

📒 Files selected for processing (1)

• crates/redb3/src/wal/mod.rs