Skip to content

Bound shielded-sync memory: process notes per-batch instead of accumulating across the whole scan range #3820

Description

@QuantumExplorer

Follow-up from PR #3819 (review by thepastaclaw).

Problem

sync_notes_across (packages/rs-platform-wallet/src/wallet/shielded/sync.rs) streams from aligned_start to the chain tip in a single call (the ShieldedSyncConfig.max_concurrent bound only caps in-flight fetches, not wallet-side buffers). Inside that one call it accumulates three in-memory buffers and only flushes them after the stream fully drains:

  • scanned_nullifiers: Vec<[u8; 32]> — pushed for every scanned action (sync.rs:429), consumed only at end-of-pass via apply_scanned_nullifier_spends (sync.rs:664). This is the dominant, dense growth: 32 B × every note.
  • decrypted_by_subwallet / recovered_outgoing_by_subwallet — sparse (only the wallet's own received / sent notes).

Resident memory therefore scales with the total scanned range, not the batch size:

Pool scanned in one call scanned_nullifiers RAM
1M notes ~32 MB steady, ~48–64 MB peak (Vec capacity doubling)
10M notes ~320 MB

The commitment tree itself is SQLite-backed, so it is not the RAM driver — this buffer is. On a full restore / large catch-up this is real heap pressure, and it is fatal in iOS app-extension contexts (notification-service / background-refresh extensions have ~24–30 MB jetsam caps — 32 MB kills them instantly).

Why the obvious quick fix does not work

thepastaclaw suggested applying apply_scanned_nullifier_spends per batch. That alone is incorrect: decrypted notes are only accumulated in decrypted_by_subwallet during the loop and saved via store.save_note once, end-of-pass (sync.rs:613). A per-batch spend replay would run against a store that has not saved this pass's receipts yet, missing every same-pass receive-then-spend.

Correct fix

Process each streamed batch fully instead of accumulate-then-flush — i.e. move the receipt-save + outgoing-record + spend-apply into the while let Some(item) = stream.next() loop:

  1. append commitments (already per-batch)
  2. trial-decrypt + save_note this batch's receipts
  3. OVK-recover + record_outgoing_note
  4. apply_scanned_nullifier_spends for this batch's nullifiers (a per-batch Vec, or a reused/drained buffer)

This bounds all three buffers to one batch (~8192 notes ≈ 256 KB, flat regardless of pool size) and drops the accumulator maps entirely (cleaner, more streaming-friendly).

Correctness: the receipt-before-spend invariant still holds — batches stream in tree/position order, and each batch saves its receipts (step 2) before applying its spends (step 4); a note received in an earlier batch is already saved when a later batch's spend matches it.

One behavior change: incoming notes' block_height becomes the batch's height instead of the pass-wide max — which makes them consistent with outgoing notes, that already use batch.block_height.

Notes

  • Pre-release v12 shielded pool; no migration concerns.
  • The Part-A tests cover apply_scanned_nullifier_spends in isolation but not the full sync_notes_across loop, so this restructure should add loop-level coverage (received-then-spent across batch boundaries; same-batch receive+spend).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions