Make the string cache completely lock-free#2
Open
bouk wants to merge 12 commits into
Open
Conversation
as_str() and as_string_cache_entry() located the len field by stepping
back one usize from the char pointer. On 32-bit targets (e.g. wasm32)
StringCacheEntry { hash: u64, len: usize } has 4 bytes of trailing
padding, so the chars start 8 bytes after len and stepping back one
usize lands in padding. Compute the header address with
size_of::<StringCacheEntry>() instead, which is correct on every
target.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-bin Mutex<StringCache> with a lock-free design so that interning never blocks. This makes the crate usable on targets where blocking synchronization is unavailable, most notably the browser main thread on wasm32, where parking_lot cannot park. Each bin is now an insert-only, open-addressed hash table of AtomicPtr<StringCacheEntry> slots, in the style of Cliff Click's non-blocking hash map: - Lookups (including the fast path of interning an already-cached string) are pure acquire-load probes: no locks, no CAS, no waiting. - Inserts claim the first empty slot in their probe sequence with a compare-exchange. Races on the same string collide on the same slot and the loser adopts the winner's entry, so pointer identity is preserved. - Growth hangs a double-size successor table off the old one and migrates entry pointers cooperatively in chunks; empty slots are sealed with a MOVED sentinel so they can never be claimed, and probers redirect to the successor when they hit one. Since only pointers move, interned addresses remain stable across growth. - String storage is claimed from the bump arenas by atomically bumping the arena pointer, and full arenas are swapped for bigger ones with a CAS. The parking_lot, spin and lazy_static dependencies are removed (the cache is now a const-initialized static); the parkinglot and spinlock features remain as deprecated no-ops for backwards compatibility. parking_lot is still a dev-dependency of the benchmark for the comparison interners. The cache iterator now walks the tables instead of the arenas, since arena regions can be mid-write while other threads intern; table slots only ever contain fully published entries. Tests: the fixed test-global INITIAL_CAPACITY is tiny under cfg(test) so the suite hammers growth/migration, and new stress tests check cross-thread pointer identity and pointer stability across growth. The suite passes under Miri (including the threaded stress test, with Miri's data-race detector). Tests that touch the global cache now all take TEST_LOCK; test_hashing and serialization_ustr previously raced against tests that clear the cache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mutex-based clear() zeroed the entry table in place. Freeing all the tables and lazily reallocating them made every benchmark iteration re-fault ~8MB of fresh zero pages, which serializes in the kernel and penalized the multithreaded benchmarks. Keep the newest (largest) table zeroed in place instead, matching the old behavior; older tables in the migration chain are still freed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cache is lock-free so they no longer select anything. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Interleaved A/B benchmarks (best of 3 rounds each, Apple M3 Pro 6P+6E), mutex -> lock-free: single raft ustr 1.204 ms -> 1.181 ms (par) raft ustr x 1 thread 1.235 ms -> 1.230 ms (par) raft ustr x 2 3.177 ms -> 3.085 ms (-3%) raft ustr x 4 4.044 ms -> 3.413 ms (-16%) raft ustr x 6 4.595 ms -> 3.669 ms (-20%) raft ustr x 8 7.292 ms -> 7.817 ms (+7%, see below) raft ustr x 12 10.864 ms -> 7.877 ms (-27%) raft large x1 1.770 ms -> 1.780 ms (par) raft large x6 3.616 ms -> 3.002 ms (-17%) The x8 number is an asymmetric-core benchmark artifact: each bench thread gets a fixed 100k ops and the harness waits for the slowest thread, so once threads spill onto E-cores the lock-free wall time is one E-core executing its whole batch (note x8 == x12 == 7.9 ms flat), while the mutex serializes all threads to the same rate regardless of core. Aggregate throughput is higher lock-free at every thread count. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Interning is dominated by hashing plus a short probe, and for the
25-95 byte path-like strings in the bench corpus the hash is a large
fraction of the ~12ns cost of an already-interned ustr() call. An
interleaved shootout of fixed-seed hashers on an M3 Pro (best-of-3
medians of the single-threaded benches):
single raft ustr raft large x1
ahash 0.7 (old) 1.177 ms 1.768 ms
ahash 0.8 1.177 ms 1.754 ms
rapidhash 4.5 1.135 ms 1.597 ms
foldhash 0.2 1.100 ms 1.495 ms
foldhash wins across the board (-7% on ~25 byte strings, -15% on ~95
byte strings), so the cache now hashes with foldhash's fixed-seed
fast variant. It is pure arithmetic with no_std support, so wasm32
and Miri keep working. Also dedupes the previously copy-pasted hash
computation into a hash_string() helper shared by Ustr::from and
Ustr::from_existing, and drops the stale ahash mention from the
fasthash deprecation note.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ustr::from, from_existing, as_str, as_char_ptr, len and precomputed_hash had no #[inline], so callers in other crates (built without LTO) paid a function call per intern/lookup and couldn't constant-fold the fixed hash seeds into the hashing code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
StringCache.count is bumped on every insert and sat on the same
128-byte cache line as root, so while a bin is taking inserts every
count update invalidated the line that all concurrent probes of that
bin must read first. Move count and the arena pointer into an
align(128) wrapper so the read side (root, first) keeps its line
clean.
Interleaved A/B of the three optimization commits together vs the
previous HEAD (best-of-3 medians, M3 Pro 6P+6E):
single raft ustr 1.228 ms -> 1.114 ms (-9%)
raft ustr x 1 threads 1.250 ms -> 1.100 ms (-12%)
raft ustr x 2 threads 3.042 ms -> 2.944 ms (-3%)
raft ustr x 4 threads 3.472 ms -> 3.325 ms (-4%)
raft ustr x 6 threads 3.758 ms -> 3.582 ms (-5%)
raft ustr x 8 threads 7.903 ms -> 8.048 ms (+2%, E-core
makespan floor, noise)
raft ustr x 12 threads 8.206 ms -> 7.923 ms (-3%)
raft large x1 1.779 ms -> 1.503 ms (-16%)
raft large x6 2.915 ms -> 2.700 ms (-7%)
Most of the win is the foldhash switch; this padding change and the
inline hints add a consistent 1-5% on the threaded benches on top of
it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Runs the test suite (debug + release, with and without serialization) on Linux/macOS/Windows, the full suite under Miri including the threaded stress tests, and a wasm32-unknown-unknown release build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
foldhash is now the only hasher; the optional fasthash/cityhash path and the deprecation notice that came with it are gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The org's Actions policy blocks dtolnay/rust-toolchain and Swatinem/rust-cache (startup_failure). Install toolchains with the preinstalled rustup instead and cache with actions/cache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
ustr is used from WASM, and
parking_lotcannot be used on the JS main thread because that thread can't be parked. This removes the per-binparking_lot::Mutexentirely and replaces it with a completely lock-free design, so interning works everywhere — including the wasm32 main thread — while also being faster.What
bd84dca): each of the 64 bins is now an insert-only, open-addressed atomic hash table in the style of Cliff Click's non-blocking hash map. Lookups of already-interned strings are pure acquire-load probes with no shared-memory writes; inserts claim slots by CAS; growth migrates entry pointers cooperatively into a successor table so interned addresses stay stable forever. String bytes live in atomically-bumped arenas. The soundness invariants are documented at the top ofsrc/stringcache.rs.d2c18ec):as_str()located the length via a fixed offset that read struct padding on 32-bit targets — fixed, which wasm32 needs anyway.8186f47,628dc82): thespinlock,parkinglotandhashcityfeatures are gone;parking_lot,spin,lazy_staticandfasthashare no longer dependencies of the library.75e5d5b, −7% on ~25B strings, −15% on ~95B strings), moved insert-path counters off the cache line probes read (19b610a), and added#[inline]to the hot entry points (af4ed45).3da5083,9c6ac4f): GitHub Actions workflow running the test suite on Linux/macOS/Windows, the full suite under Miri (including the threaded stress tests), and a wasm32-unknown-unknown build — using only GitHub-owned actions to comply with the org's Actions allowlist.Performance
Interleaved A/B on an M3 Pro (best-of-3 medians), old mutex implementation vs this branch:
* The x8 case is a benchmark artifact of asymmetric cores: each thread gets fixed work and the harness waits for the slowest thread, so once threads spill onto E-cores the wall time is one E-core's full batch (x8 ≈ x12 ≈ 8ms flat for the lock-free build). The mutex hides this by serializing all threads to the same rate; aggregate throughput is higher lock-free at every thread count.
Verification
serialization); tests build with tiny table sizes so growth/migration is exercised heavilywasm32-unknown-unknown(dev + release)🤖 Generated with Claude Code