fix(ethexe/malachite-core): wait for WAL advisory lock release in MalachiteService::shutdown

[grishasobol]

Closes #5545.

Summary

`MalachiteService::shutdown().await` is the documented restart barrier — by the time it returns, every file lock the service held must be released so the same base dir can be re-opened. For RocksDB that's true (`Store` lives inside the app task and is dropped when the task is awaited). For Malachite's WAL it is not true today.

Upstream Malachite owns the WAL file from a detached `std::thread` (see `arc-malachitebft-engine::wal::thread`). The WAL actor's `post_stop` only sends a `Shutdown` message on a channel; the writer thread releases its `advisory_lock` only when it later drops the log. The actor's `JoinHandle` resolves as soon as `post_stop` returns, so the writer thread can still be alive — and `flock(LOCK_EX)` on `consensus.wal` still held — when `MalachiteService::shutdown().await` returns. The next `MalachiteService::new` on the same base dir then panics with `Failed to acquire exclusive advisory lock`.

Fix

After awaiting engine + app handles in `shutdown`, probe the WAL advisory lock from a fresh fd. Poll every 25 ms up to a 10 s ceiling. The poll is a no-op when the WAL never existed; on the healthy path the lock is free within milliseconds, so this adds no measurable shutdown latency.

Test

`shutdown_releases_wal_advisory_lock` (new): restarts a 3-validator cohort 50 times and `try_lock(LOCK_EX)` on each `consensus.wal` from a fresh fd the instant `shutdown().await` returns. Without the fix this fails in the first cycle locally with `AlreadyLocked` (~120 ms). With the fix all 50 cycles pass in <5 s.

Diagnosis trail

• Original master flake: https://github.com/gear-tech/gear/actions/runs/26678194627/job/78634373130 (`seven_validators_full_network_restart` panic in `MalachiteService::new` with the exact `Failed to acquire exclusive advisory lock` error string).
• Repro PR (do-not-merge) with a deterministic probe: [diagnostic, do-not-merge] repro WAL advisory-lock race on master #5543.
• This PR's branch verified green against the same probe.

Test plan

• `cargo nextest run -p ethexe-malachite-core --test multi_validators --release` — full suite passes locally.
• `cargo nextest run -p ethexe-malachite-core -E 'test(shutdown_releases_wal_advisory_lock)' --release` — 50/50 cycles pass.
• `cargo nextest run -p ethexe-malachite-core -E 'test(seven_validators_full_network_restart)' --release` ×5 — 5/5 pass.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a race condition in the MalachiteService shutdown process that could lead to failures when attempting to restart the service using the same base directory. Previously, the WAL advisory lock held by a detached writer thread was not guaranteed to be released by the time shutdown().await returned, causing subsequent service initialization to panic. The changes introduce a robust polling mechanism to ensure the WAL lock is released before shutdown() completes, thereby guaranteeing a clean restart capability.

Highlights

• WAL Advisory Lock Release: Resolved a critical issue where MalachiteService::shutdown() did not reliably release the Write-Ahead Log (WAL) advisory lock, preventing subsequent service restarts on the same base directory.
• Polling Mechanism for Lock Release: Introduced a polling mechanism in MalachiteService::shutdown() to explicitly wait for the WAL advisory lock to be released by the detached WAL writer thread, ensuring a clean state for restarts.
• New Regression Test: Added a new regression test, shutdown_releases_wal_advisory_lock, to verify that the fix correctly addresses the race condition and allows for immediate service restarts without lock contention.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces the advisory-lock dependency to manage the WAL advisory lock release during service shutdown in ethexe-malachite-core. It adds a helper function wait_wal_lock_released to poll and wait for the WAL file lock to be released, ensuring that subsequent service starts on the same base directory do not fail due to lock contention. A regression test is also added to verify this behavior. The reviewer suggests improving wait_wal_lock_released by opening the WAL file with read-only permissions to avoid sharing violations and handling file open errors more robustly by only returning early on NotFound errors.