test: cover break-glass cleanup of a stranded graceful reconfiguration#37873
Open
aljoscha wants to merge 1 commit into
Open
test: cover break-glass cleanup of a stranded graceful reconfiguration#37873aljoscha wants to merge 1 commit into
aljoscha wants to merge 1 commit into
Conversation
mtabebe
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
We may need to disable the cluster controller via the
enable_cluster_controllerbreak-glass flag (for example to work around a controller bug). Doing so while a graceful reconfiguration is in flight strands durable state: the controller is the only component that drives a reconfiguration record to a terminal status, so once it is off the in-progress record and the overlap replica it provisioned are left in place with nothing to retire them.The recovery lever is a config-shape
ALTER CLUSTERto a new size. With the controller off this takes the legacy path, which drops the entire observed owned replica set by id (baseline plus stranded overlap), recreates a denser1..rNat the new size, and cancels the carried record in the same transaction. This PR adds a regression test pinning that recovery, which is especially valuable while the legacy path is being reworked.What the test covers
Added to
test/testdrive/cluster-controller.td, in the break-glass section:in-progress.ALTERback to the realized shape is a no-op under break-glass (the byte-identical config short-circuits), so it is not the cleanup lever.ALTER CLUSTERto a new size and assert the desired end state: the new realized size, exactly one replica namedr1(no orphan left behind), the recordcancelled, and astartedthencancelledaudit lifecycle.Tests
Adds a new section to
test/testdrive/cluster-controller.td.