Skip to content

test: cover break-glass cleanup of a stranded graceful reconfiguration#37873

Open
aljoscha wants to merge 1 commit into
MaterializeInc:mainfrom
aljoscha:test-break-glass-reconfig-cleanup
Open

test: cover break-glass cleanup of a stranded graceful reconfiguration#37873
aljoscha wants to merge 1 commit into
MaterializeInc:mainfrom
aljoscha:test-break-glass-reconfig-cleanup

Conversation

@aljoscha

Copy link
Copy Markdown
Contributor

Motivation

We may need to disable the cluster controller via the enable_cluster_controller break-glass flag (for example to work around a controller bug). Doing so while a graceful reconfiguration is in flight strands durable state: the controller is the only component that drives a reconfiguration record to a terminal status, so once it is off the in-progress record and the overlap replica it provisioned are left in place with nothing to retire them.

The recovery lever is a config-shape ALTER CLUSTER to a new size. With the controller off this takes the legacy path, which drops the entire observed owned replica set by id (baseline plus stranded overlap), recreates a dense r1..rN at the new size, and cancels the carried record in the same transaction. This PR adds a regression test pinning that recovery, which is especially valuable while the legacy path is being reworked.

What the test covers

Added to test/testdrive/cluster-controller.td, in the break-glass section:

  1. Start a graceful reconfiguration on a MANUAL cluster. A sleeping materialized view pins hydration on every replica, so the target set never hydrates and the reconfiguration cannot cut over. This makes the in-flight state deterministic regardless of the tick cadence.
  2. Wait until the overlap replica exists (baseline and target shapes both up), then disable the controller mid-flight.
  3. Assert the stranding holds: two replicas across the two shapes, the realized size unchanged, and the record still in-progress.
  4. Assert that an ALTER back to the realized shape is a no-op under break-glass (the byte-identical config short-circuits), so it is not the cleanup lever.
  5. Do the hard ALTER CLUSTER to a new size and assert the desired end state: the new realized size, exactly one replica named r1 (no orphan left behind), the record cancelled, and a started then cancelled audit lifecycle.

Tests

Adds a new section to test/testdrive/cluster-controller.td.

@aljoscha
aljoscha requested a review from mtabebe July 24, 2026 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants