adapter: preserve reconfiguration deadline on unrelated ALTER#37863
Open
aljoscha wants to merge 1 commit into
Open
adapter: preserve reconfiguration deadline on unrelated ALTER#37863aljoscha wants to merge 1 commit into
aljoscha wants to merge 1 commit into
Conversation
## Motivation A config-shape `ALTER CLUSTER` that arrives while a graceful reconfiguration is already in flight folds onto the in-flight durable reconfiguration record. The target shape folds per dimension, but the record's `deadline` and `on_timeout` action were recomputed from scratch from the *new* statement's `WITH (WAIT ...)` surface and written wholesale, even when the new statement carried no `WAIT` clause at all. A statement with no `WAIT` clause has no deadline contract of its own: it resolves to the `default_cluster_reconfiguration_timeout` default (24h) and an implicit `ON TIMEOUT ROLLBACK`. Folding it onto an in-flight record therefore silently replaced the deadline and timeout action that the statement which *started* the reconfiguration had set. An unrelated `ALTER` from another session could stretch a user's explicit short deadline out to 24h. That is the nightly failure this closes. A user `ALTER` started a 120s `ON TIMEOUT ROLLBACK` reconfiguration on a cluster. Roughly 14s later an unrelated, system-issued `ALTER CLUSTER ... SET (EXPERIMENTAL ARRANGEMENT COMPRESSION = true)` landed on the same cluster. Arrangement compression is a replica-shape dimension, so that statement folds onto the in-flight record too, and being a no-`WAIT` `ALTER` it reset the deadline to ~24h out. The reconfiguration could no longer roll back at its 120s deadline, so it stayed parked `in-progress`, and the test observed `Graceful reconfiguration ... did not complete (reconfiguration status: in-progress)`. ## Description Resolve the deadline and on-timeout action based on whether the folding `ALTER` carries a `WAIT` clause: - **No `WAIT` clause, reconfiguration in flight:** keep the in-flight record's `deadline` and `on_timeout`. The statement carries no contract of its own, so it must not touch the one the user set on the reconfiguration in progress. - **No `WAIT` clause, nothing in flight:** unchanged. The system-default timeout and the implicit `ROLLBACK` action. - **Explicit `WAIT FOR` / `WAIT UNTIL READY`:** unchanged. Their deadline and action fold onto the in-flight record wholesale, which is what lets a user *steer* an in-flight reconfiguration (adjust the deadline or the timeout action) without discarding hydration progress already made on the target. Only the no-`WAIT`-onto-in-flight case changes behavior. User-visible effect: an unrelated `ALTER CLUSTER` that omits `WITH (WAIT ...)` no longer resets the deadline or `ON TIMEOUT` action of a graceful reconfiguration already in progress. The contract set by the `ALTER` that started (or last steered) the reconfiguration is preserved. ## Out of scope Whether to outright *deny* config-shape `ALTER`s while a reconfiguration is in flight (today only `SCHEDULE` and `REPLICATION FACTOR` changes are rejected) is a separate design question, deferred. ## Tests Added a case to `test/testdrive/cluster-controller.td`: a sleeping materialized view keeps a target from hydrating, an `ALTER` starts a reconfiguration with `WAIT UNTIL READY (TIMEOUT '600s', ON TIMEOUT 'ROLLBACK')`, then an unrelated no-`WAIT` `SET (EXPERIMENTAL ARRANGEMENT COMPRESSION = true)` folds on. The test asserts the deadline is unchanged, `on_timeout` stays `rollback`, status stays `in-progress`, and the size dimension the second `ALTER` did not mention keeps the in-flight target's value. Before the fix the deadline jumped to `now + 24h`, failing the equality assertion. Fixes SQL-568. https://linear.app/materializeinc/issue/SQL-568/graceful-reconfiguration-of-cluster-cluster-3-to-size-scale1workers1
mtabebe
approved these changes
Jul 24, 2026
|
|
||
| > DROP CLUSTER cc_force_existing CASCADE | ||
|
|
||
| # A folding ALTER without a WAIT clause must not touch the in-flight deadline |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A config-shape
ALTER CLUSTERthat arrives while a graceful reconfiguration isalready in flight folds onto the in-flight durable reconfiguration record. The
target shape folds per dimension, but the record's
deadlineandon_timeoutaction were recomputed from scratch from the new statement's
WITH (WAIT ...)surface and written wholesale, even when the new statement carried no
WAITclause at all.
A statement with no
WAITclause has no deadline contract of its own: itresolves to the
default_cluster_reconfiguration_timeoutdefault (24h) and animplicit
ON TIMEOUT ROLLBACK. Folding it onto an in-flight record thereforesilently replaced the deadline and timeout action that the statement which
started the reconfiguration had set. An unrelated
ALTERfrom another sessioncould stretch a user's explicit short deadline out to 24h.
That is the nightly failure this closes. A user
ALTERstarted a 120sON TIMEOUT ROLLBACKreconfiguration on a cluster. Roughly 14s later anunrelated, system-issued
ALTER CLUSTER ... SET (EXPERIMENTAL ARRANGEMENT COMPRESSION = true)landed on the same cluster. Arrangement compression is areplica-shape dimension, so that statement folds onto the in-flight record too,
and being a no-
WAITALTERit reset the deadline to ~24h out. Thereconfiguration could no longer roll back at its 120s deadline, so it stayed
parked
in-progress, and the test observedGraceful reconfiguration ... did not complete (reconfiguration status: in-progress).Description
Resolve the deadline and on-timeout action based on whether the folding
ALTERcarries a
WAITclause:WAITclause, reconfiguration in flight: keep the in-flight record'sdeadlineandon_timeout. The statement carries no contract of its own, soit must not touch the one the user set on the reconfiguration in progress.
WAITclause, nothing in flight: unchanged. The system-default timeoutand the implicit
ROLLBACKaction.WAIT FOR/WAIT UNTIL READY: unchanged. Their deadline andaction fold onto the in-flight record wholesale, which is what lets a user
steer an in-flight reconfiguration (adjust the deadline or the timeout
action) without discarding hydration progress already made on the target.
Only the no-
WAIT-onto-in-flight case changes behavior.User-visible effect: an unrelated
ALTER CLUSTERthat omitsWITH (WAIT ...)no longer resets the deadline or
ON TIMEOUTaction of a gracefulreconfiguration already in progress. The contract set by the
ALTERthatstarted (or last steered) the reconfiguration is preserved.
Out of scope
Whether to outright deny config-shape
ALTERs while a reconfiguration is inflight (today only
SCHEDULEandREPLICATION FACTORchanges are rejected) isa separate design question, deferred.
Tests
Added a case to
test/testdrive/cluster-controller.td: a sleeping materializedview keeps a target from hydrating, an
ALTERstarts a reconfiguration withWAIT UNTIL READY (TIMEOUT '600s', ON TIMEOUT 'ROLLBACK'), then an unrelatedno-
WAITSET (EXPERIMENTAL ARRANGEMENT COMPRESSION = true)folds on. The testasserts the deadline is unchanged,
on_timeoutstaysrollback, status staysin-progress, and the size dimension the secondALTERdid not mention keepsthe in-flight target's value. Before the fix the deadline jumped to
now + 24h,failing the equality assertion.
Fixes SQL-568.
https://linear.app/materializeinc/issue/SQL-568/graceful-reconfiguration-of-cluster-cluster-3-to-size-scale1workers1