Skip to content

adapter: preserve reconfiguration deadline on unrelated ALTER#37863

Open
aljoscha wants to merge 1 commit into
MaterializeInc:mainfrom
aljoscha:adapter-preserve-reconfiguration-deadline-on-fold
Open

adapter: preserve reconfiguration deadline on unrelated ALTER#37863
aljoscha wants to merge 1 commit into
MaterializeInc:mainfrom
aljoscha:adapter-preserve-reconfiguration-deadline-on-fold

Conversation

@aljoscha

Copy link
Copy Markdown
Contributor

Motivation

A config-shape ALTER CLUSTER that arrives while a graceful reconfiguration is
already in flight folds onto the in-flight durable reconfiguration record. The
target shape folds per dimension, but the record's deadline and on_timeout
action were recomputed from scratch from the new statement's WITH (WAIT ...)
surface and written wholesale, even when the new statement carried no WAIT
clause at all.

A statement with no WAIT clause has no deadline contract of its own: it
resolves to the default_cluster_reconfiguration_timeout default (24h) and an
implicit ON TIMEOUT ROLLBACK. Folding it onto an in-flight record therefore
silently replaced the deadline and timeout action that the statement which
started the reconfiguration had set. An unrelated ALTER from another session
could stretch a user's explicit short deadline out to 24h.

That is the nightly failure this closes. A user ALTER started a 120s
ON TIMEOUT ROLLBACK reconfiguration on a cluster. Roughly 14s later an
unrelated, system-issued ALTER CLUSTER ... SET (EXPERIMENTAL ARRANGEMENT COMPRESSION = true) landed on the same cluster. Arrangement compression is a
replica-shape dimension, so that statement folds onto the in-flight record too,
and being a no-WAIT ALTER it reset the deadline to ~24h out. The
reconfiguration could no longer roll back at its 120s deadline, so it stayed
parked in-progress, and the test observed Graceful reconfiguration ... did not complete (reconfiguration status: in-progress).

Description

Resolve the deadline and on-timeout action based on whether the folding ALTER
carries a WAIT clause:

  • No WAIT clause, reconfiguration in flight: keep the in-flight record's
    deadline and on_timeout. The statement carries no contract of its own, so
    it must not touch the one the user set on the reconfiguration in progress.
  • No WAIT clause, nothing in flight: unchanged. The system-default timeout
    and the implicit ROLLBACK action.
  • Explicit WAIT FOR / WAIT UNTIL READY: unchanged. Their deadline and
    action fold onto the in-flight record wholesale, which is what lets a user
    steer an in-flight reconfiguration (adjust the deadline or the timeout
    action) without discarding hydration progress already made on the target.

Only the no-WAIT-onto-in-flight case changes behavior.

User-visible effect: an unrelated ALTER CLUSTER that omits WITH (WAIT ...)
no longer resets the deadline or ON TIMEOUT action of a graceful
reconfiguration already in progress. The contract set by the ALTER that
started (or last steered) the reconfiguration is preserved.

Out of scope

Whether to outright deny config-shape ALTERs while a reconfiguration is in
flight (today only SCHEDULE and REPLICATION FACTOR changes are rejected) is
a separate design question, deferred.

Tests

Added a case to test/testdrive/cluster-controller.td: a sleeping materialized
view keeps a target from hydrating, an ALTER starts a reconfiguration with
WAIT UNTIL READY (TIMEOUT '600s', ON TIMEOUT 'ROLLBACK'), then an unrelated
no-WAIT SET (EXPERIMENTAL ARRANGEMENT COMPRESSION = true) folds on. The test
asserts the deadline is unchanged, on_timeout stays rollback, status stays
in-progress, and the size dimension the second ALTER did not mention keeps
the in-flight target's value. Before the fix the deadline jumped to now + 24h,
failing the equality assertion.

Fixes SQL-568.

https://linear.app/materializeinc/issue/SQL-568/graceful-reconfiguration-of-cluster-cluster-3-to-size-scale1workers1

## Motivation

A config-shape `ALTER CLUSTER` that arrives while a graceful reconfiguration is
already in flight folds onto the in-flight durable reconfiguration record. The
target shape folds per dimension, but the record's `deadline` and `on_timeout`
action were recomputed from scratch from the *new* statement's `WITH (WAIT ...)`
surface and written wholesale, even when the new statement carried no `WAIT`
clause at all.

A statement with no `WAIT` clause has no deadline contract of its own: it
resolves to the `default_cluster_reconfiguration_timeout` default (24h) and an
implicit `ON TIMEOUT ROLLBACK`. Folding it onto an in-flight record therefore
silently replaced the deadline and timeout action that the statement which
*started* the reconfiguration had set. An unrelated `ALTER` from another session
could stretch a user's explicit short deadline out to 24h.

That is the nightly failure this closes. A user `ALTER` started a 120s
`ON TIMEOUT ROLLBACK` reconfiguration on a cluster. Roughly 14s later an
unrelated, system-issued `ALTER CLUSTER ... SET (EXPERIMENTAL ARRANGEMENT
COMPRESSION = true)` landed on the same cluster. Arrangement compression is a
replica-shape dimension, so that statement folds onto the in-flight record too,
and being a no-`WAIT` `ALTER` it reset the deadline to ~24h out. The
reconfiguration could no longer roll back at its 120s deadline, so it stayed
parked `in-progress`, and the test observed `Graceful reconfiguration ... did
not complete (reconfiguration status: in-progress)`.

## Description

Resolve the deadline and on-timeout action based on whether the folding `ALTER`
carries a `WAIT` clause:

- **No `WAIT` clause, reconfiguration in flight:** keep the in-flight record's
  `deadline` and `on_timeout`. The statement carries no contract of its own, so
  it must not touch the one the user set on the reconfiguration in progress.
- **No `WAIT` clause, nothing in flight:** unchanged. The system-default timeout
  and the implicit `ROLLBACK` action.
- **Explicit `WAIT FOR` / `WAIT UNTIL READY`:** unchanged. Their deadline and
  action fold onto the in-flight record wholesale, which is what lets a user
  *steer* an in-flight reconfiguration (adjust the deadline or the timeout
  action) without discarding hydration progress already made on the target.

Only the no-`WAIT`-onto-in-flight case changes behavior.

User-visible effect: an unrelated `ALTER CLUSTER` that omits `WITH (WAIT ...)`
no longer resets the deadline or `ON TIMEOUT` action of a graceful
reconfiguration already in progress. The contract set by the `ALTER` that
started (or last steered) the reconfiguration is preserved.

## Out of scope

Whether to outright *deny* config-shape `ALTER`s while a reconfiguration is in
flight (today only `SCHEDULE` and `REPLICATION FACTOR` changes are rejected) is
a separate design question, deferred.

## Tests

Added a case to `test/testdrive/cluster-controller.td`: a sleeping materialized
view keeps a target from hydrating, an `ALTER` starts a reconfiguration with
`WAIT UNTIL READY (TIMEOUT '600s', ON TIMEOUT 'ROLLBACK')`, then an unrelated
no-`WAIT` `SET (EXPERIMENTAL ARRANGEMENT COMPRESSION = true)` folds on. The test
asserts the deadline is unchanged, `on_timeout` stays `rollback`, status stays
`in-progress`, and the size dimension the second `ALTER` did not mention keeps
the in-flight target's value. Before the fix the deadline jumped to `now + 24h`,
failing the equality assertion.

Fixes SQL-568.

https://linear.app/materializeinc/issue/SQL-568/graceful-reconfiguration-of-cluster-cluster-3-to-size-scale1workers1
@aljoscha
aljoscha marked this pull request as ready for review July 24, 2026 15:22
@aljoscha
aljoscha requested a review from a team as a code owner July 24, 2026 15:22
@aljoscha
aljoscha requested review from def- and mtabebe July 24, 2026 15:44
@aljoscha aljoscha closed this Jul 24, 2026
@aljoscha aljoscha reopened this Jul 24, 2026

> DROP CLUSTER cc_force_existing CASCADE

# A folding ALTER without a WAIT clause must not touch the in-flight deadline

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants