diff --git a/doc/developer/generated/adapter-types/cluster_state.md b/doc/developer/generated/adapter-types/cluster_state.md index 056bc75a62a03..fffbb7436a1b4 100644 --- a/doc/developer/generated/adapter-types/cluster_state.md +++ b/doc/developer/generated/adapter-types/cluster_state.md @@ -1,6 +1,6 @@ --- source: src/adapter-types/src/cluster_state.rs -revision: 74f18a3354 +revision: 6eeaca032b --- # adapter-types::cluster_state @@ -11,7 +11,7 @@ These types carry the slices of a managed cluster's config that a caller reasons Key types: -* `ExpectedClusterState` — compare-and-append witness over a managed cluster's durable config. A caller captures it from a config snapshot and pairs it with a conditional write. The applier applies the write only if the cluster's current config still projects to an equal witness. Fields: `size`, `replication_factor`, `availability_zones`, `logging`, `auto_scaling_policy`, `reconfiguration`, `burst`. +* `ExpectedClusterState` — compare-and-append witness over a managed cluster's durable config. A caller captures it from a config snapshot and pairs it with a conditional write. The applier applies the write only if the cluster's current config still projects to an equal witness. Fields: `size`, `replication_factor`, `availability_zones`, `logging`, `arrangement_compression`, `schedule`, `auto_scaling_policy`, `reconfiguration`, `burst`. * `ReplicaShape` — the config dimensions that distinguish one replica from another. Two replicas with equal shape are interchangeable. `ReplicaShape::matches` compares availability zones as unordered pools so a reorder alone does not force a reprovision. * `AvailabilityZones` — the availability zones in configured provisioning order. Order is significant (the orchestrator round-robins placement across the list) and is part of the `ExpectedClusterState` witness. `AvailabilityZones::pool` converts to an unordered `AvailabilityZonePool` for interchangeability checks. * `AvailabilityZonePool` — unordered set of zones produced by `AvailabilityZones::pool`. Used for the one comparison that must ignore order: replica interchangeability. @@ -19,7 +19,8 @@ Key types: * `ReconfigurationStatus` — lifecycle status of a graceful reconfiguration: `InProgress`, `Finalized`, `TimedOut`, `Cancelled`, `ResourceExhausted`. * `ReconfigurationAudit` — the lifecycle transition a write to the `reconfiguration` record represents, declared by the writer at the decision point: `Started`, `Cancelled`, `Finalized { forced }`, `TimedOut`, `ResourceExhausted`. * `OnTimeout` — the action a graceful reconfiguration applies once its deadline passes with the target not yet hydrated: `Commit` (cut over anyway) or `Rollback` (revert to pre-reconfiguration shape). -* `ReconfigurationTarget` — the full config shape a reconfiguration is moving to: `size`, `replication_factor`, `availability_zones`, `logging`. `shape()` returns the per-replica `ReplicaShape` (everything but `replication_factor`). +* `ClusterSchedule` — the scheduling policy of a managed cluster: `Manual` (user-controlled replication factor) or `Refresh { hydration_time_estimate }` (cluster runs only around refresh times). A plain-data mirror of `mz_sql::plan::ClusterSchedule`. +* `ReconfigurationTarget` — the full config shape a reconfiguration is moving to: `size`, `replication_factor`, `availability_zones`, `logging`, `arrangement_compression`. `shape()` returns the per-replica `ReplicaShape` (everything but `replication_factor`). * `BurstRecord` — active hydration-burst record: `burst_size`, `linger_duration`, `steady_hydrated_at`. * `BurstAudit` — lifecycle transition for a burst record: `Started` or `Finished { cause }`. * `BurstFinishCause` — why a hydration burst finished: `LingerElapsed` or `NoLongerWarranted`. diff --git a/doc/developer/generated/adapter/catalog/_module.md b/doc/developer/generated/adapter/catalog/_module.md index 203d21cf73b6c..f05a1c38f46d6 100644 --- a/doc/developer/generated/adapter/catalog/_module.md +++ b/doc/developer/generated/adapter/catalog/_module.md @@ -1,12 +1,13 @@ --- source: src/adapter/src/catalog.rs -revision: 605cc6fd1a +revision: a60edac7f1 --- # adapter::catalog The coordinator's catalog layer: wraps the `mz_catalog` crate's durable store and in-memory objects with the additional adapter-specific logic needed to drive DDL and serve catalog queries. `Catalog` is the top-level struct exposing the full catalog API: it holds a `CatalogState` (in-memory view), an `ExpressionCacheHandle`, a `transient_revision` counter, a `shared_transient_revision: Arc` shared across all clones, and delegates persistence to the durable store. `Catalog::transient_revision_is_current` compares the clone's frozen revision against the shared latest, letting a snapshot holder detect staleness from off-thread without a Coordinator round-trip. +`CatalogUpperHandle` is a handle for advancing the durable catalog upper off the coordinator loop; its `advance_upper` method advances the upper to at least a given timestamp. Child modules divide responsibilities: `state` owns the in-memory data structures and `SessionCatalog` impl; `transact` executes atomic DDL operations; `apply` reconciles durable diffs into in-memory state; `open` bootstraps the catalog at startup; `migrate` rewrites stored SQL for syntax changes; `builtin_table_updates` maintains system-table row diffs; `consistency` provides invariant checking; `timeline` resolves timeline contexts for timestamp selection; and `cluster_state` projects live cluster config into plain-data `ExpectedClusterState` structs for conditional catalog writes. Re-exports include `InjectedAuditEvent` from `transact`. Builtin materialized views are handled alongside tables and views in descriptor tests. `is_reserved_role_name` returns true for names that match `is_reserved_name`, `is_public_role`, or the `RESERVED_ROLE_SPECIFICATION_NAMES` list. `RESERVED_ROLE_SPECIFICATION_NAMES` contains the five lowercase PostgreSQL role-specification keywords (`current_user`, `current_role`, `session_user`, `user`, `none`) that would be ambiguous in statements like `GRANT ... TO CURRENT_USER`; only the lowercase spellings are reserved, matching what unquoted identifiers normalize to. diff --git a/doc/developer/generated/adapter/catalog/apply.md b/doc/developer/generated/adapter/catalog/apply.md index b00b1d93dd36a..b888012aea3f3 100644 --- a/doc/developer/generated/adapter/catalog/apply.md +++ b/doc/developer/generated/adapter/catalog/apply.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/apply.rs -revision: 584bb9030c +revision: fca741734d --- # adapter::catalog::apply diff --git a/doc/developer/generated/adapter/catalog/cluster_state.md b/doc/developer/generated/adapter/catalog/cluster_state.md index e3de982434d58..1155324358440 100644 --- a/doc/developer/generated/adapter/catalog/cluster_state.md +++ b/doc/developer/generated/adapter/catalog/cluster_state.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/cluster_state.rs -revision: 74f18a3354 +revision: 6eeaca032b --- # adapter::catalog::cluster_state diff --git a/doc/developer/generated/adapter/catalog/open.md b/doc/developer/generated/adapter/catalog/open.md index 85bfdf20a659d..2245ee04b37f8 100644 --- a/doc/developer/generated/adapter/catalog/open.md +++ b/doc/developer/generated/adapter/catalog/open.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/open.rs -revision: dbd2c3fc06 +revision: fca741734d --- # adapter::catalog::open diff --git a/doc/developer/generated/adapter/catalog/open/_module.md b/doc/developer/generated/adapter/catalog/open/_module.md index 72f0feb241982..b59913fd7d8fb 100644 --- a/doc/developer/generated/adapter/catalog/open/_module.md +++ b/doc/developer/generated/adapter/catalog/open/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/open.rs -revision: dbd2c3fc06 +revision: fca741734d --- # adapter::catalog::open diff --git a/doc/developer/generated/adapter/catalog/transact.md b/doc/developer/generated/adapter/catalog/transact.md index 07f611759471a..f8ba0868316be 100644 --- a/doc/developer/generated/adapter/catalog/transact.md +++ b/doc/developer/generated/adapter/catalog/transact.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/transact.rs -revision: 74f18a3354 +revision: 6eeaca032b --- # adapter::catalog::transact @@ -10,13 +10,12 @@ The module enforces referential integrity, privilege checks, and naming constrai Zero-downtime deployment logic (0dt) is also managed here; DDL operations can be gated behind deployment-readiness checks during the cutover window. `InjectedAuditEvent` struct and `Op::InjectAuditEvents` variant allow manually appending audit log entries at the current oracle write timestamp. `CREATE ROLE` audit events use `EventDetails::CreateRoleV1` which includes the `auto_provision_source` field. `Op::AlterAddColumn` emits an `AlterAddColumnV1` audit log event recording the table ID, column name, column type, and nullability. `Op::AlterTimestampInterval` emits an `AlterSourceTimestampIntervalV1` audit log event recording the item ID, old interval, and new interval. -`Op::WeirdStorageUsageUpdates` has been removed; storage usage id allocation is handled outside the catalog transaction via `Catalog::allocate_storage_usage_id`. `Op::UpdateScopedSystemParameters { scoped, prune_scope }` persists the durable cache of scoped (per-cluster and per-replica) system parameters. `scoped` is the desired sparse state; `prune_scope` is an optional `ScopedParametersScope` bounding which objects' rows may be removed. The handler diffs against the current durable contents: it upserts changed or added entries, removes entries for objects within the prune scope that are no longer present in the desired state (a `None` prune_scope means all objects are in scope), and lazily prunes entries whose owning object id is absent from the catalog (object ids are never reused, so such entries are inert). `Op::CreateClusterReplica` carries a required `replica_id: ReplicaId` field that must be pre-allocated out-of-band by the caller via the durable allocator (mirroring how cluster and item IDs are allocated). The apply path always uses `insert_cluster_replica_with_id` and never allocates a replica ID in-apply. `Op::CheckClusterState { cluster_id, expected }` is a precondition op, not a mutation: it aborts the whole transaction with `AdapterError::ClusterStateChanged` unless the cluster's current managed config equals `expected`. This gives a compare-and-append semantic over the cluster's config, inseparable from the commit it guards. It is a purely internal op and is never emitted by SQL DDL. `ReplicaCreateDropReason::Retired` is the audit-log reason emitted by the cluster controller when it drops a replica because the cluster's configuration no longer calls for it (e.g. a replication-factor decrease). It maps to `CreateOrDropClusterReplicaReasonV1::Retired` with no scheduling-decisions blob. `ReplicaCreateDropReason::HydrationBurst` is the audit-log reason emitted by the cluster controller's hydration-burst strategy when it creates the transient burst replica it runs while a cluster's objects are not yet hydrated. It maps to `CreateOrDropClusterReplicaReasonV1::HydrationBurst` with no scheduling-decisions blob. -`Op::UpdatePrivilege` carries `privileges: Vec` (previously a single `privilege: MzAclItem`), allowing a bulk `GRANT`/`REVOKE` touching one object for many grantees to land as a single durable write while still emitting one audit event per grantee. +`Op::UpdatePrivilege` carries `privileges: Vec`, allowing a bulk `GRANT`/`REVOKE` touching one object for many grantees to land as a single durable write while still emitting one audit event per grantee. When dropping a cluster replica, the code expands dependent objects (including materialized views that target the replica) using `state.cluster_replica_dependents(cluster_id, replica_id, &mut seen)`, records their comments for cleanup, and adds them to the drop list; `seen` is seeded from already-collected items so the plan-driven path (which processes dependents before the replica in reverse-dependency order) does not re-add them. When dropping items, all storage collections associated with a dropped entry are scheduled for removal unconditionally; whether the item was a replacement target is not considered at this stage. `Op::UpdateItem` enforces that a non-temporary item must not depend on a temporary one, mirroring the same invariant already enforced by `Op::CreateItem`. Temporary objects are session-scoped and never persisted; a durable item referencing one is a dangling reference that would panic the coordinator when its `create_sql` is re-planned on catalog apply (apply emits durable items before temporary ones). This prevents ALTER paths such as `ALTER SINK ... SET FROM` from repointing a persistent item at a temporary object. diff --git a/doc/developer/generated/adapter/coord/_module.md b/doc/developer/generated/adapter/coord/_module.md index 478e9701497d3..950f9fbe4bf45 100644 --- a/doc/developer/generated/adapter/coord/_module.md +++ b/doc/developer/generated/adapter/coord/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord.rs -revision: 55e361b0a6 +revision: a60edac7f1 --- # adapter::coord @@ -8,12 +8,14 @@ revision: 55e361b0a6 The core coordinator: `coord.rs` defines the `Coordinator` struct (the central state machine), `ExecuteContext` (the per-statement execution handle), the internal `Message` enum, and the `serve` / `Config` entry points. The coordinator owns the `Catalog`, active compute sinks, pending peeks, read-policy manager, timeline oracles, and all inter-subsystem handles (controller, storage collections, secrets, orchestrator). The `Coordinator` struct holds a `reconcile_now: Arc` field that wakes the cluster controller task to reconcile immediately after catalog transactions that change durable cluster state, rather than waiting out the tick interval. The file also defines `IdPool`, a pre-allocated pool of user `GlobalId` integers that amortizes per-DDL persist writes by reserving batches of IDs at once; the pool is owned by the coordinator and access is serialized through its single-threaded event loop. +`ArrangementSizeRecord` is a struct for rows destined for `mz_object_arrangement_size_history`, carrying `replica_id`, `object_id`, `size`, and `hydration_complete`; records are prepared off-thread by the arrangement sizes snapshot task and stamped with a collection timestamp on the coordinator loop at write time. Child modules partition the coordinator's responsibilities: `command_handler` handles external `Command` messages; `message_handler` handles internal async `Message` responses; `sequencer` executes SQL plans; `appends` manages table and builtin-table writes; `catalog_implications` derives and applies downstream effects from catalog state changes, including pushing replica-scoped dyncfg override layers to the compute controller via `push_replica_dyncfg_overrides`; `ddl` wraps catalog transactions; `peek` and `read_policy` manage query execution and compaction; `read_then_write` implements the read-then-write protocol for DML (INSERT, UPDATE, DELETE) including dependency validation; `group_sync` computes the diff between sync-managed and manually-granted role memberships for SSO group synchronization; `timestamp_selection` and `timeline` handle temporal reasoning; `catalog_serving` serves catalog snapshots; `info_metrics` owns the catalog `*_info` Prometheus series background task; `cluster_controller` scaffolds the cluster controller task that reconciles durable cluster state with the live controller; and supporting modules cover cluster scheduling, unified compute introspection subscribes, consistency checking, index management, and statement logging. The `Coordinator` struct holds a `scoped_frontend: Option>` field populated by `Command::InstallScopedSystemParameterFrontend`. `reconcile_scoped_system_parameters` persists the diff between the current durable working copy and the desired `ScopedParameters` via `Op::UpdateScopedSystemParameters`, which also updates the in-memory working copy and introspection relations. `scoped_overrides_create_op` evaluates scoped overrides for freshly-created clusters and replicas from explicit `ClusterEvalContext` / `ReplicaEvalContext` values and returns an `Op::UpdateScopedSystemParameters` to fold into the same transaction that creates them, so the committed diff drives the replica-scoped controller push before `create_replica`. `push_replica_dyncfg_overrides` propagates the replica-scoped overrides from the catalog working copy to the compute controller's per-replica dyncfg layer. `cluster_scoped_optimizer_overrides` retrieves the cluster-coherent `OptimizerFeatureOverrides` from the catalog working copy for use at plan time. The `Coordinator` struct holds a `catalog_info_metrics_registry: MetricsRegistry` field used to hand the metrics registry to the catalog info-metrics background task spawned at bootstrap. Bootstrap handles derived builtin storage collections (builtin MVs) separately: after registering input-less collections in dependency order, it bumps their sinces based on transitive dependency frontiers to satisfy as-of selection invariants. Bootstrap restores compaction policies for all MV version IDs (via `global_ids()`, not just the write ID) so that no version blocks capability propagation through the `primary` ownership chain and pins compaction. When applying replacement MVs, each `CollectionDescription` receives a `primary` field pointing to its predecessor's latest collection ID, chaining shard ownership from the oldest version through each replacement in order. Bootstrap also calls `push_replica_dyncfg_overrides` once before dataflows are rendered so that replica-scoped dyncfg flags take effect on the first configuration of each replica, and calls `spawn_cluster_controller_task` to start the cluster controller background task. The `clusters_caught_up_check` built during `serve` excludes both migrated MVs and new builtin MVs (and their transitive dependents) from the caught-up frontier check: migrated MV dataflows do not advance their write frontier in read-only mode, and new builtin MVs have no writer until the deployment promotes, so both sets would otherwise stall the check. The `Message` enum includes a `ClusterControllerRequest(cluster_controller::ClusterControllerRequest)` variant, carrying one pull/apply call from the cluster controller task to be answered on the main coordinator message loop. +`Message::ArrangementSizesWrite(Vec)` carries the records prepared by the off-thread arrangement sizes snapshot task back to the coordinator loop for stamping and appending to `mz_object_arrangement_size_history`. `ship_dataflow`, `try_ship_dataflow`, and `ship_dataflow_and_notice_builtin_table_updates` accept `DataflowDescription` (previously `DataflowDescription`). `ExecuteContextInner` carries a `response_barriers: Vec` field (Debug-ignored). `ExecuteContext::from_parts` is a convenience wrapper around `from_parts_with_response_barriers`, which accepts an explicit barrier list. `into_parts` returns the barrier list as a fifth element alongside `tx`, `internal_cmd_tx`, `session`, and `extra`; callers that repack an `ExecuteContext` must preserve and forward these barriers. `delay_response_until` appends a `BuiltinTableAppendCompletion` barrier; on retirement, the response is held until all barriers resolve. `ExplainTimestampStage` has four variants in order: `Optimize`, `RealTimeRecency`, `LinearizeTimestamp` (carries `ExplainTimestampLinearizeTimestamp`: validity, format, optimized_plan, cluster_id, source_ids, when, real_time_recency_ts), and `Finish` (carries `ExplainTimestampFinish`: validity, format, cluster_id, source_ids, when, real_time_recency_ts, timeline_context, oracle_read_ts). The `LinearizeTimestamp` stage reads the oracle timestamp off the coordinator loop and passes it forward as `oracle_read_ts` in `ExplainTimestampFinish`. diff --git a/doc/developer/generated/adapter/coord/appends.md b/doc/developer/generated/adapter/coord/appends.md index 95528aad1d7e0..a131d1e4d0d10 100644 --- a/doc/developer/generated/adapter/coord/appends.md +++ b/doc/developer/generated/adapter/coord/appends.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/appends.rs -revision: 1c17d34993 +revision: a60edac7f1 --- # adapter::coord::appends @@ -10,3 +10,4 @@ Implements all append operations executed by the coordinator: group-commit of us `BuiltinTableAppendNotify` is a pinned future that resolves when a builtin-table write completes. `BuiltinTableAppendCompletion` is a completion handle wrapping a `BuiltinTableAppendNotify`; callers obtain one to use as a response barrier and convert it to a notify via `into_notify`. `GroupCommitWriteLocks` manages the locking primitives used to serialise group commits. `group_commit` always performs the append call even when there are no user writes, relying on the storage layer to periodically advance the upper of all tables; it does not iterate catalog entries to inject empty advancement entries per table. `UserWriteResponder` is an enum with a single `Session(PendingTxn)` variant that wraps the per-session `PendingTxn` for `PendingWriteTxn::User`; the `User` variant's `pending_txn` field is replaced by `responder: UserWriteResponder`. The write timestamp for `User` writes is picked by the oracle during group commit; the write lock is either handed off from the submitting session (`write_locks: Some(..)`) or acquired during group commit (`write_locks: None`). +`GroupCommitter` serializes runtime txns-shard writes off the coordinator loop via `TableWriteCmd` variants: `GroupCommit`, `Register`, and `Forget`. `DeferredOp` represents an operation awaiting a resource; `DeferredOp::Plan` wraps a `DeferredPlan` that must uniquely hold write locks (e.g. UPDATE), while `DeferredOp::Write` wraps a `DeferredWrite` for blind inserts that can be optimistically retried. `GroupCommitRequest` accumulates appends, responses, statement-logging IDs, notifies, write locks, and permits for a single commit batch. diff --git a/doc/developer/generated/adapter/coord/catalog_implications.md b/doc/developer/generated/adapter/coord/catalog_implications.md index 8708da0f67d4d..db867c6f56458 100644 --- a/doc/developer/generated/adapter/coord/catalog_implications.md +++ b/doc/developer/generated/adapter/coord/catalog_implications.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/catalog_implications.rs -revision: 1c17d34993 +revision: a60edac7f1 --- # adapter::coord::catalog_implications diff --git a/doc/developer/generated/adapter/coord/cluster_controller.md b/doc/developer/generated/adapter/coord/cluster_controller.md index 9b2fbc04f8bb8..b503c14d2acc3 100644 --- a/doc/developer/generated/adapter/coord/cluster_controller.md +++ b/doc/developer/generated/adapter/coord/cluster_controller.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/cluster_controller.rs -revision: 74f18a3354 +revision: 6eeaca032b --- # adapter::coord::cluster_controller @@ -9,15 +9,15 @@ Driver and glue for the `mz-cluster-controller` reconciler. The controller crate is pure and knows nothing about the Coordinator. This module is the half of the `ClusterControllerCtx` boundary that does: it runs the controller as a separate task and implements the ctx by marshaling each pull/apply to the Coordinator over the internal command channel, because the catalog and live compute/storage signals are reachable only from the coordinator loop. The two whole-tick reads are batched; the per-cluster live signals are pulled on demand, so a tick's round-trips scale with the number of managed clusters that need a live signal, not with a constant. -Everything here is gated by `ENABLE_CLUSTER_CONTROLLER` (default off). With the gate off the task does not tick, so the legacy scheduling and graceful paths remain the sole writers of the replica set. With the gate on the controller owns the *user* managed-cluster replica set; the legacy entry points no-op. System/builtin clusters are never controller-owned: the catalog's bootstrap migration owns their replicas. +Everything here is gated by `ENABLE_CLUSTER_CONTROLLER` (default on). With the gate off the task does not tick, so the legacy scheduling and graceful paths remain the sole writers of the replica set. With the gate on the controller owns the *user* managed-cluster replica set; the legacy entry points no-op. System/builtin clusters are never controller-owned: the catalog's bootstrap migration owns their replicas. Key types and functions: -* `ClusterControllerRequest` — the enum of requests the controller task marshals to the Coordinator: `ManagedClusterIds`, `ClusterStates` (batched read; reply also carries `now`), `HydratedReplicas` (per-cluster live signal a strategy pulls on demand), `HasHydratableObjects` (per-cluster live signal reporting whether the cluster has any dataflow-backed objects bound to it), `Apply` (tick's decision batch), `TickInterval`. Each variant carries a oneshot for the reply. +* `ClusterControllerRequest` — the enum of requests the controller task marshals to the Coordinator: `ManagedClusterIds`, `ClusterStates` (batched read; reply also carries `now`), `HydratedReplicas` (per-cluster live signal a strategy pulls on demand), `HasHydratableObjects` (per-cluster live signal reporting whether the cluster has any dataflow-backed objects bound to it), `RefreshWindowInputs` (per-cluster live signal supplying the oracle read timestamp, compaction estimate, and bound REFRESH MV frontiers for scheduled clusters; `None` for non-scheduled clusters), `Apply` (tick's decision batch), `TickInterval`. Each variant carries a oneshot for the reply. * `CoordCtx` — the controller-task side of the `ClusterControllerCtx` boundary. Marshals every ctx call to the Coordinator via `internal_cmd_tx`. Latches `now` from the most recent `ClusterStates` reply so both phases of a tick see a consistent time. -* `Coordinator::spawn_cluster_controller_task` — spawns the controller task. The task loops: reads the tick interval via a `TickInterval` request, sleeps, then calls `ClusterController::reconcile` if the gate is on. Called once at bootstrap. The `ClusterController` is constructed with a shared dyncfg handle so flag flips take effect without a restart. -* `Coordinator::handle_cluster_controller_request` — the Coordinator-side handler for `Message::ClusterControllerRequest`. Dispatches each variant to the appropriate catalog/controller reads or applies the decision batch via `apply_cluster_controller_decisions`. For `HydratedReplicas`, starts per-replica hydration checks on the coordinator loop (via `start_hydration_checks`) then waits for compute's replies off-loop in a spawned task. For `HasHydratableObjects`, delegates to `cluster_has_hydratable_objects`. -* `Coordinator::start_hydration_checks` — checks storage and compute hydration for a set of replicas on a cluster. Returns only replicas that are already storage-hydrated and known to the compute controller. Replica-pinned materialized views (those with `IN CLUSTER ... REPLICA`) are excluded from the hydration check for replicas they are not pinned to, so a graceful reconfiguration does not wait forever for a new replica to hydrate an MV bound to a replica being replaced. +* `Coordinator::spawn_cluster_controller_task` — spawns the controller task. The task loops: reads the tick interval via a `TickInterval` request, then waits for either the sleep to elapse or a `reconcile_now` notification (sent after catalog transactions that change durable cluster state), then calls `ClusterController::reconcile`. The gate and tick interval are re-read each iteration so runtime changes take effect without a restart. Called once at bootstrap. The `ClusterController` is constructed with a shared dyncfg handle so flag flips take effect without a restart. +* `Coordinator::handle_cluster_controller_request` — the Coordinator-side handler for `Message::ClusterControllerRequest`. The controller is inactive when the gate is off or while the deployment is in read-only mode; when inactive, `ManagedClusterIds` returns an empty list and `Apply` is rejected, so no catalog state is read or written. Dispatches each variant to the appropriate catalog/controller reads or applies the decision batch via `apply_cluster_decisions`. For `HydratedReplicas`, starts per-replica hydration checks on the coordinator loop (via `start_hydration_checks`) then waits for compute's replies off-loop in a spawned task. For `HasHydratableObjects`, delegates to `cluster_has_hydratable_objects`. For `RefreshWindowInputs`, reads catalog and storage inputs on the coordinator loop then completes the oracle timestamp read in a spawned task. +* `Coordinator::start_hydration_checks` — checks that replicas are online, storage-hydrated, and known to the compute controller for a set of replicas on a cluster. Returns only replicas whose processes are all online, that are already storage-hydrated, and that are known to the compute controller. Replica-pinned materialized views (those with `IN CLUSTER ... REPLICA`) are excluded from the hydration check for replicas they are not pinned to, so a graceful reconfiguration does not wait forever for a new replica to hydrate an MV bound to a replica being replaced. * `Coordinator::cluster_has_hydratable_objects` — returns whether any object bound to the cluster is hydratable (dataflow-backed). Backs the `HasHydratableObjects` ctx pull; a `false` answer prevents the burst strategy from arming when there is nothing to hydrate. -* `Coordinator::apply_cluster_controller_decisions` — transacts a decision batch: for each `Decision`, checks `check_cluster_state` and, if the state still matches, creates or drops the replica or writes the state. Returns `ApplyOutcome::Rejected` if any check fails, `ApplyOutcome::ResourceExhausted` if the batch exceeds the resource budget, and `Applied` otherwise. Create decisions carry a strategy attribution translated to a `ReplicaCreateDropReason` via `reason_from_strategies`: the graceful reconfiguration strategy maps to `GracefulReconfiguration`; the hydration-burst strategy maps to `HydrationBurst`; all others map to `Manual`. `build_mutation_ops` (called from `apply_cluster_controller_decisions`) checks whether the target replica of a `DropReplica` decision still exists in the catalog before building the drop op. If it has already been removed (e.g. by a concurrent user `DROP CLUSTER`), the function returns `None` to reject the batch, avoiding a panic from resource-limit validation running against a missing replica. +* `Coordinator::apply_cluster_decisions` — transacts a decision batch in four steps: collect per-cluster compare-and-append guards, pre-allocate replica ids for creates, build mutation ops, then commit guards and mutations together. Returns `ApplyOutcome::Rejected` if any step finds the batch incoherent (missing cluster, allocation failure, or a concurrent `ALTER` that moved cluster state), `ApplyOutcome::ResourceExhausted` if the batch exceeds the resource budget, and `Applied` otherwise. Create decisions carry a strategy attribution translated to a `ReplicaCreateDropReason` via `audit_reason_for_create`: `CreateReason::Baseline` maps to `Manual`, `GracefulReconfiguration` maps to `GracefulReconfiguration`, `HydrationBurst` maps to `HydrationBurst`, and `OnRefresh` maps to `OnRefresh` carrying the window decision through to the audit detail. All drops are uniformly attributed `Retired`. `build_mutation_ops` checks whether the target replica of a `DropReplica` decision still exists in the catalog before building the drop op; if it has already been removed (e.g. by a concurrent user `DROP CLUSTER`), the function returns `None` to reject the batch, avoiding a panic from resource-limit validation running against a missing replica. * Observed replicas passed to `ClusterController::reconcile` are built from all replicas on the cluster (not pre-filtered): each `ObservedReplica` carries `internal`, `billed_as`, and `pending` flags so the controller's ownership classifier (`ObservedReplica::owned_shape`) can decide which replicas it governs. The `StateWrite` applied to the managed config is exhaustively destructured (no `..`) so that adding a field to `StateWrite` is a compile error until the overlay logic accounts for it. diff --git a/doc/developer/generated/adapter/coord/cluster_scheduling.md b/doc/developer/generated/adapter/coord/cluster_scheduling.md index 88fd932988542..2f005d8a59328 100644 --- a/doc/developer/generated/adapter/coord/cluster_scheduling.md +++ b/doc/developer/generated/adapter/coord/cluster_scheduling.md @@ -1,10 +1,10 @@ --- source: src/adapter/src/coord/cluster_scheduling.rs -revision: 8598d82c1c +revision: 6eeaca032b --- # adapter::coord::cluster_scheduling -Implements automated cluster scheduling policies: currently the `REFRESH HYDRATION TIME ESTIMATE` policy, which keeps track of per-cluster hydration time estimates and uses them to schedule cluster replica creation and teardown. -`check_scheduling_policies` is called on a timer from the coordinator's main loop and emits `AlterCluster` operations when scheduling decisions change. -When `ENABLE_CLUSTER_CONTROLLER` is on and the cluster is user-owned, `handle_scheduling_decisions` defers its replication-factor write for any cluster that has a graceful reconfiguration in flight, because the controller owns that transition. The deferral is skipped when the gate is off (the legacy path retains the record as cancelled instead). +Implements the legacy automated cluster scheduling policy: `SCHEDULE = ON REFRESH`, which keeps track of per-cluster hydration time estimates and uses them to schedule cluster replica creation and teardown. +Both `check_scheduling_policies` and `handle_scheduling_decisions` are no-ops when `ENABLE_CLUSTER_CONTROLLER` (default on) is set: the controller's `OnRefreshStrategy` is then the sole authority over scheduled clusters, so the legacy policy must not also toggle their replication factor. +When the gate is off, `check_scheduling_policies` is called on a timer from the coordinator's main loop and sends `Message::SchedulingDecisions` to drive `handle_scheduling_decisions`. `handle_scheduling_decisions` sums decisions across all policies per cluster and turns replicas on or off as needed, but skips any cluster that has a pending (graceful-reconfiguration) replica, because that reconfiguration owns the replica set until it finalizes. When turning a cluster off it drops replicas by id rather than by altering the replication factor, so it can retire replicas left behind by the controller after a gate-off. When the factor is already 0 but owned replicas exist (controller handoff scenario), it adopts one replica and retires any surplus in the same transaction. diff --git a/doc/developer/generated/adapter/coord/ddl.md b/doc/developer/generated/adapter/coord/ddl.md index a63b54ca47548..07fdba1e71768 100644 --- a/doc/developer/generated/adapter/coord/ddl.md +++ b/doc/developer/generated/adapter/coord/ddl.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/ddl.rs -revision: 8598d82c1c +revision: a60edac7f1 --- # adapter::coord::ddl diff --git a/doc/developer/generated/adapter/coord/info_metrics.md b/doc/developer/generated/adapter/coord/info_metrics.md index d1c520936ff1d..e3db10b6f70ef 100644 --- a/doc/developer/generated/adapter/coord/info_metrics.md +++ b/doc/developer/generated/adapter/coord/info_metrics.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/info_metrics.rs -revision: dbd2c3fc06 +revision: fca741734d --- # adapter::coord::info_metrics diff --git a/doc/developer/generated/adapter/coord/introspection.md b/doc/developer/generated/adapter/coord/introspection.md index 3e43a002ae090..60953ddcfbb5d 100644 --- a/doc/developer/generated/adapter/coord/introspection.md +++ b/doc/developer/generated/adapter/coord/introspection.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/introspection.rs -revision: 277b33e9c0 +revision: 7f6c52776d --- # adapter::coord::introspection @@ -8,3 +8,4 @@ revision: 277b33e9c0 Implements unified compute introspection: the process of collecting introspection data exported by individual replicas through their logging indexes and writing that data, tagged with the respective replica ID, to unified storage collections. `install_introspection_subscribes` installs all defined introspection subscribes on a given replica (and `bootstrap_introspection_subscribes` calls it for all existing replicas during coordinator startup); `handle_introspection_subscribe_batch` processes each batch response, writing updates to the corresponding storage-managed collection and reinstalling failed subscribes on disconnect; `drop_introspection_subscribes` removes all subscribes installed on a replica before it is dropped. Each introspection subscribe is sequenced through a multi-stage pipeline (`OptimizeMir` → `TimestampOptimizeLir` → `Finish`) using the `sequence_staged` driver. The optimizer config for introspection subscribes includes cluster-coherent scoped overrides via `Coordinator::cluster_scoped_optimizer_overrides`. +`IntrospectionSubscribe` tracks a `first_data_at: Option` field recording when the subscribe first appended data to its target storage collection in the current process. Rows in that collection before that point may describe a previous environmentd process or a prior replica incarnation and must not be trusted. `invalidate_introspection_freshness(replica_id)` clears `first_data_at` for all subscribes targeting a given replica when a cluster event reports the replica offline or restarted. `fresh_introspection_replicas(introspection_type, margin)` returns the string replica IDs whose subscribe of the given type delivered data at least `margin` ago; consumers such as the arrangement sizes snapshot use this to exclude stale replicas. diff --git a/doc/developer/generated/adapter/coord/message_handler.md b/doc/developer/generated/adapter/coord/message_handler.md index cb8a8cdddde8f..20baf071f7374 100644 --- a/doc/developer/generated/adapter/coord/message_handler.md +++ b/doc/developer/generated/adapter/coord/message_handler.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/message_handler.rs -revision: 1c17d34993 +revision: a60edac7f1 --- # adapter::coord::message_handler @@ -10,3 +10,4 @@ Handles controller responses (compute peek results, subscribe batches, copy-to r This is the heart of the coordinator's reactive loop; every asynchronous response from the storage/compute layers arrives here. `storage_usage_update` obtains a write timestamp from the oracle, allocates a single durable batch id via `Catalog::allocate_storage_usage_id`, builds `BuiltinTableUpdate` rows via `pack_storage_usage_update` for each shard, and submits them via `builtin_table_update().execute()` without going through `catalog_transact_inner`. In read-only mode, `storage_usage_fetch` logs an info message and reschedules via `Message::StorageUsageSchedule` without performing any shard scan or writes. +`ARRANGEMENT_SIZES_FRESHNESS_MARGIN` (10 seconds) is the minimum age of a replica's introspection-subscribe delivery before `arrangement_sizes_snapshot` trusts its data; the margin covers the collection manager's batched write flush and the oracle read timestamp trailing the wall clock. `arrangement_sizes_snapshot` resolves catalog IDs on the main loop, then spawns a task that takes a read timestamp and snapshots `mz_object_arrangement_sizes_unified` and `mz_compute_hydration_times`; the task calls `arrangement_sizes_records` to prepare `Vec` and sends them back as `Message::ArrangementSizesWrite` (or reschedules directly if empty or on failure). `arrangement_sizes_write` receives the prepared records, re-validates replica freshness (cluster events between snapshot and write can invalidate previously-fresh replicas), obtains a write timestamp, packs and appends the rows to `mz_object_arrangement_size_history`, and reschedules the next collection. In read-only mode, `arrangement_sizes_snapshot` reschedules without spawning a task. diff --git a/doc/developer/generated/adapter/coord/sequencer.md b/doc/developer/generated/adapter/coord/sequencer.md index f096886d61bff..6607f7e85eb7c 100644 --- a/doc/developer/generated/adapter/coord/sequencer.md +++ b/doc/developer/generated/adapter/coord/sequencer.md @@ -1,10 +1,11 @@ --- source: src/adapter/src/coord/sequencer.rs -revision: 5dc2cf510a +revision: 6eeaca032b --- # adapter::coord::sequencer Top-level sequencer module: implements `Coordinator::sequence_plan`, which matches on each `Plan` variant and dispatches to the appropriate `sequence_*` method. -The module also contains shared utilities used across statement types: `statistics_oracle` (wraps the transform statistics oracle), `eval_copy_to_uri` (validates the COPY TO URI, accepting `s3://` and `gs://` schemes), and `return_if_err!` macro (short-circuits on error, sending a response to the client). +The module also contains shared utilities used across statement types: `statistics_oracle` (builds a cardinality-estimate oracle for query optimization), `eval_copy_to_uri` (validates the COPY TO URI, accepting `s3://` and `gs://` schemes), `check_log_reads` (validates introspection-source reads and enforces replica targeting on multi-replica clusters), `emit_optimizer_notices` (forwards optimizer notices to the session), `explain_pushdown_future_inner` (computes filter-pushdown statistics for EXPLAIN FILTER PUSHDOWN), `explain_plan_inner` (generates EXPLAIN PLAN output), and the `return_if_err!` macro (short-circuits on error, sending a response to the client). +`cancel_carried_reconfiguration` is re-exported from the `inner` sub-module for use by callers outside the sequencer tree. The `inner` sub-module holds most per-statement implementations; this file ties them together and handles generic concerns like RBAC checks and transaction validation. diff --git a/doc/developer/generated/adapter/coord/sequencer/_module.md b/doc/developer/generated/adapter/coord/sequencer/_module.md index 8899aa13758f7..d46d727bc6220 100644 --- a/doc/developer/generated/adapter/coord/sequencer/_module.md +++ b/doc/developer/generated/adapter/coord/sequencer/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer.rs -revision: 5dc2cf510a +revision: 6eeaca032b --- # adapter::coord::sequencer diff --git a/doc/developer/generated/adapter/coord/sequencer/inner.md b/doc/developer/generated/adapter/coord/sequencer/inner.md index 48edb8174b7f5..90d1ca6bda194 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner.rs -revision: 3713eed996 +revision: 6eeaca032b --- # adapter::coord::sequencer::inner diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/_module.md b/doc/developer/generated/adapter/coord/sequencer/inner/_module.md index 03826fe7e747f..79cce6c865969 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/_module.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner.rs -revision: 3713eed996 +revision: 6eeaca032b --- # adapter::coord::sequencer::inner diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md index e4085a758f715..ee3bd489afd7b 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/cluster.rs -revision: 74f18a3354 +revision: 6eeaca032b --- # adapter::coord::sequencer::inner::cluster @@ -11,7 +11,7 @@ Replica IDs are pre-allocated out-of-band via `Catalog::allocate_replica_ids` be Cluster and replica creation and deletion are applied through the catalog implications pipeline rather than directly in this module. For `CREATE CLUSTER` (both managed and unmanaged variants), `scoped_overrides_create_op` is called with `ClusterEvalContext` and `ReplicaEvalContext` values built from plan data and pre-allocated replica IDs, and the resulting `Op::UpdateScopedSystemParameters` is folded into the same catalog transaction that creates the cluster. This makes the committed diff drive the replica-scoped controller push before `create_replica`, so the new cluster's optimizer and each new replica's render-frozen flags take effect before any dataflow renders, rather than falling back to environment-wide values until the next sync tick. `CREATE CLUSTER` validates the `HYDRATION SIZE` of an `AUTO SCALING STRATEGY` against the allowed replica sizes for the session role, just as `SIZE` itself is validated. This prevents a typo from failing silently at burst-arm time and prevents a size-restricted role from bursting at a size it may not `CREATE` with. -When `ENABLE_CLUSTER_CONTROLLER` is on and the cluster is user-owned, a shape-changing managed-to-managed `ALTER CLUSTER` (one that changes `SIZE`, `AVAILABILITY ZONES`, or `INTROSPECTION`) reshapes into a durable `reconfiguration` record via `reshape_alter_cluster_managed` instead of going through the legacy 3-stage machine. The record carries the full target config shape, a deadline, and an `on_timeout` action (`ROLLBACK` by default). Non-shape fields (`workload_class`, `schedule`, `auto_scaling_strategy`, etc.) are applied to the realized config immediately. The controller converges the replica set onto the target and cuts the realized config over at hydration. `ClusterStage::AwaitReconfiguration` (carrying `AlterClusterAwaitReconfiguration`) is a foreground wait-shim that polls the durable record until it reaches a terminal status, then reports success or failure depending on whether the realized config reached the target; it is used when `ENABLE_BACKGROUND_ALTER_CLUSTER` is off. +When `ENABLE_CLUSTER_CONTROLLER` is on and the cluster is user-owned, a shape-changing managed-to-managed `ALTER CLUSTER` (one that changes `SIZE`, `AVAILABILITY ZONES`, or `INTROSPECTION`) reshapes into a durable `reconfiguration` record via `reshape_alter_cluster_managed` instead of going through the legacy 3-stage machine. The record carries the full target config shape, a deadline, and an `on_timeout` action (`ROLLBACK` by default). Non-shape fields (`workload_class`, `schedule`, `auto_scaling_strategy`, etc.) are applied to the realized config immediately. The controller converges the replica set onto the target and cuts the realized config over at hydration. `ClusterStage::AwaitReconfiguration` (carrying `AlterClusterAwaitReconfiguration`) is a foreground wait-shim that polls the durable record until it reaches a terminal status, then reports success or failure depending on whether the realized config reached the target; it is used when `ENABLE_BACKGROUND_ALTER_CLUSTER` is off. The legacy foreground `AlterClusterWaitForHydrated` readiness check additionally requires every pending replica to be online (per `mz_cluster_replica_statuses`) before declaring the reconfiguration ready to cut over, so a cluster hosting only single-replica sources (which are never placed on a pending replica until cut-over) does not declare readiness vacuously the moment the pending replicas are created. When a reconfiguration is in flight, a subsequent shape-changing `ALTER` folds onto the in-flight target via `fold_reconfiguration_target`: dimensions the `ALTER` set explicitly take the new value, while dimensions left `Unchanged` keep the in-flight target's value (not the realized config's value, which would silently revert the in-flight transition). An `ALTER` back to the realized shape cancels the record. Changing the replication factor while a reconfiguration is in flight is refused with `AdapterError::AlterClusterReplicationFactorWhileReconfiguring`. `ALTER CLUSTER` validates a newly set or changed `AUTO SCALING STRATEGY`'s `HYDRATION SIZE` like `SIZE` itself: it must name a real replica size the session role may use. Only a changed strategy is checked, so an existing policy does not block unrelated `ALTER`s if the size allow-list later shrinks. Additionally, if a reconfiguration is in flight, `HYDRATION SIZE` must differ from the in-flight reconfiguration's target `SIZE`: equality would end the reshape with a no-op burst shape and a stored statement that fails its own re-plan. Unmanaged clusters reject `AUTO SCALING STRATEGY` changes with an error, mirroring how other managed-only options are handled. Converting a cluster to unmanaged while a hydration burst is in flight is refused with `AdapterError::AlterClusterUnmanagedWhileBursting`. The unmanaged variant carries no burst field, so converting would drop the record without a `Finished` audit event and strand the burst replica as an ordinary unmanaged replica nothing ever tears down. diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/create_index.md b/doc/developer/generated/adapter/coord/sequencer/inner/create_index.md index b3d4995ac090f..b20f3f0dbfa4b 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/create_index.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/create_index.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/create_index.rs -revision: 277b33e9c0 +revision: a60edac7f1 --- # adapter::coord::sequencer::inner::create_index diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/create_materialized_view.md b/doc/developer/generated/adapter/coord/sequencer/inner/create_materialized_view.md index 10d3624083720..c235755f4efc4 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/create_materialized_view.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/create_materialized_view.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/create_materialized_view.rs -revision: 277b33e9c0 +revision: a60edac7f1 --- # adapter::coord::sequencer::inner::create_materialized_view diff --git a/doc/developer/generated/adapter/coord/sql.md b/doc/developer/generated/adapter/coord/sql.md index 8337f6154c58a..17a8d5f78d43b 100644 --- a/doc/developer/generated/adapter/coord/sql.md +++ b/doc/developer/generated/adapter/coord/sql.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sql.rs -revision: 1c17d34993 +revision: a60edac7f1 --- # adapter::coord::sql diff --git a/doc/developer/generated/adapter/coord/statement_logging.md b/doc/developer/generated/adapter/coord/statement_logging.md index 8ec44d8840567..68c2ecd294ba5 100644 --- a/doc/developer/generated/adapter/coord/statement_logging.md +++ b/doc/developer/generated/adapter/coord/statement_logging.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/statement_logging.rs -revision: 770c31e9f8 +revision: a60edac7f1 --- # adapter::coord::statement_logging diff --git a/doc/developer/generated/adapter/coord/timeline.md b/doc/developer/generated/adapter/coord/timeline.md index 58bce3b9aaa0a..52d2b85a2426e 100644 --- a/doc/developer/generated/adapter/coord/timeline.md +++ b/doc/developer/generated/adapter/coord/timeline.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/timeline.rs -revision: 4002e6039c +revision: a60edac7f1 --- # adapter::coord::timeline diff --git a/doc/developer/generated/adapter/error.md b/doc/developer/generated/adapter/error.md index 229bd9d822511..9dcfd6998b814 100644 --- a/doc/developer/generated/adapter/error.md +++ b/doc/developer/generated/adapter/error.md @@ -1,12 +1,12 @@ --- source: src/adapter/src/error.rs -revision: 74f18a3354 +revision: 6eeaca032b --- # adapter::error Defines `AdapterError`, the central error type for the coordinator, with variants covering planning errors, catalog errors, storage/compute controller errors, RBAC violations, timestamp errors, authentication errors, and miscellaneous unstructured errors. -Notable variants include `ReadThenWriteDependencyLimitExceeded { max_rw_dependencies: usize }` (maps to `SqlState::PROGRAM_LIMIT_EXCEEDED`) when a read-then-write statement's selection transitively depends on more objects than the configured `max_rw_dependencies` limit; the error message includes the limit and a hint to raise `read_then_write_max_dependencies`. `ConcurrentDependencyDrop { dependency_kind, dependency_id }` for mid-flight dependency drops, `ConcurrentDependencyMutation { dependency_id }` (maps to `SqlState::T_R_SERIALIZATION_FAILURE`) when a dependency's `create_sql` changed between plan time and commit (detected by `PlanValidity`'s optional hash check), `AuthenticationError` (with an inner `AuthenticationError` enum covering `InvalidCredentials`, `NonLogin`, `RoleNotFound`, `PasswordRequired`), `ReplacementSchemaMismatch(RelationDescDiff)` for schema-incompatible replacements, `ReplaceMaterializedViewSealed`, `CollectionUnreadable`, `AlterClusterWhilePendingReplicas`, `AlterClusterUnmanagedWhileReconfiguring` (maps to `SqlState::OBJECT_IN_USE`) for attempts to convert a cluster to unmanaged while a graceful reconfiguration is in progress, `AlterClusterUnmanagedWhileBursting` (maps to `SqlState::OBJECT_IN_USE`) for attempts to convert a cluster to unmanaged while a hydration burst is in flight, `AlterClusterReplicationFactorWhileReconfiguring` (maps to `SqlState::OBJECT_IN_USE`) for attempts to change a cluster's replication factor while a graceful reconfiguration is in progress, `ImpossibleTimestampConstraints`, `OidcGroupSyncFailed(String)` for OIDC group-to-role sync failures in strict mode (maps to `SqlState::INTERNAL_ERROR`), `ClusterStateChanged { cluster_id }` (maps to `SqlState::T_R_SERIALIZATION_FAILURE`) produced by `Op::CheckClusterState` when a conditional cluster-config write fails its precondition, and four bounded-staleness variants: `BoundedStalenessExceeded { bound, gap_ms, slowest_input }` (maps to `SqlState::T_R_SERIALIZATION_FAILURE`), `BoundedStalenessReadOnly` (maps to `SqlState::READ_ONLY_SQL_TRANSACTION`), `BoundedStalenessRealTimeRecencyConflict` (maps to `SqlState::FEATURE_NOT_SUPPORTED`), and `BoundedStalenessTimelineUnsupported` (maps to `SqlState::FEATURE_NOT_SUPPORTED`). +Notable variants include `ReadThenWriteDependencyLimitExceeded { max_rw_dependencies: usize }` (maps to `SqlState::PROGRAM_LIMIT_EXCEEDED`) when a read-then-write statement's selection transitively depends on more objects than the configured `max_rw_dependencies` limit; the error message includes the limit and a hint to raise `read_then_write_max_dependencies`. `ConcurrentDependencyDrop { dependency_kind, dependency_id }` for mid-flight dependency drops, `ConcurrentDependencyMutation { dependency_id }` (maps to `SqlState::T_R_SERIALIZATION_FAILURE`) when a dependency's `create_sql` changed between plan time and commit (detected by `PlanValidity`'s optional hash check), `AuthenticationError` (with an inner `AuthenticationError` enum covering `InvalidCredentials`, `NonLogin`, `RoleNotFound`, `PasswordRequired`), `ReplacementSchemaMismatch(RelationDescDiff)` for schema-incompatible replacements, `ReplaceMaterializedViewSealed`, `CollectionUnreadable`, `AlterClusterWhilePendingReplicas`, `AlterClusterUnmanagedWhileReconfiguring` (maps to `SqlState::OBJECT_IN_USE`) for attempts to convert a cluster to unmanaged while a graceful reconfiguration is in progress, `AlterClusterUnmanagedWhileBursting` (maps to `SqlState::OBJECT_IN_USE`) for attempts to convert a cluster to unmanaged while a hydration burst is in flight, `AlterClusterReplicationFactorWhileReconfiguring` (maps to `SqlState::OBJECT_IN_USE`) for attempts to change a cluster's replication factor while a graceful reconfiguration is in progress, `AlterClusterScheduleWhileReconfiguring` (maps to `SqlState::OBJECT_IN_USE`) for attempts to change a cluster's schedule while a graceful reconfiguration is in progress, `AlterClusterWaitOnScheduledCluster` (maps to `SqlState::FEATURE_NOT_SUPPORTED`) for attempts to use `WAIT` on an `ALTER` of a scheduled (non-`MANUAL`) cluster whose replica set the controller reshapes in the background, `ImpossibleTimestampConstraints`, `OidcGroupSyncFailed(String)` for OIDC group-to-role sync failures in strict mode (maps to `SqlState::INTERNAL_ERROR`), `ClusterStateChanged { cluster_id }` (maps to `SqlState::T_R_SERIALIZATION_FAILURE`) produced by `Op::CheckClusterState` when a conditional cluster-config write fails its precondition, and four bounded-staleness variants: `BoundedStalenessExceeded { bound, gap_ms, slowest_input }` (maps to `SqlState::T_R_SERIALIZATION_FAILURE`), `BoundedStalenessReadOnly` (maps to `SqlState::READ_ONLY_SQL_TRANSACTION`), `BoundedStalenessRealTimeRecencyConflict` (maps to `SqlState::FEATURE_NOT_SUPPORTED`), and `BoundedStalenessTimelineUnsupported` (maps to `SqlState::FEATURE_NOT_SUPPORTED`). Implements `From` conversions from many downstream error types (`PlanError`, `StorageError`, `OptimizerError`, `VarError`, etc.) and maps each variant to a PostgreSQL `SqlState` error code via `code()`; error details and hints are provided through `detail()` and `hint()`. The `From` conversion maps `OptimizerError::RestrictedFunction` to `AdapterError::Unauthorized(UnauthorizedError::RestrictedSystemObject)` rather than the generic `Optimizer` variant; within `code()`, `OptimizerError::RestrictedFunction` maps to `SqlState::INSUFFICIENT_PRIVILEGE`. Several constructor helpers on `AdapterError` (e.g. `concurrent_dependency_drop_from_instance_missing`, `concurrent_dependency_drop_from_peek_error`) perform explicit error-kind conversions that are intentionally not automatic `From` impls. diff --git a/doc/developer/generated/adapter/metrics.md b/doc/developer/generated/adapter/metrics.md index ba5434a726c3c..f61b9ec8ce7bc 100644 --- a/doc/developer/generated/adapter/metrics.md +++ b/doc/developer/generated/adapter/metrics.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/metrics.rs -revision: dbd2c3fc06 +revision: a60edac7f1 --- # adapter::metrics @@ -10,5 +10,6 @@ Registers and vends all Prometheus metrics for the adapter and coordinator. `Metrics` includes a `timestamp_difference_for_bounded_staleness_ms` histogram (per-compute-instance label) that records how much older bounded-staleness timestamps are compared to serializable, measuring the actual staleness incurred. `SessionMetrics` exposes this via `timestamp_difference_for_bounded_staleness_ms`. The `mz_time_to_first_row_seconds` histogram carries an `application_name` label in addition to `instance_id`, `isolation_level`, and `strategy`. `Metrics` tracks catalog snapshot cache behavior via `catalog_snapshot_seconds` (a `HistogramVec` labeled by `context`, observed only on cache misses) and `catalog_snapshot_cache` (an `IntCounterVec` labeled by `context` and `result`, counting hits and misses). `catalog_arc_strong_count` and `catalog_arc_weak_count` are `UIntGauge` metrics tracking the number of strong and weak references to the current catalog snapshot `Arc`, respectively. +`Metrics` includes `catalog_transact_seconds` (a `HistogramVec` labeled by `method`) for timing catalog transact methods, `apply_catalog_implications_seconds` for timing catalog implication application, and `group_commit_catalog_upper_seconds` for timing catalog shard upper advances during group commits and table register/forget operations. Several public metrics carry `MetricTag` annotations for categorization: `mz_query_total`, `mz_active_sessions`, `mz_active_subscribes`, and `mz_adapter_commands` carry `MetricTag::Environment`. Helper functions `session_type_label_value`, `statement_type_label_value`, and `subscribe_output_label_value` produce the label strings used for partitioning these metrics. diff --git a/doc/developer/generated/adapter/optimize/_module.md b/doc/developer/generated/adapter/optimize/_module.md index 07fe83f29c978..182492f3e3d1c 100644 --- a/doc/developer/generated/adapter/optimize/_module.md +++ b/doc/developer/generated/adapter/optimize/_module.md @@ -1,12 +1,12 @@ --- source: src/adapter/src/optimize.rs -revision: 3d7eb1c1da +revision: 94054eb165 --- # adapter::optimize Defines the high-level optimizer interface used throughout the coordinator: the `Optimize` trait (one implementation per statement type and pipeline stage), `OptimizerConfig` (feature flags and execution mode), `OptimizerCatalog` (a minimal catalog view for the optimizer), and `OptimizerError` (aggregated error type for all optimization failures). -Each child module (`peek`, `index`, `materialized_view`, `subscribe`, `copy_to`, `view`) implements the full optimization pipeline for one statement type as a sequence of `Optimize` impls that transform opaque stage-result structs; `dataflows` provides the shared `DataflowBuilder` utility. +Each child module (`peek`, `index`, `materialized_view`, `subscribe`, `copy_to`, `view`) implements the full optimization pipeline for one statement type as a sequence of `Optimize` impls that transform opaque stage-result structs; `dataflows` provides the shared `DataflowBuilder` utility. A private `metric_sink` module handles the metric sink optimization path. `OptimizerError` includes a `RestrictedFunction(UnmaterializableFunc)` variant for unmaterializable functions blocked by the `restrict_to_user_objects` session variable; it carries a hint directing users to contact their administrator. `OptimizerConfig` can be overridden from `ExplainContext` (for `EXPLAIN ... WITH(...)`) or from `OptimizerFeatureOverrides` (for `CLUSTER ... FEATURES(...)`), keeping the optimizer API stable while allowing ad-hoc experimentation. `PeekOptimizer` is an enum that carries either a `peek::Optimizer` (for `SELECT` and `EXPLAIN`) or a `copy_to::Optimizer` (for `COPY TO`) through the shared peek sequencing state machine. Both statement types go through identical surrounding stages (timestamp selection, read holds, off-thread optimization), and `PeekOptimizer` lets that shared machinery carry either optimizer without caring which one it is. The companion `PeekGlobalLirPlan` enum tags the optimization result with the same two variants so that downstream stages can recover the concrete plan type. The internal `LirDataflowDescription` type alias resolves to `DataflowDescription`. The shared optimization pipeline steps (HIR-to-local-MIR, timestamp resolution, global-MIR-to-LIR) are factored into the `pub(crate) optimize_oneshot` helper, which is also used by the frontend peek path in `frontend_peek.rs`. diff --git a/doc/developer/generated/adapter/session.md b/doc/developer/generated/adapter/session.md index 5b25e809610c4..dd613841c15f0 100644 --- a/doc/developer/generated/adapter/session.md +++ b/doc/developer/generated/adapter/session.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/session.rs -revision: fd5141bb8c +revision: a60edac7f1 --- # adapter::session diff --git a/doc/developer/generated/adapter/util.md b/doc/developer/generated/adapter/util.md index 1ab224ca4a1e7..defe7db551055 100644 --- a/doc/developer/generated/adapter/util.md +++ b/doc/developer/generated/adapter/util.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/util.rs -revision: 4002e6039c +revision: a60edac7f1 --- # adapter::util diff --git a/doc/developer/generated/catalog-protos/_crate.md b/doc/developer/generated/catalog-protos/_crate.md index c774cceca0bf2..cf855b97b2361 100644 --- a/doc/developer/generated/catalog-protos/_crate.md +++ b/doc/developer/generated/catalog-protos/_crate.md @@ -1,12 +1,12 @@ --- source: src/catalog-protos/src/lib.rs -revision: 8598d82c1c +revision: fca741734d --- # mz-catalog-protos Provides all Rust types durably persisted in the Materialize catalog, along with `RustType` conversion impls bridging those types to protobuf. -The crate exposes the current schema as `objects` (currently v89) plus frozen snapshots `objects_v74` through `objects_v89` used for migrations. -`CATALOG_VERSION` (89) and `MIN_CATALOG_VERSION` (74) constants bound the supported migration range; the build script validates file hashes to prevent accidental mutation of snapshots. +The crate exposes the current schema as `objects` (currently v90) plus frozen snapshots `objects_v74` through `objects_v90` used for migrations. +`CATALOG_VERSION` (90) and `MIN_CATALOG_VERSION` (74) constants bound the supported migration range; the build script validates file hashes to prevent accidental mutation of snapshots. The crate has an optional `proptest` feature; `derive(Arbitrary)` on catalog types is compiled only when the `test` cfg or the `proptest` feature is enabled. Key dependencies are `mz-proto`, `mz-repr`, `mz-sql`, `mz-audit-log`, `mz-compute-types`, and `mz-storage-types`; the primary consumer is `mz-catalog`. diff --git a/doc/developer/generated/catalog-protos/objects.md b/doc/developer/generated/catalog-protos/objects.md index 89f5c9c83876e..c41de25cfb3b3 100644 --- a/doc/developer/generated/catalog-protos/objects.md +++ b/doc/developer/generated/catalog-protos/objects.md @@ -1,11 +1,11 @@ --- source: src/catalog-protos/src/objects.rs -revision: 6c07c975be +revision: fca741734d --- # mz-catalog-protos::objects -Defines the current (v89) set of Rust structs and enums that represent all durably persisted catalog objects, generated from protobuf definitions. +Defines the current (v90) set of Rust structs and enums that represent all durably persisted catalog objects, generated from protobuf definitions. This file is the canonical snapshot of the current catalog schema; `objects_v.rs` files are frozen snapshots used as migration sources. Key types include `ConfigKey`, `ConfigValue`, `SettingKey`, `SettingValue`, `IdAllocKey`, `RoleId`, `DatabaseId`, `SchemaId`, `AutoProvisionSource`, `RoleAttributes`, and many more covering every catalog entity. Scoped system configuration is represented by `ClusterSystemConfigurationKey`/`ClusterSystemConfigurationValue` (keyed by `ClusterId` and parameter name) and `ReplicaSystemConfigurationKey`/`ReplicaSystemConfigurationValue` (keyed by `ReplicaId` and parameter name), along with their corresponding `ClusterSystemConfiguration` and `ReplicaSystemConfiguration` wrapper structs and variants in `StateUpdateKind`. The `ManagedCluster` struct carries durable cluster autoscaling state: `auto_scaling_strategy` (`Option`), `reconfiguration` (`Option`), and `burst` (`Option`). `AutoScalingStrategy` holds an optional `OnHydration` sub-policy. `ReconfigurationState` records an in-flight graceful reconfiguration with a `ReconfigurationTarget`, deadline, and `OnTimeoutAction`. `BurstState` records an active hydration burst. The managed `ReplicaLocation` (`ManagedLocation`) stores `availability_zones` as a list rather than a single optional zone. The `audit_log_event_v1` submodule includes `CreateRoleV1`, `AlterAddColumnV1`, `AlterSourceTimestampIntervalV1`, `AlterClusterReconfigurationV1` (with `ReconfigurationLifecycleV1` variants `Started`, `Finalized`, `TimedOut`, `Cancelled`, `ResourceExhausted` and `ClusterReplicaLoggingV1`), and `ClusterHydrationBurstV1` (with `HydrationBurstLifecycleV1` and `BurstFinishCauseV1` variants `LingerElapsed`, `NoLongerWarranted`) for audit-logging corresponding catalog events. `CreateOrDropClusterReplicaReasonV1` carries reasons `Manual`, `Schedule`, `System`, `Reconfiguration`, `HydrationBurst`, and `Retired`. diff --git a/doc/developer/generated/catalog/builtin/mz_internal.md b/doc/developer/generated/catalog/builtin/mz_internal.md index b876ee9c17fb5..6b65c95b1151e 100644 --- a/doc/developer/generated/catalog/builtin/mz_internal.md +++ b/doc/developer/generated/catalog/builtin/mz_internal.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/builtin/mz_internal.rs -revision: 34813f0a3f +revision: 7f6c52776d --- # catalog::builtin::mz_internal @@ -15,7 +15,7 @@ This is the largest builtin submodule, exporting 199 public items: sources, tabl **Materialized views** (`BuiltinMaterializedView`) — Derived catalog views backed by queries over `mz_catalog_raw` and other sources: aggregated statistics, lag histograms, and other derived metrics. `MZ_POSTGRES_SOURCES` is a `BuiltinMaterializedView` (OID constant `MV_MZ_POSTGRES_SOURCES_OID`) that derives PostgreSQL source details (`replication_slot`, `timeline_id`) by parsing the raw catalog create SQL via `mz_internal.parse_postgres_source_details`; the MV filters to postgres sources first via `parse_catalog_create_sql` because the detail decoder must not be applied to non-postgres rows. `MZ_OVERRIDDEN_SYSTEM_PARAMETERS` projects the `system_configurations` durable collection out of `mz_catalog_raw`, exposing environment-wide system parameter overrides as `(name, value)` pairs with `PUBLIC_SELECT` access and an `Ontology` annotation; only parameters with an explicit override appear (defaults are absent). `MZ_CLUSTER_SYSTEM_PARAMETERS` projects the `cluster_system_configurations` durable collection out of `mz_catalog_raw`, exposing per-cluster system parameter overrides keyed by `(cluster_id, name)`. `MZ_REPLICA_SYSTEM_PARAMETERS` does the same for the `replica_system_configurations` collection, exposing per-replica overrides keyed by `(replica_id, name)`. `MZ_COMMENTS` is a `BuiltinMaterializedView` backed by a query over `mz_catalog_raw` that reads `Comment` entries, decodes each `proto::CommentObject` variant via CASE expressions, and projects `(id, object_type, object_sub_id, comment)` rows; new `CommentObject` variants must be added to both CASE expressions in its SQL. `MZ_CLUSTER_RECONFIGURATIONS` (OID `MV_MZ_CLUSTER_RECONFIGURATIONS_OID`) is a `BuiltinMaterializedView` with a unique key on `cluster_id` that exposes the latest graceful reconfiguration record per managed cluster, with columns `cluster_id`, `status`, `deadline`, `on_timeout`, `target` (JSONB), and `changes` (JSONB). Terminal records are retained after settling so the outcome stays inspectable until a later reconfiguration overwrites it. The `status` column maps serde variant names from `ReconfigurationStatus` to kebab-case (e.g. `InProgress` -> `in-progress`); unmapped variants pass through verbatim. `changes` diffs `target` against the realized config per dimension. `MZ_CLUSTER_AUTO_SCALING_STRATEGIES` (OID `MV_MZ_CLUSTER_AUTO_SCALING_STRATEGIES_OID`) is a `BuiltinMaterializedView` with a unique key on `cluster_id` that exposes the configured autoscaling strategy and in-flight state per managed cluster; `state` is keyed by strategy type so a future strategy's state is an additional key rather than a schema change. Both views have corresponding `BuiltinIndex` constants (`MZ_CLUSTER_RECONFIGURATIONS_IND`, `MZ_CLUSTER_AUTO_SCALING_STRATEGIES_IND`) on `cluster_id`. -**Views** (`BuiltinView`) — 82 entries covering the full `mz_internal` SQL surface: cluster introspection, compute operator metrics, peek durations, arrangement sizes, source/sink status, wall-clock lag, statement execution history, privilege management, dependency graph views, and more. `MZ_SOURCE_STATUSES` and `MZ_SINK_STATUSES` filter their `latest_per_replica_events` CTE to include events from still-existing replicas or the global sentinel, and additionally retain rows where `status = 'paused'` because a per-replica `paused` event is only written when the replica is dropped and acts as a terminal drop marker. Precedence ties in `latest_events` are broken by `occurred_at DESC` in addition to status precedence number, so a dropped replica's `paused` event wins over an older source-global `paused`. `MZ_SHOW_COLUMNS` applies an `object_type` predicate on its comment join to guard against stale comment rows from type changes. `MZ_SHOW_CLUSTERS` includes an `activity` column (nullable string) that summarizes any in-flight reconfiguration or autoscaling burst: it joins `mz_cluster_reconfigurations` (filtering to `status = 'in-progress'`) and `mz_cluster_auto_scaling_strategies` (checking for a non-null `state`) and produces a human-readable summary of the dimensions being changed and the burst size; `NULL` when the cluster is steady. Views carry inline SQL queries over `mz_catalog`, `mz_introspection`, and `mz_internal` tables. Access levels vary: most grant `PUBLIC_SELECT`; sensitive views use `MONITOR_SELECT`, `MONITOR_REDACTED_SELECT`, `SUPPORT_SELECT`, or `ANALYTICS_SELECT`. `MZ_MCP_DATA_PRODUCTS` and `MZ_MCP_DATA_PRODUCT_DETAILS` are MCP-agent-facing views listing data products and their column/key details that are exempted from `restrict_to_user_objects` blocking. `MZ_MCP_DATA_PRODUCT_DETAILS` includes a `hydration` column (JSONB) reporting readiness across the cluster's replicas, with fields `hydrated` (bool), `replica_count` (int), and `hydrated_replica_count` (int). `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW` (1-hour buckets, 14-day retention), `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW_24H` (5-minute buckets, 24-hour retention), and `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW_3H` (un-binned raw samples, 3-hour retention) are three cluster utilization views for the Console's time-range graphs; their SQL bodies are generated by shared helpers so their fingerprints stay in sync with the Console's equivalent ad-hoc query. +**Views** (`BuiltinView`) — 82 entries covering the full `mz_internal` SQL surface: cluster introspection, compute operator metrics, peek durations, arrangement sizes, source/sink status, wall-clock lag, statement execution history, privilege management, dependency graph views, and more. `MZ_SOURCE_STATUSES` and `MZ_SINK_STATUSES` filter their `latest_per_replica_events` CTE to include events from still-existing replicas or the global sentinel, and additionally retain rows where `status = 'paused'` because a per-replica `paused` event is only written when the replica is dropped and acts as a terminal drop marker. Precedence ties in `latest_events` are broken by `occurred_at DESC` in addition to status precedence number, so a dropped replica's `paused` event wins over an older source-global `paused`. `MZ_SHOW_COLUMNS` applies an `object_type` predicate on its comment join to guard against stale comment rows from type changes. `MZ_SHOW_CLUSTERS` includes an `activity` column (nullable string) that summarizes any in-flight reconfiguration or autoscaling burst: it joins `mz_cluster_reconfigurations` (filtering to `status = 'in-progress'`) and `mz_cluster_auto_scaling_strategies` (checking for a non-null `state`) and produces a human-readable summary of the dimensions being changed and the burst size; `NULL` when the cluster is steady. Views carry inline SQL queries over `mz_catalog`, `mz_introspection`, and `mz_internal` tables. Access levels vary: most grant `PUBLIC_SELECT`; sensitive views use `MONITOR_SELECT`, `MONITOR_REDACTED_SELECT`, `SUPPORT_SELECT`, or `ANALYTICS_SELECT`. `MZ_MCP_DATA_PRODUCTS` and `MZ_MCP_DATA_PRODUCT_DETAILS` are MCP-agent-facing views listing data products and their column/key details that are exempted from `restrict_to_user_objects` blocking. The `cluster` column in both views is null unless the session role has `USAGE` on the object's index/compute cluster (checked via `mz_show_my_cluster_privileges`); materialized views still appear regardless because they serve from persist, but plain indexed views without at least one usable index cluster are excluded. `MZ_MCP_DATA_PRODUCT_DETAILS` includes a `hydration` column (JSONB) reporting readiness across the cluster's replicas, with fields `hydrated` (bool), `replica_count` (int), and `hydrated_replica_count` (int); the hydration join uses the real cluster name even when the `cluster` column is null. `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW` (1-hour buckets, 14-day retention), `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW_24H` (5-minute buckets, 24-hour retention), and `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW_3H` (un-binned raw samples, 3-hour retention) are three cluster utilization views for the Console's time-range graphs; their SQL bodies are generated by shared helpers so their fingerprints stay in sync with the Console's equivalent ad-hoc query. `MZ_BUILTIN_SOURCES` is a view in `mz_internal` listing builtin and log sources that do not appear in `mz_catalog_raw`; user sources are exposed via `mz_catalog.mz_sources`. **Connections** (`BuiltinConnection`) — System-level connection definitions. diff --git a/doc/developer/generated/catalog/durable/_module.md b/doc/developer/generated/catalog/durable/_module.md index 0b1fcaa95f010..22efee414577b 100644 --- a/doc/developer/generated/catalog/durable/_module.md +++ b/doc/developer/generated/catalog/durable/_module.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable.rs -revision: 8598d82c1c +revision: a60edac7f1 --- # catalog::durable diff --git a/doc/developer/generated/catalog/durable/error.md b/doc/developer/generated/catalog/durable/error.md index 35ffdd431b882..918b87fe8df28 100644 --- a/doc/developer/generated/catalog/durable/error.md +++ b/doc/developer/generated/catalog/durable/error.md @@ -1,11 +1,11 @@ --- source: src/catalog/src/durable/error.rs -revision: 00cc513fa5 +revision: a60edac7f1 --- # catalog::durable::error Defines the error hierarchy for durable catalog operations. `CatalogError` wraps either a `SqlCatalogError` (logical catalog errors) or a `DurableCatalogError` (storage-level errors). -`DurableCatalogError` covers fencing (`FenceError`), incompatible data versions (`IncompatibleDataVersion`, `IncompatiblePersistVersion`), uninitialized catalog, not-writable catalog, protobuf deserialization errors, duplicate key, uniqueness violations, storage errors, and internal errors. The method `can_recover_with_write_mode` reports whether the error can be recovered by reopening the catalog in writable mode. +`DurableCatalogError` covers fencing (`FenceError`), incompatible data versions (`IncompatibleDataVersion`, `IncompatiblePersistVersion`), uninitialized catalog, not-writable catalog, dry-run transaction commit attempts (`DryRunTransaction`), protobuf deserialization errors, duplicate key, uniqueness violations, storage errors, catalog out-of-sync (`CatalogOutOfSync`, reported when the durable catalog has advanced past what this process has applied), and internal errors. The method `can_recover_with_write_mode` reports whether the error can be recovered by reopening the catalog in writable mode. `FenceError` has three variants: `DeployGeneration` (fenced by a newer deployment generation), `Epoch` (fenced by a newer epoch), and `MigrationUpper` (fenced during 0dt builtin table migration). The enum is ordered from most to least informative. diff --git a/doc/developer/generated/catalog/durable/initialize.md b/doc/developer/generated/catalog/durable/initialize.md index b31009f9615c9..9eae6e45d457c 100644 --- a/doc/developer/generated/catalog/durable/initialize.md +++ b/doc/developer/generated/catalog/durable/initialize.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/initialize.rs -revision: 80f8711523 +revision: fca741734d --- # catalog::durable::initialize diff --git a/doc/developer/generated/catalog/durable/objects/_module.md b/doc/developer/generated/catalog/durable/objects/_module.md index 973b88aba6087..95a1881d1ff7e 100644 --- a/doc/developer/generated/catalog/durable/objects/_module.md +++ b/doc/developer/generated/catalog/durable/objects/_module.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/objects.rs -revision: 8598d82c1c +revision: fca741734d --- # catalog::durable::objects diff --git a/doc/developer/generated/catalog/durable/objects/serialization.md b/doc/developer/generated/catalog/durable/objects/serialization.md index 27093c4163ac3..68494502b8c76 100644 --- a/doc/developer/generated/catalog/durable/objects/serialization.md +++ b/doc/developer/generated/catalog/durable/objects/serialization.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/objects/serialization.rs -revision: 8598d82c1c +revision: fca741734d --- # catalog::durable::objects::serialization diff --git a/doc/developer/generated/catalog/durable/objects/state_update.md b/doc/developer/generated/catalog/durable/objects/state_update.md index 43f50ce421e2b..94244394f21ac 100644 --- a/doc/developer/generated/catalog/durable/objects/state_update.md +++ b/doc/developer/generated/catalog/durable/objects/state_update.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/objects/state_update.rs -revision: 584bb9030c +revision: a60edac7f1 --- # catalog::durable::objects::state_update diff --git a/doc/developer/generated/catalog/durable/persist.md b/doc/developer/generated/catalog/durable/persist.md index 27c2e74a537af..7568e98eb38dd 100644 --- a/doc/developer/generated/catalog/durable/persist.md +++ b/doc/developer/generated/catalog/durable/persist.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/persist.rs -revision: 584bb9030c +revision: a60edac7f1 --- # catalog::durable::persist diff --git a/doc/developer/generated/catalog/durable/transaction.md b/doc/developer/generated/catalog/durable/transaction.md index 27791c578e4d0..99814634473af 100644 --- a/doc/developer/generated/catalog/durable/transaction.md +++ b/doc/developer/generated/catalog/durable/transaction.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/transaction.rs -revision: 55e361b0a6 +revision: a60edac7f1 --- # catalog::durable::transaction diff --git a/doc/developer/generated/catalog/durable/upgrade/_module.md b/doc/developer/generated/catalog/durable/upgrade/_module.md index 5bb24ebf235d6..91b31eb2ec829 100644 --- a/doc/developer/generated/catalog/durable/upgrade/_module.md +++ b/doc/developer/generated/catalog/durable/upgrade/_module.md @@ -1,14 +1,14 @@ --- source: src/catalog/src/durable/upgrade.rs -revision: 8598d82c1c +revision: fca741734d --- # catalog::durable::upgrade Orchestrates catalog schema migrations by replaying version-specific upgrade functions over a sequence of protobuf snapshots. -`CATALOG_VERSION` is the current schema version (89); `run_upgrade` chains individual `v{N}_to_v{N+1}` functions (v74 through v89) until the stored data matches the current version. +`CATALOG_VERSION` is the current schema version (90); `run_upgrade` chains individual `v{N}_to_v{N+1}` functions (v74 through v90) until the stored data matches the current version. Each version-specific submodule operates entirely on frozen `objects_v{N}` types, ensuring that future code changes cannot retroactively break old migrations. -The `objects\!` macro generates per-version support code, handling both old protobuf-based snapshots (v74--v78) and newer serde-based snapshots (v79+). +The `objects!` macro generates per-version support code, handling both old protobuf-based snapshots (v74--v78) and newer serde-based snapshots (v79+). The `json_compatible` submodule provides helpers for reading old JSON-encoded state when proto types have changed. The v82→v83 migration is a byte-level repair pass that plugs directly into `run_upgrade` rather than going through `run_versioned_upgrade`, because it requires raw access to snapshot diffs. The v83→v84 migration similarly calls `v83_to_v84::upgrade` directly from `run_upgrade`. @@ -17,3 +17,4 @@ The v85→v86 migration runs through `run_versioned_upgrade` and is a no-op that The v86→v87 migration runs through `run_versioned_upgrade` and adds durable cluster autoscaling state: `ManagedCluster` gains `auto_scaling_strategy`, `reconfiguration`, and `burst` (all defaulted to `None`), and the managed `ReplicaLocation`'s single `availability_zone` user-pin becomes an `availability_zones` list recording the zones the replica was provisioned under. The v87→v88 migration runs through `run_versioned_upgrade` and is a no-op: new `CreateOrDropClusterReplicaReasonV1` reasons (`Reconfiguration`, `HydrationBurst`, `Retired`) and new `AlterClusterReconfigurationV1` / `ClusterHydrationBurstV1` event details are additive and confined to the append-only audit log; no existing record changes shape, so a v87-serialized record is already valid v88. The v88→v89 migration runs through `run_versioned_upgrade` and adds `ReconfigurationState::status`, backfilling any in-flight reconfiguration records as `InProgress`. All other v88→v89 changes (new `ReconfigurationLifecycleV1::ResourceExhausted` variant, `ClusterReplicaLoggingV1`, `BurstFinishCauseV1`, updated `AlterClusterReconfigurationV1` and `ClusterHydrationBurstV1` audit event fields) are additive and confined to the append-only audit log; no existing record changes shape beyond the reconfiguration status backfill. +The v89→v90 migration runs through `run_versioned_upgrade` and adds the `arrangement_compression` flag to managed clusters and cluster replicas, backfilling it as `false`. Managed `Cluster` and `ClusterReplica` records are rewritten to include the new field; unmanaged clusters and all other records pass through unchanged. The flag also appears on a managed cluster's in-flight `reconfiguration` target, backfilled the same way. diff --git a/doc/developer/generated/catalog/memory/objects.md b/doc/developer/generated/catalog/memory/objects.md index 98a7483cb08d6..0465b9d3840bc 100644 --- a/doc/developer/generated/catalog/memory/objects.md +++ b/doc/developer/generated/catalog/memory/objects.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/memory/objects.rs -revision: 74f18a3354 +revision: fca741734d --- # catalog::memory::objects diff --git a/doc/developer/generated/cluster-controller/_crate.md b/doc/developer/generated/cluster-controller/_crate.md index 0e609c9e5a21d..aa1a37271cc6f 100644 --- a/doc/developer/generated/cluster-controller/_crate.md +++ b/doc/developer/generated/cluster-controller/_crate.md @@ -1,6 +1,6 @@ --- source: src/cluster-controller/src/lib.rs -revision: 74f18a3354 +revision: 6eeaca032b --- # mz-cluster-controller @@ -11,15 +11,15 @@ The crate depends only on primitive id/shape types and the `ClusterControllerCtx Key modules: -* `ctx` — The `ClusterControllerCtx` trait (the pull/apply boundary), `ClusterState`, `StateWrite`, `Decision`, `ApplyOutcome`, `ObservedReplica`, `ReconfigurationWrite`, `BurstWrite`, and the compare-and-append witness types re-exported from `mz-adapter-types`. -* `strategy` — The `Strategy` trait (three pure functions per tick plus `signal_request`), `DesiredReplica`, `SignalRequest`, `ConfigSignals`, `LiveSignals`, `BaselineStrategy` (the always-present implicit strategy that holds the steady-state replica set), `GracefulReconfigurationStrategy` (engaged while a reconfiguration record is in progress), and `HydrationBurstStrategy` (engaged for clusters with an `ON HYDRATION` auto-scaling policy). +* `ctx` — The `ClusterControllerCtx` trait (the pull/apply boundary), `ClusterState`, `StateWrite`, `Decision`, `ApplyOutcome`, `ObservedReplica`, `CreateReason`, `RefreshMvInfo`, `RefreshWindowInputs`, `RefreshWindowDecision`, `ReconfigurationWrite`, `BurstWrite`, and the compare-and-append witness types re-exported from `mz-adapter-types`. +* `strategy` — The `Strategy` trait (three pure functions per tick plus `signal_request`), `DesiredReplica`, `SignalRequest`, `ConfigSignals`, `LiveSignals`, `BaselineStrategy` (the always-present implicit strategy that holds the steady-state replica set for MANUAL clusters), `GracefulReconfigurationStrategy` (engaged while a reconfiguration record is in progress), `OnRefreshStrategy` (engaged for clusters with a non-MANUAL `ClusterSchedule`), and `HydrationBurstStrategy` (engaged for clusters with an `ON HYDRATION` auto-scaling policy). Key types defined in `lib.rs`: -* `ClusterController` — holds the strategy set (baseline, graceful-reconfiguration, hydration-burst) and a `ConfigSet` handle for dyncfgs; drives `reconcile` ticks. +* `ClusterController` — holds the strategy set (baseline, graceful-reconfiguration, on-refresh, hydration-burst) and a `ConfigSet` handle for dyncfgs; drives `reconcile` ticks. * `reconcile` — two-phase tick: phase 1 (`update_state`) fetches live signals, merges and applies durable writes per cluster; phase 2 (`desired_replicas`) diffs the unified desired set against the actual replicas and emits creates/drops. A `ResourceExhausted` outcome from phase 2 triggers `shed_decision`. * `merge_state_writes` — disjoint-union join of per-strategy `StateWrite`s under a conflict alarm via `soft_panic_or_log!`. -* `reconcile_replicas` — pure multiset union/diff kernel that closes the replica gap; only controller-owned replicas (those passing `ObservedReplica::owned_shape`) participate in the diff. +* `reconcile_replicas` — pure multiset diff kernel that closes the replica gap. The desired count per shape is the `max` over strategies (not the sum), so a replica satisfies every strategy that wants one of its shape. Only controller-owned replicas (those passing `ObservedReplica::owned_shape`) participate in the diff. * `ReplicaNameGen` — generates deterministic fresh replica names past the highest observed `rNN` index. * `shed_decision` — emits an `UpdateClusterState` that marks the graceful reconfiguration as `ResourceExhausted` when a phase-2 apply is rejected for exceeding the resource budget. * `config_signals` — latches `ConfigSignals` from the controller's `ConfigSet` once per tick, so every strategy evaluates against a consistent environment-wide config. diff --git a/doc/developer/generated/cluster-controller/ctx.md b/doc/developer/generated/cluster-controller/ctx.md index 6c7f1958cd1d9..7149454be44f6 100644 --- a/doc/developer/generated/cluster-controller/ctx.md +++ b/doc/developer/generated/cluster-controller/ctx.md @@ -1,6 +1,6 @@ --- source: src/cluster-controller/src/ctx.rs -revision: 74f18a3354 +revision: 6eeaca032b --- # cluster-controller::ctx @@ -11,13 +11,17 @@ The boundary between the controller and its environment. Key types: -* `ClusterControllerCtx` (trait) — six async methods: `now`, `managed_cluster_ids`, `cluster_states`, `hydrated_replicas`, `has_hydratable_objects`, `apply`. Reads are batched to bound round-trips in a separate-task deployment. `hydrated_replicas` returns the subset of given replicas that have all current collections hydrated; it is only called when a strategy declares it needs hydration via `SignalRequest`. `has_hydratable_objects` returns whether a cluster has at least one dataflow-backed object (index, materialized view, ingestion source, or sink); it is only called when a strategy declares `hydratable_objects` in its `SignalRequest`. -* `ClusterState` — durable config plus observed replicas of one managed cluster for one tick. Carries `cluster_id`, `size`, `replication_factor`, `availability_zones`, `logging`, optional `auto_scaling_policy`, `reconfiguration` and `burst` records, and `replicas`. Unmanaged clusters are not represented. +* `ClusterControllerCtx` (trait) — seven async methods: `now`, `managed_cluster_ids`, `cluster_states`, `hydrated_replicas`, `has_hydratable_objects`, `refresh_window_inputs`, `apply`. Reads are batched to bound round-trips in a separate-task deployment. `hydrated_replicas` returns the subset of given replicas that are online and have all current collections hydrated; it is only called when a strategy declares it needs hydration via `SignalRequest`. `has_hydratable_objects` returns whether a cluster has at least one dataflow-backed object (index, materialized view, ingestion source, or sink); it is only called when a strategy declares `hydratable_objects` in its `SignalRequest`. `refresh_window_inputs` pulls the read timestamp, compaction estimate, and bound REFRESH MV frontiers/schedules for a scheduled cluster; called only for clusters the on-refresh strategy needs to evaluate, returning `None` if the cluster has moved off a scheduled schedule mid-tick. +* `ClusterState` — durable config plus observed replicas of one managed cluster for one tick. Carries `cluster_id`, `size`, `replication_factor`, `availability_zones`, `logging`, `arrangement_compression`, `schedule` (the `ClusterSchedule` driving whether the baseline or on-refresh strategy owns the replica set), optional `auto_scaling_policy`, `reconfiguration` and `burst` records, and `replicas`. Unmanaged clusters are not represented. * `ObservedReplica` — a replica that actually exists: `replica_id`, `name`, `shape` (optional; `None` for unmanaged-location replicas), `internal`, `billed_as`, `pending`. `ObservedReplica::owned_shape` returns the replica's shape only when the controller owns it: `INTERNAL`, `BILLED AS`, and pending replicas return `None` and are excluded from the desired/actual diff, though their names still block the name generator. -* `StateWrite` — the durable mutations a strategy's `update_state` requests: cut-overs (`new_size`, `new_replication_factor`, `new_availability_zones`, `new_logging`) and record writes/clears (`reconfiguration`, `burst`). `None` fields are no-ops. +* `RefreshMvInfo` — one REFRESH materialized view bound to a scheduled cluster. Carries the MV's `GlobalId`, its storage write frontier (`write_frontier: Antichain`), and its `refresh_schedule`. Used by the on-refresh strategy to determine whether a refresh is due and whether compaction time is still needed. +* `RefreshWindowInputs` — the live signals the on-refresh strategy reads to decide whether a scheduled cluster is inside a refresh window: `read_ts` (the local oracle read timestamp), `compaction_estimate` (how long after a refresh an MV is estimated to still need Persist compaction), and `refresh_mvs` (the bound REFRESH MVs). Pulled on demand only for scheduled clusters. +* `RefreshWindowDecision` — the on-refresh strategy's per-tick window decision. Lists which bound REFRESH MVs keep the cluster on (`objects_needing_refresh`, `objects_needing_compaction`) and the `hydration_time_estimate` the window was widened by. The window is open iff either list is non-empty. Carried inside `CreateReason::OnRefresh` on the create decisions an open window produces. +* `CreateReason` — why a strategy desires a replica slot: the audit attribution a create decision carries. Variants: `Baseline` (audited as manual create), `GracefulReconfiguration`, `HydrationBurst`, `OnRefresh(RefreshWindowDecision)` (embeds the window decision so the audit detail appears iff this reason wins). When several strategies desire the same shape, `CreateReason::outranks` picks the winner: `GracefulReconfiguration` > `HydrationBurst` > `OnRefresh` > `Baseline`. +* `StateWrite` — the durable mutations a strategy's `update_state` requests: cut-overs (`new_size`, `new_replication_factor`, `new_availability_zones`, `new_logging`, `new_arrangement_compression`) and record writes/clears (`reconfiguration`, `burst`). `None` fields are no-ops. * `ReconfigurationWrite` — a write to the `reconfiguration` record bundled with the `ReconfigurationAudit` lifecycle intent, so a writer cannot move the record without simultaneously declaring what the audit trail should say. * `BurstWrite` — analogous bundle for the `burst` record. -* `Decision` — a single command the controller emits: `CreateReplica`, `DropReplica`, or `UpdateClusterState`. Every variant carries an `ExpectedClusterState` for compare-and-append; the apply path rejects the whole batch if any target cluster's state has since diverged. +* `Decision` — a single command the controller emits: `CreateReplica`, `DropReplica`, or `UpdateClusterState`. Every variant carries an `ExpectedClusterState` for compare-and-append; the apply path rejects the whole batch if any target cluster's state has since diverged. `CreateReplica` also carries a `CreateReason` for audit attribution. * `ApplyOutcome` — `Applied`, `Rejected` (at least one compare-and-append guard failed), or `ResourceExhausted` (the batch exceeded the environment's resource budget; nothing was transacted). -The compare-and-append witness types (`ExpectedClusterState`, `ReplicaShape`, `AvailabilityZones`, `AutoScalingPolicy`, `OnHydrationPolicy`, `BurstRecord`, `BurstAudit`, `BurstFinishCause`, `ReconfigurationRecord`, `ReconfigurationAudit`, `ReconfigurationStatus`, `ReconfigurationTarget`, `OnTimeout`) are re-exported from `mz-adapter-types::cluster_state` so the catalog transaction that applies a decision can share them without depending on this crate. +The compare-and-append witness types (`ExpectedClusterState`, `ReplicaShape`, `AvailabilityZones`, `AutoScalingPolicy`, `ClusterSchedule`, `OnHydrationPolicy`, `BurstRecord`, `BurstAudit`, `BurstFinishCause`, `ReconfigurationRecord`, `ReconfigurationAudit`, `ReconfigurationStatus`, `ReconfigurationTarget`, `OnTimeout`) are re-exported from `mz-adapter-types::cluster_state` so the catalog transaction that applies a decision can share them without depending on this crate. diff --git a/doc/developer/generated/cluster-controller/strategy.md b/doc/developer/generated/cluster-controller/strategy.md index 19c89ab2b9483..32f9cc6360949 100644 --- a/doc/developer/generated/cluster-controller/strategy.md +++ b/doc/developer/generated/cluster-controller/strategy.md @@ -1,6 +1,6 @@ --- source: src/cluster-controller/src/strategy.rs -revision: 74f18a3354 +revision: 6eeaca032b --- # cluster-controller::strategy @@ -16,14 +16,12 @@ Both are pure: same inputs, same output, no I/O. Strategies never touch `Cluster Key types: -* `Strategy` (trait) — `name() -> &'static str`, `signal_request` (default: no signals), `update_state` (default: no write), `desired_replicas`. All methods receive both `&LiveSignals` and `&ConfigSignals`. `Send + Sync` so the controller can hold boxed strategies on its own task. -* `DesiredReplica` — a replica slot a strategy desires this tick, characterized by a `ReplicaShape`. -* `SignalRequest` — declares which live signals a strategy needs for a cluster this tick: `hydration: bool` (probe per-replica hydration) and `hydratable_objects: bool` (check whether the cluster has at least one dataflow-backed object). The controller unions requests across strategies and fetches only what is needed. +* `Strategy` (trait) — `signal_request` (default: no signals), `update_state` (default: no write), `desired_replicas`. All methods receive both `&LiveSignals` and `&ConfigSignals`. `Send + Sync` so the controller can hold boxed strategies on its own task. +* `DesiredReplica` — a replica slot a strategy desires this tick, characterized by a `ReplicaShape` and a `CreateReason` (the audit attribution this slot contributes; the highest-precedence reason among all slots for a shape wins). +* `SignalRequest` — declares which live signals a strategy needs for a cluster this tick: `hydration: bool` (probe per-replica hydration), `hydratable_objects: bool` (check whether the cluster has at least one dataflow-backed object), and `refresh_window: bool` (pull bound REFRESH MV frontiers, schedules, and the current read timestamp). The controller unions requests across strategies and fetches only what is needed. * `ConfigSignals` — environment-wide dyncfg values latched by the kernel once per tick: `burst_enabled` (the break-glass flag for the hydration-burst strategy) and `default_burst_linger` (the system-default linger duration written into new burst records when the policy omits one). Not durable state, so never witness material. -* `LiveSignals` — the fulfilled live signals for one cluster: `hydrated_replicas` is the set of replica IDs (among controller-owned replicas) that have all current collections hydrated; `has_hydratable_objects` is whether the cluster has at least one dataflow-backed object. A signal not requested by any strategy is left at its empty default. -* `BaselineStrategy` — the always-present implicit strategy. Desires `replication_factor` replicas at the cluster's realized shape. With only the baseline engaged the desired set equals the realized set, so a steady-state managed cluster reconciles to no decisions. The baseline is what makes other strategies purely additive. -* `BASELINE_STRATEGY_NAME` — the audit-attribution string `"baseline"`. +* `LiveSignals` — the fulfilled live signals for one cluster: `hydrated_replicas` is the set of replica IDs (among controller-owned replicas) that are online and have all current collections hydrated; `has_hydratable_objects` is whether the cluster has at least one dataflow-backed object; `refresh_window` is the `RefreshWindowInputs` for a scheduled cluster (`None` when not requested or when the cluster was no longer scheduled at pull time). A signal not requested by any strategy is left at its empty default. +* `BaselineStrategy` — the always-present implicit strategy. Desires `replication_factor` replicas at the cluster's realized shape, but only for MANUAL clusters. On a scheduled cluster the on-refresh strategy owns the replica set and the baseline contributes nothing. With only the baseline engaged on a MANUAL cluster the desired set equals the realized set, so a steady-state managed cluster reconciles to no decisions. * `GracefulReconfigurationStrategy` — engaged whenever the durable `reconfiguration` record is in progress. Desires `target.replication_factor` replicas at the target shape in addition to the baseline's realized-shape replicas. Once enough target replicas are hydrated, `update_state` cuts over (advances the realized config, marks the record finalized). On a timeout, behavior is governed by the record's `on_timeout`: `Commit` cuts over to the un-hydrated target, `Rollback` (the default) marks the record timed out and stops desiring the target replicas. -* `GRACEFUL_RECONFIGURATION_STRATEGY_NAME` — the audit-attribution string `"graceful-reconfiguration"`. +* `OnRefreshStrategy` — engaged for clusters with a non-MANUAL `ClusterSchedule`. Contributes one replica at the cluster's realized shape while the cluster is inside a refresh window (determined from `RefreshWindowInputs`), and nothing otherwise. Also normalizes the realized `replication_factor` to `0` via `update_state` so the baseline does not desire replicas on a scheduled cluster. * `HydrationBurstStrategy` — engaged for clusters whose `AUTO SCALING STRATEGY` includes an `ON HYDRATION` policy, provided the global break-glass dyncfg is on and the cluster is On (`replication_factor > 0`). While at least one hydratable object exists that no steady-state (realized-config) replica has hydrated, it arms a durable `burst` record and desires one extra replica at the policy's `HYDRATION SIZE`. Once the steady set hydrates, the burst replica lingers for `linger_duration` then the record is cleared and the replica dropped. A burst record no longer consistent with the current policy (size changed, policy removed, or cluster turned off) is torn down immediately via `update_state`. There is no TTL: if the steady set never hydrates, the burst replica runs indefinitely. Burst coexists with a graceful reconfiguration without suppression. -* `HYDRATION_BURST_STRATEGY_NAME` — the audit-attribution string `"hydration-burst"`. diff --git a/doc/developer/generated/cluster/client.md b/doc/developer/generated/cluster/client.md index 179e113f17509..6bfa75b728d18 100644 --- a/doc/developer/generated/cluster/client.md +++ b/doc/developer/generated/cluster/client.md @@ -1,6 +1,6 @@ --- source: src/cluster/src/client.rs -revision: 225aeaa79f +revision: 445c5ce0f4 --- # client @@ -8,4 +8,4 @@ revision: 225aeaa79f Manages the lifecycle of a local Timely cluster and the client connection to it. `ClusterClient` wraps a `Partitioned` that fans commands out across all Timely worker threads; the first message received must be a protocol nonce that triggers `connect`, which wires up per-worker channels. `ClusterSpec` is the trait compute and storage implement: they supply `Command`/`Response` types, a cluster name, and a `run_worker` function that drives each Timely worker. -`build_cluster` initializes Timely networking (optionally with lgalloc-backed zero-copy buffers) and launches worker threads, returning a `TimelyContainer` that keeps them alive. +`build_cluster` is a provided method on `ClusterSpec` that initializes Timely networking (optionally with lgalloc-backed zero-copy buffers), launches worker threads with disambiguated OS thread names, and returns a `TimelyContainer` that keeps them alive. diff --git a/doc/developer/generated/clusterd-test-driver/dataflow.md b/doc/developer/generated/clusterd-test-driver/dataflow.md index a7a3cf6882df2..553332fc20f5a 100644 --- a/doc/developer/generated/clusterd-test-driver/dataflow.md +++ b/doc/developer/generated/clusterd-test-driver/dataflow.md @@ -1,16 +1,24 @@ --- source: src/clusterd-test-driver/src/dataflow.rs -revision: 43a933b189 +revision: 94054eb165 --- # mz-clusterd-test-driver::dataflow Assembly of compute `DataflowDescription`s for the headless test driver. -`DataflowBuilder` is the generic boundary between tests and dataflow assembly. A test describes its dataflow in terms of persist imports, MIR objects, and index exports; the builder handles MIR-to-LIR lowering via `LirRelationExpr::finalize_dataflow`, `RenderPlan` conversion, `CollectionMetadata` attachment, and `SqlRelationType`/`ReprRelationType` bookkeeping, producing a `DataflowDescription` ready to ship as `ComputeCommand::CreateDataflow`. +`DataflowBuilder` is the generic boundary between tests and dataflow assembly. A test describes its dataflow in terms of persist imports, index imports, MIR objects, and index/sink exports; the builder handles MIR-to-LIR lowering via `LirRelationExpr::finalize_dataflow`, `RenderPlan` conversion, `CollectionMetadata` attachment, and `SqlRelationType`/`ReprRelationType` bookkeeping, producing a `DataflowDescription` ready to ship as `ComputeCommand::CreateDataflow`. + +By default the builder lowers the caller's MIR faithfully without optimization. The `optimize` method enables the MIR dataflow optimizer (`mz_transform::optimize_dataflow`) before lowering, which is required for plans containing a `Join` whose `implementation` is `Unimplemented`. When optimizing, the builder supplies an `ImportedIndexOracle` built from the dataflow's own `index_imports` so the optimizer recognizes imported arrangements. `index_dataflow` is sugar over `DataflowBuilder` for the common single-index shape. +`count_over_index` constructs a count-aggregate MIR over an index import, for simple correctness checks. + `PersistSource` and `PersistSink` are helper structs describing a persist-backed source import and a sink export respectively. -`count_over_index` constructs a count-aggregate MIR over an index import, for simple correctness checks. +`Input` is a handle to an imported or built collection, carrying its `GlobalId` and `ReprRelationType`, used to construct typed `Get` nodes without hand-rolling the type. + +`ImportedIndexOracle` is a private `IndexOracle` implementation built from a dataflow's `index_imports`, exposing only the arrangements the dataflow itself imports. It is passed to the MIR optimizer when `optimize` is enabled. + +The private `augment` function converts a lowered `DataflowDescription` into `DataflowDescription` by flattening each object's plan via `RenderPlan::try_from` and splicing `CollectionMetadata` into source and materialized-view sink entries, mirroring what `compute-client`'s `Instance::create_dataflow` does. diff --git a/doc/developer/generated/compute-client/controller/_module.md b/doc/developer/generated/compute-client/controller/_module.md index bce3dd538ad16..08494aae3962e 100644 --- a/doc/developer/generated/compute-client/controller/_module.md +++ b/doc/developer/generated/compute-client/controller/_module.md @@ -1,6 +1,6 @@ --- source: src/compute-client/src/controller.rs -revision: e926ec3a86 +revision: fca741734d --- # mz-compute-client::controller diff --git a/doc/developer/generated/compute-client/controller/instance.md b/doc/developer/generated/compute-client/controller/instance.md index 51fe6cb9f9415..99d65d611a261 100644 --- a/doc/developer/generated/compute-client/controller/instance.md +++ b/doc/developer/generated/compute-client/controller/instance.md @@ -1,6 +1,6 @@ --- source: src/compute-client/src/controller/instance.rs -revision: e926ec3a86 +revision: 94054eb165 --- # mz-compute-client::controller::instance diff --git a/doc/developer/generated/compute-types/config.md b/doc/developer/generated/compute-types/config.md index 9b709a58810eb..0b50ac80bbea9 100644 --- a/doc/developer/generated/compute-types/config.md +++ b/doc/developer/generated/compute-types/config.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/config.rs -revision: 4267863081 +revision: fca741734d --- # compute-types::config diff --git a/doc/developer/generated/compute-types/dyncfgs.md b/doc/developer/generated/compute-types/dyncfgs.md index 107598f1486bf..7ccca474ee80b 100644 --- a/doc/developer/generated/compute-types/dyncfgs.md +++ b/doc/developer/generated/compute-types/dyncfgs.md @@ -1,13 +1,13 @@ --- source: src/compute-types/src/dyncfgs.rs -revision: 0d67fec095 +revision: 93dcb0ef5a --- # compute-types::dyncfgs Declares all dynamic configuration (`dyncfg`) constants for the compute layer. Key groups include: -- **Rendering**: `ENABLE_HALF_JOIN2`, `ENABLE_MZ_JOIN_CORE`, `LINEAR_JOIN_YIELDING`, `ENABLE_COMPUTE_TEMPORAL_BUCKETING`/`TEMPORAL_BUCKETING_SUMMARY`, `COMPUTE_FLAT_MAP_FUEL`, `ENABLE_COMPUTE_RENDER_FUELED_AS_SPECIFIC_COLLECTION`, `COMPUTE_APPLY_COLUMN_DEMANDS`, `ENABLE_COLUMN_PAGED_BATCHER` (replica-scoped), `ENABLE_COLUMN_PAGED_BATCHER_SPILL` (replica-scoped), `COLUMN_PAGED_BATCHER_BUDGET_FRACTION`, `COLUMN_PAGED_BATCHER_LZ4` (replica-scoped), `COLUMN_PAGED_BATCHER_SWAP_PAGEOUT`. +- **Rendering**: `ENABLE_HALF_JOIN2`, `ENABLE_MZ_JOIN_CORE`, `LINEAR_JOIN_YIELDING`, `ENABLE_COMPUTE_TEMPORAL_BUCKETING`/`TEMPORAL_BUCKETING_SUMMARY`, `COMPUTE_FLAT_MAP_FUEL`, `ENABLE_COMPUTE_RENDER_FUELED_AS_SPECIFIC_COLLECTION`, `COMPUTE_APPLY_COLUMN_DEMANDS`, `ENABLE_COLUMN_PAGED_BATCHER` (replica-scoped), `ENABLE_COLUMN_PAGED_BATCHER_SPILL` (replica-scoped), `COLUMN_PAGED_BATCHER_BUDGET_FRACTION` (replica-scoped), `COLUMN_PAGED_BATCHER_SPILL_WORKER_COUNT` (replica-scoped), `COLUMN_PAGED_BATCHER_LZ4` (replica-scoped), `COLUMN_PAGED_BATCHER_SWAP_PAGEOUT` (replica-scoped), `COLUMN_PAGED_BATCHER_EAGER_BACKING` (replica-scoped), `COLUMN_PAGED_BATCHER_POOL_RSS_TARGET_FRACTION` (replica-scoped). - **MV sink**: `ENABLE_SYNC_MV_SINK`, `ENABLE_CORRECTION_V2`, `CONSOLIDATING_VEC_GROWTH_DAMPENER`, `CORRECTION_V2_CHAIN_PROPORTIONALITY`, `CORRECTION_V2_CHUNK_SIZE`, `MV_SINK_ADVANCE_PERSIST_FRONTIERS`. - **Memory management**: `ENABLE_LGALLOC` (replica-scoped), `ENABLE_LGALLOC_EAGER_RECLAMATION`, `LGALLOC_BACKGROUND_INTERVAL`, `LGALLOC_FILE_GROWTH_DAMPENER`, `LGALLOC_LOCAL_BUFFER_BYTES`, `LGALLOC_SLOW_CLEAR_BYTES`, `ENABLE_COLUMNATION_LGALLOC`, `MEMORY_LIMITER_INTERVAL`, `MEMORY_LIMITER_USAGE_BIAS`, `MEMORY_LIMITER_BURST_FACTOR`. - **Backpressure**: `DATAFLOW_MAX_INFLIGHT_BYTES`, `DATAFLOW_MAX_INFLIGHT_BYTES_CC`, `ENABLE_COMPUTE_LOGICAL_BACKPRESSURE`, `COMPUTE_LOGICAL_BACKPRESSURE_MAX_RETAINED_CAPABILITIES`, `COMPUTE_LOGICAL_BACKPRESSURE_INFLIGHT_SLACK`. diff --git a/doc/developer/generated/compute-types/explain/text.md b/doc/developer/generated/compute-types/explain/text.md index ed5bc1f6be38d..8d4cb88a45a01 100644 --- a/doc/developer/generated/compute-types/explain/text.md +++ b/doc/developer/generated/compute-types/explain/text.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/explain/text.rs -revision: e926ec3a86 +revision: 58b5ff70c8 --- # compute-types::explain::text diff --git a/doc/developer/generated/compute-types/plan/_module.md b/doc/developer/generated/compute-types/plan/_module.md index 8dddab97ba1d5..0addbdd91e0e3 100644 --- a/doc/developer/generated/compute-types/plan/_module.md +++ b/doc/developer/generated/compute-types/plan/_module.md @@ -1,11 +1,11 @@ --- source: src/compute-types/src/plan.rs -revision: 92047d0776 +revision: a991ae1139 --- # compute-types::plan -Defines `LirRelationExpr` (the LIR plan type) and `LirRelationNode` (its node enum), along with `AvailableCollections` (which arrangements and raw forms a plan node produces), `GetPlan` (how a Get node reads a collection), `LirId` (a unique node identifier), and `LoweringMetrics` (Prometheus counters collected during MIR-to-LIR lowering). Fields that previously held `MirScalarExpr` or `MapFilterProject` now hold `LirScalarExpr` and `MfpPlan` / `SafeMfpPlan` respectively; `AvailableCollections::arranged` stores `Vec<(Vec, Vec, Vec)>`. +Defines `LirRelationExpr` (the LIR plan type) and `LirRelationNode` (its node enum), along with `AvailableCollections` (which arrangements and raw forms a plan node produces), `GetPlan` (how a Get node reads a collection), `ArrangementStrategy` (whether an `ArrangeBy` forms arrangements directly or via temporal bucketing), `LirId` (a unique node identifier), and `LoweringMetrics` (Prometheus counters collected during MIR-to-LIR lowering). Scalar expressions in plan nodes use `LirScalarExpr` and `MfpPlan` / `SafeMfpPlan`; `AvailableCollections::arranged` stores `Vec<(Vec, Vec, Vec)>`. `LoweringMetrics` is registered into a `MetricsRegistry` via `LoweringMetrics::register_into`. It exposes `inc_literal_constraints(case: &str)` to record successful `MapFilterProject::literal_constraints` calls by call site. The metric name is `mz_optimizer_lowering_literal_constraints_total`. `LirRelationExpr::finalize_dataflow` is the entry point for the full MIR→LIR pipeline: it lowers the dataflow (selecting monotonic operator variants for single-time dataflows during lowering itself), refines source MFPs, and for single-time dataflows relaxes `must_consolidate` flags via `RelaxMustConsolidate`. It accepts an `Option<&LoweringMetrics>` that is forwarded to the lowering `Context`. `LirRelationExpr::pretty` and `LirRelationExpr::debug_explain` provide text rendering for debugging and tests. diff --git a/doc/developer/generated/compute-types/plan/join/delta_join.md b/doc/developer/generated/compute-types/plan/join/delta_join.md index bba1e6dbf3f5c..9a37c6d933d24 100644 --- a/doc/developer/generated/compute-types/plan/join/delta_join.md +++ b/doc/developer/generated/compute-types/plan/join/delta_join.md @@ -1,10 +1,10 @@ --- source: src/compute-types/src/plan/join/delta_join.rs -revision: e926ec3a86 +revision: 58b5ff70c8 --- # compute-types::plan::join::delta_join Plans delta joins: a join over multiple inputs implemented as one independent dataflow path per input relation. -`DeltaJoinPlan` contains a `DeltaPathPlan` for each source relation; each path specifies a `source_key: Vec` and a sequence of `DeltaStagePlan` lookup stages, each carrying `stream_key` and `lookup_key` as `Vec`. +`DeltaJoinPlan` contains a `DeltaPathPlan` for each source relation; each path specifies a `source_key: Option>` (the source arrangement key, or `None` when the source is consumed as a raw unarranged collection in single-time dataflows) and a sequence of `DeltaStagePlan` lookup stages, each carrying `stream_key` and `lookup_key` as `Vec`. Delta joins re-use existing arrangements and create no new stateful operators, making them efficient when all input arrangements already exist. diff --git a/doc/developer/generated/compute-types/plan/lowering.md b/doc/developer/generated/compute-types/plan/lowering.md index bc52539d4bc0e..1205e52203b29 100644 --- a/doc/developer/generated/compute-types/plan/lowering.md +++ b/doc/developer/generated/compute-types/plan/lowering.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/plan/lowering.rs -revision: f74a121770 +revision: 380c8dd1a1 --- # compute-types::plan::lowering diff --git a/doc/developer/generated/compute-types/plan/scalar.md b/doc/developer/generated/compute-types/plan/scalar.md index 4e74ca1e36733..6017ef4188f46 100644 --- a/doc/developer/generated/compute-types/plan/scalar.md +++ b/doc/developer/generated/compute-types/plan/scalar.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/plan/scalar.rs -revision: 822256a77d +revision: 380c8dd1a1 --- # compute-types::plan::scalar diff --git a/doc/developer/generated/compute-types/sinks.md b/doc/developer/generated/compute-types/sinks.md index 5e55edf1f9328..e724ebd11c40d 100644 --- a/doc/developer/generated/compute-types/sinks.md +++ b/doc/developer/generated/compute-types/sinks.md @@ -1,11 +1,12 @@ --- source: src/compute-types/src/sinks.rs -revision: 9d0a7c3c6f +revision: 94054eb165 --- # compute-types::sinks -Defines `ComputeSinkDesc` and `ComputeSinkConnection`, the descriptor types for compute dataflow sinks. -The three connection variants are: `Subscribe` (streaming query output), `MaterializedView` (persist-backed MV), and `CopyToS3Oneshot` (one-shot COPY TO S3). +Defines `ComputeSinkDesc` and `ComputeSinkConnection`, the descriptor types for compute dataflow sinks. +The four connection variants are: `Subscribe` (streaming query output), `MaterializedView` (persist-backed MV), `CopyToS3Oneshot` (one-shot COPY TO S3), and `MetricSink` (writes rows into the in-process Prometheus metrics registry). `SubscribeSinkConnection` carries an `output` field (`Vec`) that specifies the ordering for rows emitted by the subscribe. `MaterializedViewSinkConnection` carries a `storage_metadata` field that is filled in by the storage/persist layer. +`MetricSinkConnection` carries no payload: the identity of the metric to update is the sink's `GlobalId`, and the sink does not write to persist. diff --git a/doc/developer/generated/compute/compute_state/_module.md b/doc/developer/generated/compute/compute_state/_module.md index 0022b44526d06..4aca550fa6139 100644 --- a/doc/developer/generated/compute/compute_state/_module.md +++ b/doc/developer/generated/compute/compute_state/_module.md @@ -1,6 +1,6 @@ --- source: src/compute/src/compute_state.rs -revision: 1c55de49eb +revision: 93dcb0ef5a --- # mz-compute::compute_state diff --git a/doc/developer/generated/compute/render/columnar.md b/doc/developer/generated/compute/render/columnar.md new file mode 100644 index 0000000000000..543950fd64d1d --- /dev/null +++ b/doc/developer/generated/compute/render/columnar.md @@ -0,0 +1,18 @@ +--- +source: src/compute/src/render/columnar.rs +revision: d43cd78803 +--- + +# mz-compute::render::columnar + +Columnar dataflow edge support. + +Defines `CollectionEdge`, a wrapper that lets dataflow edges between Plan nodes carry either row-based (`VecCollection`) or columnar (`ColumnarCollection`) batches of `(D, T, R)` updates. + +`ColumnarCollection` mirrors differential's `VecCollection` with `Column<(D, T, R)>` as the container instead of `Vec<(D, T, R)>`. + +`CollectionEdge` is an enum with two variants: +- `Vec` — row-formatted collection; the current default for all producers. +- `Columnar` — columnar collection; reserved for producers once the migration completes. + +The migration is consumer-first: every Plan-node consumer learns to accept both variants before any producer emits the columnar variant. Consumers that have not yet learned the columnar form fall back to `CollectionEdge::into_vec`, which decodes through a named `ColumnarToVec` operator. These repack seams remain visible in dataflow introspection so they can be found and retired. Pure passthrough consumers (Negate, Union) round-trip the columnar variant without decoding. diff --git a/doc/developer/generated/compute/render/join/delta_join.md b/doc/developer/generated/compute/render/join/delta_join.md index 5f72c25fc9f71..1e04070e23cd9 100644 --- a/doc/developer/generated/compute/render/join/delta_join.md +++ b/doc/developer/generated/compute/render/join/delta_join.md @@ -1,6 +1,6 @@ --- source: src/compute/src/render/join/delta_join.rs -revision: 1c55de49eb +revision: 58b5ff70c8 --- # mz-compute::render::join::delta_join diff --git a/doc/developer/generated/compute/render/sinks.md b/doc/developer/generated/compute/render/sinks.md index f2b20e28c9035..b5003d123e923 100644 --- a/doc/developer/generated/compute/render/sinks.md +++ b/doc/developer/generated/compute/render/sinks.md @@ -1,10 +1,10 @@ --- source: src/compute/src/render/sinks.rs -revision: d43cd78803 +revision: 94054eb165 --- # mz-compute::render::sinks -Entry point for rendering sink dataflow fragments; dispatches on `ComputeSinkConnection` variant to render either a `Subscribe`, `MaterializedView`, or `CopyToS3Oneshot` sink. +Entry point for rendering sink dataflow fragments; dispatches on `ComputeSinkConnection` variant to render either a `Subscribe`, `MaterializedView`, `CopyToS3Oneshot`, or `MetricSink` sink. Applies a `MapFilterProject` to the collection before passing it to the sink-specific rendering function, so that sinks only observe the columns they need. Also handles the `StartSignal` probe that delays sink writes until the dataflow has fully hydrated. diff --git a/doc/developer/generated/compute/server.md b/doc/developer/generated/compute/server.md index a66aa1b2a17b1..6da6220b6cd61 100644 --- a/doc/developer/generated/compute/server.md +++ b/doc/developer/generated/compute/server.md @@ -1,12 +1,12 @@ --- source: src/compute/src/server.rs -revision: bfa6499c3b +revision: 93dcb0ef5a --- # mz-compute::server Provides the `serve` entry point that starts a Timely compute cluster and returns a factory for `ComputeClient` handles. -`serve` accepts a `storage_log_readers` vec — one `StorageTimelyLogReader` per local worker — which is distributed to each `Worker` so that compute workers can replay storage timely log events alongside their own logging dataflow. It also registers column-pager Prometheus metrics via `mz_timely_util::column_pager::metrics::register`, wiring the process-wide `TieredPolicy` into the metrics registry. +`serve` accepts a `storage_log_readers` vec — one `StorageTimelyLogReader` per local worker — which is distributed to each `Worker` so that compute workers can replay storage timely log events alongside their own logging dataflow. It also registers column-pager Prometheus metrics via `mz_timely_util::column_pager::metrics::register`, wiring the process-wide `TieredPolicy` into the metrics registry, and registers pool configuration metrics via `mz_timely_util::pool_config::metrics::register`. `StorageTimelyLogReader` is a type alias for an `Arc>>` shared with the corresponding storage worker. The `Worker` struct drives the per-worker event loop: it steps Timely, handles incoming `ComputeCommand`s, processes pending peeks and subscribes, and performs periodic maintenance (frontier reporting, arrangement compaction, expiration checks). On reconnect, the `reconcile` method diffs the old command history against the new command batch to reuse compatible existing dataflows and compact or drop stale ones, ensuring workers remain consistent across client reconnects. diff --git a/doc/developer/generated/compute/sink/_module.md b/doc/developer/generated/compute/sink/_module.md index 51dda90629a01..be7d403296ad4 100644 --- a/doc/developer/generated/compute/sink/_module.md +++ b/doc/developer/generated/compute/sink/_module.md @@ -1,9 +1,9 @@ --- source: src/compute/src/sink.rs -revision: f9be03fda1 +revision: 94054eb165 --- # mz-compute::sink -Groups all compute sink implementations: `materialized_view` (parallel self-correcting persist sink), `materialized_view_v2` (an alternative MV sink implementation), `subscribe` (streaming update delivery), `copy_to_s3_oneshot` (snapshot export to S3), and `refresh` (timestamp rounding for scheduled MVs). +Groups all compute sink implementations: `materialized_view` (parallel self-correcting persist sink), `materialized_view_v2` (an alternative MV sink implementation), `subscribe` (streaming update delivery), `copy_to_s3_oneshot` (snapshot export to S3), `refresh` (timestamp rounding for scheduled MVs), and `metric_sink` (sink for emitting metrics). `correction` and `correction_v2` provide the update-buffering data structure used by the materialized view sink's `write_batches` operator; the active implementation is selected by a dyncfg flag. Both submodules are re-exported as `pub` when the crate is built with the `bench` feature, to allow benchmarks to exercise the correction buffer directly. diff --git a/doc/developer/generated/environmentd/_crate.md b/doc/developer/generated/environmentd/_crate.md index 90fbcb731f7a2..83b31c90fccbf 100644 --- a/doc/developer/generated/environmentd/_crate.md +++ b/doc/developer/generated/environmentd/_crate.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/lib.rs -revision: 34effa9dc0 +revision: 4e2b6c0984 --- # environmentd diff --git a/doc/developer/generated/environmentd/deployment/preflight.md b/doc/developer/generated/environmentd/deployment/preflight.md index 5f6408505fd51..1be88b8918c85 100644 --- a/doc/developer/generated/environmentd/deployment/preflight.md +++ b/doc/developer/generated/environmentd/deployment/preflight.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/deployment/preflight.rs -revision: 584bb9030c +revision: a60edac7f1 --- # environmentd::deployment::preflight @@ -8,3 +8,5 @@ revision: 584bb9030c Implements zero-downtime (0dt) preflight checks for new `environmentd` deployments. Compares the catalog's deploy generation against the incoming generation to determine whether to boot in read-only mode, and spawns a background task that waits for the deployment to catch up before fencing out the old environment and rebooting as leader. Also periodically checks for DDL changes on the old environment during the catch-up period, restarting the new process in read-only mode if new objects or replicas appear that need to be hydrated first. + +The DDL check (`check_ddl_changes`) computes baseline IDs from `get_next_ids`, which derives the next user item ID and replica ID from the maximum existing IDs in the catalog rather than from the allocator counter. This avoids false negatives when batch ID allocation (`IdPool`) has advanced the counter ahead of actually-committed items. diff --git a/doc/developer/generated/environmentd/http/mcp.md b/doc/developer/generated/environmentd/http/mcp.md index e9b70698eb580..0d61b11019996 100644 --- a/doc/developer/generated/environmentd/http/mcp.md +++ b/doc/developer/generated/environmentd/http/mcp.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/http/mcp.rs -revision: 780d92be42 +revision: 2736da1635 --- # environmentd::http::mcp @@ -10,7 +10,8 @@ Provides two endpoints: `/api/mcp/agent` (tools: `get_data_products`, `get_data_ Each endpoint is gated by a dynamic feature flag (`ENABLE_MCP_AGENT`, `ENABLE_MCP_DEVELOPER`, `ENABLE_MCP_AGENT_QUERY_TOOL`, `ENABLE_MCP_DEVELOPER_QUERY_TOOL`, `ENABLE_MCP_AGENT_READ_DATA_PRODUCT_TOOL`); `ENABLE_MCP_AGENT_READ_DATA_PRODUCT_TOOL` gates the `read_data_product` tool independently of the per-endpoint query tool flag. Response size is bounded by `MCP_MAX_RESPONSE_SIZE`. Request execution is bounded by `MCP_REQUEST_TIMEOUT`, a dynamic config that controls how long a tool call may run before a clean JSON-RPC timeout error is returned to the caller. Each request tags the session's `application_name` with `mz_mcp_agents` or `mz_mcp_developer` (via `set_default`, so a caller-supplied value still takes precedence) to make MCP-originated sessions visible in `mz_session_history` and `mz_statement_execution_history`. Implements MCP protocol version `2025-11-25`: the `initialize` response includes a `protocolVersion` field set to `MCP_PROTOCOL_VERSION`, tool definitions carry `title`, `annotations` (`ToolAnnotations` with `readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`), and `ToolContentResult` includes an `isError` field. -Both endpoints' `initialize` responses include usage instructions (via `endpoint_instructions`). The developer endpoint guides AI agents to use `mz_internal.mz_ontology_*` tables for discovering table schemas, join paths, and column names before writing queries; when `ENABLE_MCP_DEVELOPER_QUERY_TOOL` is on, the instructions also advertise the `query` tool. The agent endpoint instructs agents to prefer indexed objects (served from memory) over unindexed materialized views, and that `read_data_product` automatically routes the read to the cluster recorded in the data product catalog so indexes are used (provided the role has `USAGE` on that cluster; if not, the call fails with a `ClusterPrivilegeMissing` error rather than silently falling back to a slower path), and to set the `cluster` parameter only when intentionally targeting a different cluster (e.g. one with larger or more replicas). The agent endpoint also instructs agents to check the `hydration` object returned by `get_data_product_details` before reading: if `hydrated` is false and `replica_count > 0` the dataflow is warming up and a read would block until it catches up; if `replica_count` is 0 the cluster has no replicas and reading cannot make progress. +Both endpoints' `initialize` responses include usage instructions (via `endpoint_instructions`). The developer endpoint guides AI agents to use `mz_internal.mz_ontology_*` tables for discovering table schemas, join paths, and column names before writing queries; when `ENABLE_MCP_DEVELOPER_QUERY_TOOL` is on, the instructions also advertise the `query` tool. The agent endpoint instructs agents to prefer indexed objects (served from memory) over unindexed materialized views, and that `read_data_product` automatically routes the read to the cluster recorded in the data product catalog so indexes are used. The `cluster` field in `mz_mcp_data_products` is null when the session role lacks `USAGE` on the object's index/compute cluster; in that case `read_data_product` without a cluster override runs on the session's default cluster (safe: only materialized views appear this way and they serve from persist). Instructions also tell agents to set the `cluster` parameter only when intentionally targeting a different cluster (e.g. one with larger or more replicas). The agent endpoint also instructs agents to check the `hydration` object returned by `get_data_product_details` before reading: if `hydrated` is false and `replica_count > 0` the dataflow is warming up and a read would block until it catches up; if `replica_count` is 0 the cluster has no replicas and reading cannot make progress. +The developer endpoint's system-catalog query validation (`validate_system_catalog_query`) accepts unqualified `pg_catalog` names in addition to `mz_`-prefixed names; both are unambiguously system under the pinned `search_path` so a user-created view named `public.pg_class` cannot be reached via an unqualified reference. Both endpoints accept GET requests with a 405 Method Not Allowed response (via `handle_mcp_method_not_allowed`) and validate the `Origin` header against the CORS allowlist (injected as `Arc>` via Axum `Extension`) to prevent DNS rebinding attacks, rejecting origins not on the allowlist with 403 Forbidden. The CORS layer alone is insufficient because DNS rebinding causes the browser to treat the request as same-origin, bypassing preflight; the server-side check via `mz_http_util::origin_is_allowed` closes this gap. SQL parsing in MCP uses `parse_with_limit` for statements and `parse_item_name_with_limit` for item names; both enforce the 1 MB `MAX_STATEMENT_BATCH_SIZE` size check before allocating, preventing oversized inputs from being parsed. Enforces read-only SQL validation and AST-based system-table access control before executing queries; the developer endpoint allows SHOW and EXPLAIN statements without table references but rejects constant SELECT queries (e.g., `SELECT 1`) to prevent misuse for arbitrary computation. @@ -18,4 +19,4 @@ Enforces read-only SQL validation and AST-based system-table access control befo `execute_sql` delegates to the private `select_single_rows(results: Vec)` function, which returns the rows of the single row-returning statement in the response. `select_single_rows` surfaces the first error encountered; a second row-returning statement is an `Internal` error rather than a silently dropped result. The framing statements around a user query (`BEGIN`, `SET`, `COMMIT`) report `Ok` with no rows, so only the user's statement contributes rows. `McpResponse` is the top-level JSON-RPC 2.0 response envelope; `McpResponse::success(id, result)` builds a response carrying `result` and `McpResponse::error(id, error)` builds one carrying `error`, with exactly one of the two fields set. `McpRequestError` maps domain errors to standard JSON-RPC error codes. -Prometheus metrics are collected via `McpMetrics` (injected as an Axum `Extension`): request counts labeled by endpoint, JSON-RPC method, and status; tool call counts and durations labeled by endpoint and tool name. `ToolCallGuard` is an RAII guard that records tool call duration and status on drop, ensuring metrics are recorded even when a future is dropped early (e.g. by a timeout). +Prometheus metrics are collected via `McpMetrics` (injected as an Axum `Extension`): request counts labeled by endpoint, JSON-RPC method, and `McpCallStatus`; tool call counts and durations labeled by endpoint and tool name. `ToolCallGuard` is an RAII guard that records tool call duration and `McpCallStatus` on drop, ensuring metrics are recorded even when a future is dropped early (e.g. by a timeout). The `call_status` helper maps a handler result to `McpCallStatus::Ok` or `McpCallStatus::Error(e.error_type())`; timeout and endpoint-disabled outcomes are mapped directly at their decision points. diff --git a/doc/developer/generated/environmentd/http/mcp_metrics.md b/doc/developer/generated/environmentd/http/mcp_metrics.md index 6408a67117ec0..1d4eb6bc85738 100644 --- a/doc/developer/generated/environmentd/http/mcp_metrics.md +++ b/doc/developer/generated/environmentd/http/mcp_metrics.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/http/mcp_metrics.rs -revision: d137bfbf58 +revision: 2736da1635 --- # environmentd::http::mcp_metrics @@ -11,12 +11,16 @@ Defines Prometheus metrics for the MCP HTTP endpoints and the RAII guard used to `McpMetrics` holds three Prometheus collectors, all registered via `MetricsRegistry`: -* `mz_mcp_requests_total` (`IntCounterVec`) — total MCP requests, labeled by `endpoint_type` (`agent` or `developer`), `method` (JSON-RPC method name), and `status` (`ok` or an `McpRequestError` variant such as `ToolNotFound` or `DataProductNotFound`). +* `mz_mcp_requests_total` (`IntCounterVec`) — total MCP requests, labeled by `endpoint_type` (`agent` or `developer`), `method` (JSON-RPC method name), and `status` (a `McpCallStatus` variant: `ok`, `cancelled`, `timeout`, `endpoint_disabled`, or an `McpRequestError::error_type()` string such as `ToolNotFound`). * `mz_mcp_tool_calls_total` (`IntCounterVec`) — total `tools/call` invocations, labeled by `endpoint_type`, `tool_name`, and `status`. * `mz_mcp_tool_call_duration_seconds` (`HistogramVec`) — duration of `tools/call` invocations, labeled by `endpoint_type` and `tool_name`, using `histogram_seconds_buckets(0.000_128, 8.0)`. `McpMetrics` implements `Clone`; Prometheus collector handles are `Arc`-shared internally, so cloning is cheap. The struct is stored and passed as an Axum `Extension`. +## McpCallStatus + +`McpCallStatus` is a closed enum of outcomes recorded in the MCP `status` metric label. Using an enum rather than free-form strings at call sites pins the label cardinality and prevents typos from silently creating new label values. Variants: `Ok`, `Cancelled`, `Timeout`, `EndpointDisabled`, and `Error(&'static str)` (carrying the `McpRequestError::error_type()` string). `as_str()` converts each variant to its wire label string. + ## ToolCallGuard -`ToolCallGuard` is an RAII guard for a single `tools/call` invocation. On construction it starts a `HistogramTimer` for `mz_mcp_tool_call_duration_seconds`. On drop it increments `mz_mcp_tool_calls_total` with the current status label. The default status is `"cancelled"`, so if the surrounding future is dropped before completion (e.g. by `tokio::time::timeout`), the metric records as cancelled rather than being silently lost. Callers set the status to the actual outcome via `set_status` before the guard goes out of scope. +`ToolCallGuard` is an RAII guard for a single `tools/call` invocation. On construction it starts a `HistogramTimer` for `mz_mcp_tool_call_duration_seconds`. On drop it increments `mz_mcp_tool_calls_total` with the current `McpCallStatus`. The default status is `McpCallStatus::Cancelled`, so if the surrounding future is dropped before completion (e.g. by `tokio::time::timeout`), the metric records as cancelled rather than being silently lost. Callers set the status to the actual outcome via `set_status` before the guard goes out of scope. `McpMetrics::record_request` encapsulates the label ordering and `McpCallStatus` to `&str` conversion so call sites never touch raw label strings. diff --git a/doc/developer/generated/environmentd/telemetry.md b/doc/developer/generated/environmentd/telemetry.md index 2b91d7e7b7600..c1cb247bc9a3d 100644 --- a/doc/developer/generated/environmentd/telemetry.md +++ b/doc/developer/generated/environmentd/telemetry.md @@ -1,9 +1,10 @@ --- source: src/environmentd/src/telemetry.rs -revision: 61212870d6 +revision: 4e2b6c0984 --- # environmentd::telemetry Runs a periodic reporting loop that queries the adapter for environment statistics (active sources, sinks, views, clusters, etc.) and sends them to Segment via `group` (latest state) and `track` ("Environment Rolled Up" event with delta statistics). The loop starts via `start_reporting` and fires at a configurable `report_interval`; the first tick only sends traits, subsequent ticks also compute and report delta counters for DML operations. +`Config` carries a `license_key: ValidatedLicenseKey` field. Each reporting interval, the loop merges the license key's organization ID, environment ID, key ID, expiration timestamp, expired flag, and expiration behavior into the traits map before the Segment `group` and `track` calls. Expiry is recomputed against the wall clock each interval so a key that lapses while environmentd keeps running is reported as expired. diff --git a/doc/developer/generated/expr/scalar/func/impls/date.md b/doc/developer/generated/expr/scalar/func/impls/date.md index aa933d8d7d096..f9837a2d1f618 100644 --- a/doc/developer/generated/expr/scalar/func/impls/date.md +++ b/doc/developer/generated/expr/scalar/func/impls/date.md @@ -1,6 +1,6 @@ --- source: src/expr/src/scalar/func/impls/date.rs -revision: 703a0c27c8 +revision: 26cb0194dc --- # mz-expr::scalar::func::impls::date diff --git a/doc/developer/generated/expr/scalar/func/impls/timestamp.md b/doc/developer/generated/expr/scalar/func/impls/timestamp.md index 3c9eb99d0a5f8..faef25b21554d 100644 --- a/doc/developer/generated/expr/scalar/func/impls/timestamp.md +++ b/doc/developer/generated/expr/scalar/func/impls/timestamp.md @@ -1,6 +1,6 @@ --- source: src/expr/src/scalar/func/impls/timestamp.rs -revision: cda6e6e0af +revision: 26cb0194dc --- # mz-expr::scalar::func::impls::timestamp diff --git a/doc/developer/generated/expr/scalar/func/variadic.md b/doc/developer/generated/expr/scalar/func/variadic.md index 6f2dfd1f4e5ee..d9501a54dd110 100644 --- a/doc/developer/generated/expr/scalar/func/variadic.md +++ b/doc/developer/generated/expr/scalar/func/variadic.md @@ -1,6 +1,6 @@ --- source: src/expr/src/scalar/func/variadic.rs -revision: 641dc71585 +revision: 26cb0194dc --- # mz-expr::scalar::func::variadic diff --git a/doc/developer/generated/interchange/envelopes.md b/doc/developer/generated/interchange/envelopes.md index 572fd9cd34ffd..a2f273e402862 100644 --- a/doc/developer/generated/interchange/envelopes.md +++ b/doc/developer/generated/interchange/envelopes.md @@ -1,10 +1,14 @@ --- source: src/interchange/src/envelopes.rs -revision: 1c55de49eb +revision: 2d842fe6d1 --- # interchange::envelopes -Provides `for_each_diff_pair`, a function that walks a single arrangement batch and invokes a callback for each `DiffPair` at each `(key, timestamp)`. -The function is generic over both a batch type `B` (bounded by `BatchReader