From c690900b912d3e20f1213b61681869a1d59d6ffc Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Mon, 20 Jul 2026 07:05:49 +0000 Subject: [PATCH 1/6] doc: update generated developer documentation Automated daily refresh of doc/developer/generated/ to reflect source changes since each doc was last written. Co-Authored-By: Claude Opus 4.6 (1M context) --- doc/developer/generated/sql/session/vars/definitions.md | 4 ++-- doc/developer/generated/sqllogictest/ast.md | 5 +++-- doc/developer/generated/sqllogictest/parser.md | 6 +++++- doc/developer/generated/sqllogictest/runner.md | 6 +++++- 4 files changed, 15 insertions(+), 6 deletions(-) diff --git a/doc/developer/generated/sql/session/vars/definitions.md b/doc/developer/generated/sql/session/vars/definitions.md index 6a78ddeb58db0..7ab27ea1309d4 100644 --- a/doc/developer/generated/sql/session/vars/definitions.md +++ b/doc/developer/generated/sql/session/vars/definitions.md @@ -1,6 +1,6 @@ --- source: src/sql/src/session/vars/definitions.rs -revision: 2ea92e64a5 +revision: a7b4b4f70f --- # mz-sql::session::vars::definitions @@ -11,5 +11,5 @@ The `feature_flags!` macro accepts an optional `scope: ,` field per flag e The `lazy_value!` and `value!` macros (from `polyfill`) are used extensively to express default values that cannot be computed at compile time. This file is the authoritative source of truth for which variables exist and their defaults. `RESTRICT_TO_USER_OBJECTS` is a read-only `bool` session variable (default `false`) that restricts queries from accessing system catalog objects; it is designed to be set only via `ALTER ROLE ... SET` by superusers. -Feature flags include `enable_repeat_row_non_negative` (guards the `repeat_row_non_negative` table function), `enable_storage_introspection_logs` (guards forwarding storage timely logging events into the compute introspection dataflow), `enable_kafka_broker_matching_rules` (guards `MATCHING` broker rules in `BROKERS` for Kafka PrivateLink connections), `enable_glue_schema_registry` (guards `CREATE CONNECTION ... TO AWS GLUE SCHEMA REGISTRY`), `enable_bounded_staleness_isolation` (guards the `bounded staleness ` transaction isolation level; defaults to `true`), `enable_fixed_correlated_cte_lowering` (gates the corrected HIR-to-MIR lowering path that uses CTE-aware branch keys when decorrelating CTEs referenced from nested correlated scopes; defaults to `true`), `enable_statement_arrival_logging` (when enabled, SQL frontends log incoming statements at info level as they arrive, before processing, with literals redacted; intended as an emergency diagnostic for incidents where statements crash the process before reaching the statement log; defaults to `false`), `unsafe_enable_unbounded_custom_type_resolution` (default `false`, `enable_for_item_parsing: true`; when enabled, custom type resolution proceeds without the nesting-depth and total-node limits that ordinarily bound resolution work), and `enable_auto_scaling_strategy` (guards the `AUTO SCALING STRATEGY` cluster option; default `true`, `enable_for_item_parsing: true`; `RESET (AUTO SCALING STRATEGY)` is intentionally not gated by this flag so clusters can shed an autoscaling policy after a flag rollback). +Feature flags include `enable_repeat_row_non_negative` (guards the `repeat_row_non_negative` table function), `enable_storage_introspection_logs` (guards forwarding storage timely logging events into the compute introspection dataflow), `enable_kafka_broker_matching_rules` (guards `MATCHING` broker rules in `BROKERS` for Kafka PrivateLink connections), `enable_glue_schema_registry` (guards `CREATE CONNECTION ... TO AWS GLUE SCHEMA REGISTRY`; defaults to `true`), `enable_bounded_staleness_isolation` (guards the `bounded staleness ` transaction isolation level; defaults to `true`), `enable_fixed_correlated_cte_lowering` (gates the corrected HIR-to-MIR lowering path that uses CTE-aware branch keys when decorrelating CTEs referenced from nested correlated scopes; defaults to `true`), `enable_statement_arrival_logging` (when enabled, SQL frontends log incoming statements at info level as they arrive, before processing, with literals redacted; intended as an emergency diagnostic for incidents where statements crash the process before reaching the statement log; defaults to `false`), `unsafe_enable_unbounded_custom_type_resolution` (default `false`, `enable_for_item_parsing: true`; when enabled, custom type resolution proceeds without the nesting-depth and total-node limits that ordinarily bound resolution work), and `enable_auto_scaling_strategy` (guards the `AUTO SCALING STRATEGY` cluster option; default `true`, `enable_for_item_parsing: true`; `RESET (AUTO SCALING STRATEGY)` is intentionally not gated by this flag so clusters can shed an autoscaling policy after a flag rollback). `DEFAULT_TIMESTAMP_INTERVAL`, `CLUSTER_CHECK_SCHEDULING_POLICIES_INTERVAL`, `SSH_CHECK_INTERVAL`, and `STORAGE_STATISTICS_INTERVAL` each carry a `NON_ZERO_DURATION` constraint, so setting any of them to zero raises an `InvalidParameterValue` error. diff --git a/doc/developer/generated/sqllogictest/ast.md b/doc/developer/generated/sqllogictest/ast.md index d9a2ea22a2af1..2515dcbe55782 100644 --- a/doc/developer/generated/sqllogictest/ast.md +++ b/doc/developer/generated/sqllogictest/ast.md @@ -1,11 +1,12 @@ --- source: src/sqllogictest/src/ast.rs -revision: 80d40a68c8 +revision: fc9d219c84 --- # sqllogictest::ast -Defines the abstract syntax tree for sqllogictest files: `Record` (a single directive such as `statement`, `query`, `simple`, `halt`, `copy`, or `replace`), `Output` (expected results as literal values or an MD5 hash), `Sort` (no-sort, rowsort, or valuesort), `Type` (column type hint), and `Mode` (Standard vs Cockroach output format). +Defines the abstract syntax tree for sqllogictest files: `Record` (a single directive such as `statement`, `query`, `simple`, `halt`, `copy`, `user`, or `replace`), `Output` (expected results as literal values or an MD5 hash), `Sort` (no-sort, rowsort, or valuesort), `Type` (column type hint), and `Mode` (Standard vs Cockroach output format). `QueryOutput` bundles sort order, column types, mode, and expected output for a `query` directive. +`Record::User` carries a `location` and a `user` name; it switches the session user for subsequent records (creating the role if it does not exist yet), as in CockroachDB's logic tests. `Record::Replace` carries a `pattern: String` (a validated regex) and a `replacement: String`; it registers a substitution applied to the actual output of every subsequent `query` before comparison or rewriting, used to mask non-deterministic tokens such as a folded `mz_now()` timestamp in an `EXPLAIN` plan. These types are produced by the parser and consumed by the runner. diff --git a/doc/developer/generated/sqllogictest/parser.md b/doc/developer/generated/sqllogictest/parser.md index 6322266d48ee4..5284600cb8b12 100644 --- a/doc/developer/generated/sqllogictest/parser.md +++ b/doc/developer/generated/sqllogictest/parser.md @@ -1,6 +1,6 @@ --- source: src/sqllogictest/src/parser.rs -revision: 80d40a68c8 +revision: fc9d219c84 --- # sqllogictest::parser @@ -8,4 +8,8 @@ revision: 80d40a68c8 Implements `Parser<'a>`, a line-oriented parser that converts raw sqllogictest file contents into an iterator of `Record` values defined in `ast`. Tracks current file name and line number for error reporting via `Location`. Supports both Standard and Cockroach dialect modes, switching when it encounters a `mode cockroach` directive. +Parses `user ` directives into `Record::User` to switch the active session user. +Parses `let $var` directives by skipping the directive and its following query block; variables referenced in later records fail at execution time rather than parse time. +Recognizes `skip_on_retry`, `retry`, and `noticetrace` query options: `retry` queries are run once like any other; `noticetrace` queries are skipped entirely because notices are not observable. +Accepts `statement notice ` (treated as plain success), and a lenient `ok`-prefix match for statement dispositions to accommodate CockroachDB's typos (`oK`, `ok;`, `oko`). Parses `replace ` directives (regex and replacement separated by two-or-more spaces) into `Record::Replace`; the regex is validated at parse time so errors are reported with file/line context. diff --git a/doc/developer/generated/sqllogictest/runner.md b/doc/developer/generated/sqllogictest/runner.md index 0264bebf8a97c..e0e7feb362ac7 100644 --- a/doc/developer/generated/sqllogictest/runner.md +++ b/doc/developer/generated/sqllogictest/runner.md @@ -1,6 +1,6 @@ --- source: src/sqllogictest/src/runner.rs -revision: 34effa9dc0 +revision: fc9d219c84 --- # sqllogictest::runner @@ -9,4 +9,8 @@ Implements the Materialize-specific sqllogictest runner: spins up a full `mz-env Provides `Runner`, `RunConfig`, `Outcomes`, and `WriteFmt` as the primary API for running test files; supports result rewriting, multiple connections/users, COPY directives, and JUnit XML output. The runner handles timestamp management needed for serialized execution semantics and translates Materialize-specific types (numerics, intervals, dates, ACL items, etc.) into the text format sqllogictest expects. When `--auto-index-selects` is enabled, the runner executes each SELECT query normally first; only if that query succeeds does it additionally execute the query via a view-backed indexed path to verify consistency. If the view-backed execution produces a different result, the runner reports an `InconsistentViewOutcome` (or a warning if the inconsistency matches known acceptable patterns). +`RunnerInner` holds an `active_user: Option` field tracking the user set by the most recent `user` directive; records run on that user's cached connection instead of the default connection. The `run_user` method ensures the role exists, connects as it, and updates `active_user`. On `reset-server`, all non-default user roles are dropped (best-effort) and `active_user` is cleared. +When a column's declared type does not match the value the query returned, `format_datum` falls back to text encoding rather than panicking, so the mismatch surfaces as an output diff instead of aborting the run. +`error_matches` wraps expected-error regex matching: if the expected-error string is not a valid regex (common in CockroachDB-imported files), it returns `false` so the record fails with a mismatch outcome rather than aborting the run. +`strip_crdb_table_items` removes CockroachDB physical-layout items (`INDEX`, `UNIQUE INDEX`, `INVERTED INDEX`, `FAMILY`) from `CREATE TABLE` column lists so that CockroachDB test files can be parsed; constraints are kept unchanged. `Runner` holds a `replacements: Vec<(Regex, String)>` field that accumulates substitutions from `Record::Replace` directives. These are applied to the actual output of each `query` before comparison or rewriting, masking non-deterministic tokens. Replacements are stored on the outer `Runner` (not on `RunnerInner`) so they survive a `reset-server` directive. From 004d46b99c3040a15ba72b38d33a4c6db8775aee Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Tue, 21 Jul 2026 07:02:52 +0000 Subject: [PATCH 2/6] doc: update generated developer documentation Automated daily refresh of doc/developer/generated/ to reflect source changes since each doc was last written. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../generated/adapter/coord/cluster_controller.md | 4 ++-- .../generated/adapter/coord/sequencer/inner/cluster.md | 4 ++-- doc/developer/generated/catalog/builtin/mz_internal.md | 4 ++-- doc/developer/generated/cluster-controller/ctx.md | 4 ++-- doc/developer/generated/cluster-controller/strategy.md | 4 ++-- doc/developer/generated/environmentd/_crate.md | 2 +- doc/developer/generated/environmentd/http/mcp.md | 7 ++++--- .../generated/environmentd/http/mcp_metrics.md | 10 +++++++--- doc/developer/generated/environmentd/telemetry.md | 3 ++- .../generated/sql/plan/statement/ddl/connection.md | 4 ++-- doc/developer/generated/sql/plan/transform_ast.md | 3 ++- doc/developer/generated/sql/rbac.md | 4 ++-- .../generated/sql/session/vars/definitions.md | 4 ++-- doc/developer/generated/storage-client/controller.md | 3 ++- doc/developer/generated/storage-controller/_crate.md | 2 +- doc/developer/generated/storage-controller/instance.md | 3 ++- .../generated/storage-types/sources/_module.md | 3 ++- 17 files changed, 39 insertions(+), 29 deletions(-) diff --git a/doc/developer/generated/adapter/coord/cluster_controller.md b/doc/developer/generated/adapter/coord/cluster_controller.md index 9b2fbc04f8bb8..22fb8e70eadd9 100644 --- a/doc/developer/generated/adapter/coord/cluster_controller.md +++ b/doc/developer/generated/adapter/coord/cluster_controller.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/cluster_controller.rs -revision: 74f18a3354 +revision: bbe8b6977e --- # adapter::coord::cluster_controller @@ -17,7 +17,7 @@ Key types and functions: * `CoordCtx` — the controller-task side of the `ClusterControllerCtx` boundary. Marshals every ctx call to the Coordinator via `internal_cmd_tx`. Latches `now` from the most recent `ClusterStates` reply so both phases of a tick see a consistent time. * `Coordinator::spawn_cluster_controller_task` — spawns the controller task. The task loops: reads the tick interval via a `TickInterval` request, sleeps, then calls `ClusterController::reconcile` if the gate is on. Called once at bootstrap. The `ClusterController` is constructed with a shared dyncfg handle so flag flips take effect without a restart. * `Coordinator::handle_cluster_controller_request` — the Coordinator-side handler for `Message::ClusterControllerRequest`. Dispatches each variant to the appropriate catalog/controller reads or applies the decision batch via `apply_cluster_controller_decisions`. For `HydratedReplicas`, starts per-replica hydration checks on the coordinator loop (via `start_hydration_checks`) then waits for compute's replies off-loop in a spawned task. For `HasHydratableObjects`, delegates to `cluster_has_hydratable_objects`. -* `Coordinator::start_hydration_checks` — checks storage and compute hydration for a set of replicas on a cluster. Returns only replicas that are already storage-hydrated and known to the compute controller. Replica-pinned materialized views (those with `IN CLUSTER ... REPLICA`) are excluded from the hydration check for replicas they are not pinned to, so a graceful reconfiguration does not wait forever for a new replica to hydrate an MV bound to a replica being replaced. +* `Coordinator::start_hydration_checks` — checks that replicas are online, storage-hydrated, and known to the compute controller for a set of replicas on a cluster. Returns only replicas whose processes are all online, that are already storage-hydrated, and that are known to the compute controller. Replica-pinned materialized views (those with `IN CLUSTER ... REPLICA`) are excluded from the hydration check for replicas they are not pinned to, so a graceful reconfiguration does not wait forever for a new replica to hydrate an MV bound to a replica being replaced. * `Coordinator::cluster_has_hydratable_objects` — returns whether any object bound to the cluster is hydratable (dataflow-backed). Backs the `HasHydratableObjects` ctx pull; a `false` answer prevents the burst strategy from arming when there is nothing to hydrate. * `Coordinator::apply_cluster_controller_decisions` — transacts a decision batch: for each `Decision`, checks `check_cluster_state` and, if the state still matches, creates or drops the replica or writes the state. Returns `ApplyOutcome::Rejected` if any check fails, `ApplyOutcome::ResourceExhausted` if the batch exceeds the resource budget, and `Applied` otherwise. Create decisions carry a strategy attribution translated to a `ReplicaCreateDropReason` via `reason_from_strategies`: the graceful reconfiguration strategy maps to `GracefulReconfiguration`; the hydration-burst strategy maps to `HydrationBurst`; all others map to `Manual`. `build_mutation_ops` (called from `apply_cluster_controller_decisions`) checks whether the target replica of a `DropReplica` decision still exists in the catalog before building the drop op. If it has already been removed (e.g. by a concurrent user `DROP CLUSTER`), the function returns `None` to reject the batch, avoiding a panic from resource-limit validation running against a missing replica. * Observed replicas passed to `ClusterController::reconcile` are built from all replicas on the cluster (not pre-filtered): each `ObservedReplica` carries `internal`, `billed_as`, and `pending` flags so the controller's ownership classifier (`ObservedReplica::owned_shape`) can decide which replicas it governs. The `StateWrite` applied to the managed config is exhaustively destructured (no `..`) so that adding a field to `StateWrite` is a compile error until the overlay logic accounts for it. diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md index e4085a758f715..34c92564ebaae 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/cluster.rs -revision: 74f18a3354 +revision: bbe8b6977e --- # adapter::coord::sequencer::inner::cluster @@ -11,7 +11,7 @@ Replica IDs are pre-allocated out-of-band via `Catalog::allocate_replica_ids` be Cluster and replica creation and deletion are applied through the catalog implications pipeline rather than directly in this module. For `CREATE CLUSTER` (both managed and unmanaged variants), `scoped_overrides_create_op` is called with `ClusterEvalContext` and `ReplicaEvalContext` values built from plan data and pre-allocated replica IDs, and the resulting `Op::UpdateScopedSystemParameters` is folded into the same catalog transaction that creates the cluster. This makes the committed diff drive the replica-scoped controller push before `create_replica`, so the new cluster's optimizer and each new replica's render-frozen flags take effect before any dataflow renders, rather than falling back to environment-wide values until the next sync tick. `CREATE CLUSTER` validates the `HYDRATION SIZE` of an `AUTO SCALING STRATEGY` against the allowed replica sizes for the session role, just as `SIZE` itself is validated. This prevents a typo from failing silently at burst-arm time and prevents a size-restricted role from bursting at a size it may not `CREATE` with. -When `ENABLE_CLUSTER_CONTROLLER` is on and the cluster is user-owned, a shape-changing managed-to-managed `ALTER CLUSTER` (one that changes `SIZE`, `AVAILABILITY ZONES`, or `INTROSPECTION`) reshapes into a durable `reconfiguration` record via `reshape_alter_cluster_managed` instead of going through the legacy 3-stage machine. The record carries the full target config shape, a deadline, and an `on_timeout` action (`ROLLBACK` by default). Non-shape fields (`workload_class`, `schedule`, `auto_scaling_strategy`, etc.) are applied to the realized config immediately. The controller converges the replica set onto the target and cuts the realized config over at hydration. `ClusterStage::AwaitReconfiguration` (carrying `AlterClusterAwaitReconfiguration`) is a foreground wait-shim that polls the durable record until it reaches a terminal status, then reports success or failure depending on whether the realized config reached the target; it is used when `ENABLE_BACKGROUND_ALTER_CLUSTER` is off. +When `ENABLE_CLUSTER_CONTROLLER` is on and the cluster is user-owned, a shape-changing managed-to-managed `ALTER CLUSTER` (one that changes `SIZE`, `AVAILABILITY ZONES`, or `INTROSPECTION`) reshapes into a durable `reconfiguration` record via `reshape_alter_cluster_managed` instead of going through the legacy 3-stage machine. The record carries the full target config shape, a deadline, and an `on_timeout` action (`ROLLBACK` by default). Non-shape fields (`workload_class`, `schedule`, `auto_scaling_strategy`, etc.) are applied to the realized config immediately. The controller converges the replica set onto the target and cuts the realized config over at hydration. `ClusterStage::AwaitReconfiguration` (carrying `AlterClusterAwaitReconfiguration`) is a foreground wait-shim that polls the durable record until it reaches a terminal status, then reports success or failure depending on whether the realized config reached the target; it is used when `ENABLE_BACKGROUND_ALTER_CLUSTER` is off. The legacy foreground `AlterClusterWaitForHydrated` readiness check additionally requires every pending replica to be online (per `mz_cluster_replica_statuses`) before declaring the reconfiguration ready to cut over, so a cluster hosting only single-replica sources (which are never placed on a pending replica until cut-over) does not declare readiness vacuously the moment the pending replicas are created. When a reconfiguration is in flight, a subsequent shape-changing `ALTER` folds onto the in-flight target via `fold_reconfiguration_target`: dimensions the `ALTER` set explicitly take the new value, while dimensions left `Unchanged` keep the in-flight target's value (not the realized config's value, which would silently revert the in-flight transition). An `ALTER` back to the realized shape cancels the record. Changing the replication factor while a reconfiguration is in flight is refused with `AdapterError::AlterClusterReplicationFactorWhileReconfiguring`. `ALTER CLUSTER` validates a newly set or changed `AUTO SCALING STRATEGY`'s `HYDRATION SIZE` like `SIZE` itself: it must name a real replica size the session role may use. Only a changed strategy is checked, so an existing policy does not block unrelated `ALTER`s if the size allow-list later shrinks. Additionally, if a reconfiguration is in flight, `HYDRATION SIZE` must differ from the in-flight reconfiguration's target `SIZE`: equality would end the reshape with a no-op burst shape and a stored statement that fails its own re-plan. Unmanaged clusters reject `AUTO SCALING STRATEGY` changes with an error, mirroring how other managed-only options are handled. Converting a cluster to unmanaged while a hydration burst is in flight is refused with `AdapterError::AlterClusterUnmanagedWhileBursting`. The unmanaged variant carries no burst field, so converting would drop the record without a `Finished` audit event and strand the burst replica as an ordinary unmanaged replica nothing ever tears down. diff --git a/doc/developer/generated/catalog/builtin/mz_internal.md b/doc/developer/generated/catalog/builtin/mz_internal.md index b876ee9c17fb5..92ff22d054016 100644 --- a/doc/developer/generated/catalog/builtin/mz_internal.md +++ b/doc/developer/generated/catalog/builtin/mz_internal.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/builtin/mz_internal.rs -revision: 34813f0a3f +revision: 2ba315025c --- # catalog::builtin::mz_internal @@ -15,7 +15,7 @@ This is the largest builtin submodule, exporting 199 public items: sources, tabl **Materialized views** (`BuiltinMaterializedView`) — Derived catalog views backed by queries over `mz_catalog_raw` and other sources: aggregated statistics, lag histograms, and other derived metrics. `MZ_POSTGRES_SOURCES` is a `BuiltinMaterializedView` (OID constant `MV_MZ_POSTGRES_SOURCES_OID`) that derives PostgreSQL source details (`replication_slot`, `timeline_id`) by parsing the raw catalog create SQL via `mz_internal.parse_postgres_source_details`; the MV filters to postgres sources first via `parse_catalog_create_sql` because the detail decoder must not be applied to non-postgres rows. `MZ_OVERRIDDEN_SYSTEM_PARAMETERS` projects the `system_configurations` durable collection out of `mz_catalog_raw`, exposing environment-wide system parameter overrides as `(name, value)` pairs with `PUBLIC_SELECT` access and an `Ontology` annotation; only parameters with an explicit override appear (defaults are absent). `MZ_CLUSTER_SYSTEM_PARAMETERS` projects the `cluster_system_configurations` durable collection out of `mz_catalog_raw`, exposing per-cluster system parameter overrides keyed by `(cluster_id, name)`. `MZ_REPLICA_SYSTEM_PARAMETERS` does the same for the `replica_system_configurations` collection, exposing per-replica overrides keyed by `(replica_id, name)`. `MZ_COMMENTS` is a `BuiltinMaterializedView` backed by a query over `mz_catalog_raw` that reads `Comment` entries, decodes each `proto::CommentObject` variant via CASE expressions, and projects `(id, object_type, object_sub_id, comment)` rows; new `CommentObject` variants must be added to both CASE expressions in its SQL. `MZ_CLUSTER_RECONFIGURATIONS` (OID `MV_MZ_CLUSTER_RECONFIGURATIONS_OID`) is a `BuiltinMaterializedView` with a unique key on `cluster_id` that exposes the latest graceful reconfiguration record per managed cluster, with columns `cluster_id`, `status`, `deadline`, `on_timeout`, `target` (JSONB), and `changes` (JSONB). Terminal records are retained after settling so the outcome stays inspectable until a later reconfiguration overwrites it. The `status` column maps serde variant names from `ReconfigurationStatus` to kebab-case (e.g. `InProgress` -> `in-progress`); unmapped variants pass through verbatim. `changes` diffs `target` against the realized config per dimension. `MZ_CLUSTER_AUTO_SCALING_STRATEGIES` (OID `MV_MZ_CLUSTER_AUTO_SCALING_STRATEGIES_OID`) is a `BuiltinMaterializedView` with a unique key on `cluster_id` that exposes the configured autoscaling strategy and in-flight state per managed cluster; `state` is keyed by strategy type so a future strategy's state is an additional key rather than a schema change. Both views have corresponding `BuiltinIndex` constants (`MZ_CLUSTER_RECONFIGURATIONS_IND`, `MZ_CLUSTER_AUTO_SCALING_STRATEGIES_IND`) on `cluster_id`. -**Views** (`BuiltinView`) — 82 entries covering the full `mz_internal` SQL surface: cluster introspection, compute operator metrics, peek durations, arrangement sizes, source/sink status, wall-clock lag, statement execution history, privilege management, dependency graph views, and more. `MZ_SOURCE_STATUSES` and `MZ_SINK_STATUSES` filter their `latest_per_replica_events` CTE to include events from still-existing replicas or the global sentinel, and additionally retain rows where `status = 'paused'` because a per-replica `paused` event is only written when the replica is dropped and acts as a terminal drop marker. Precedence ties in `latest_events` are broken by `occurred_at DESC` in addition to status precedence number, so a dropped replica's `paused` event wins over an older source-global `paused`. `MZ_SHOW_COLUMNS` applies an `object_type` predicate on its comment join to guard against stale comment rows from type changes. `MZ_SHOW_CLUSTERS` includes an `activity` column (nullable string) that summarizes any in-flight reconfiguration or autoscaling burst: it joins `mz_cluster_reconfigurations` (filtering to `status = 'in-progress'`) and `mz_cluster_auto_scaling_strategies` (checking for a non-null `state`) and produces a human-readable summary of the dimensions being changed and the burst size; `NULL` when the cluster is steady. Views carry inline SQL queries over `mz_catalog`, `mz_introspection`, and `mz_internal` tables. Access levels vary: most grant `PUBLIC_SELECT`; sensitive views use `MONITOR_SELECT`, `MONITOR_REDACTED_SELECT`, `SUPPORT_SELECT`, or `ANALYTICS_SELECT`. `MZ_MCP_DATA_PRODUCTS` and `MZ_MCP_DATA_PRODUCT_DETAILS` are MCP-agent-facing views listing data products and their column/key details that are exempted from `restrict_to_user_objects` blocking. `MZ_MCP_DATA_PRODUCT_DETAILS` includes a `hydration` column (JSONB) reporting readiness across the cluster's replicas, with fields `hydrated` (bool), `replica_count` (int), and `hydrated_replica_count` (int). `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW` (1-hour buckets, 14-day retention), `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW_24H` (5-minute buckets, 24-hour retention), and `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW_3H` (un-binned raw samples, 3-hour retention) are three cluster utilization views for the Console's time-range graphs; their SQL bodies are generated by shared helpers so their fingerprints stay in sync with the Console's equivalent ad-hoc query. +**Views** (`BuiltinView`) — 82 entries covering the full `mz_internal` SQL surface: cluster introspection, compute operator metrics, peek durations, arrangement sizes, source/sink status, wall-clock lag, statement execution history, privilege management, dependency graph views, and more. `MZ_SOURCE_STATUSES` and `MZ_SINK_STATUSES` filter their `latest_per_replica_events` CTE to include events from still-existing replicas or the global sentinel, and additionally retain rows where `status = 'paused'` because a per-replica `paused` event is only written when the replica is dropped and acts as a terminal drop marker. Precedence ties in `latest_events` are broken by `occurred_at DESC` in addition to status precedence number, so a dropped replica's `paused` event wins over an older source-global `paused`. `MZ_SHOW_COLUMNS` applies an `object_type` predicate on its comment join to guard against stale comment rows from type changes. `MZ_SHOW_CLUSTERS` includes an `activity` column (nullable string) that summarizes any in-flight reconfiguration or autoscaling burst: it joins `mz_cluster_reconfigurations` (filtering to `status = 'in-progress'`) and `mz_cluster_auto_scaling_strategies` (checking for a non-null `state`) and produces a human-readable summary of the dimensions being changed and the burst size; `NULL` when the cluster is steady. Views carry inline SQL queries over `mz_catalog`, `mz_introspection`, and `mz_internal` tables. Access levels vary: most grant `PUBLIC_SELECT`; sensitive views use `MONITOR_SELECT`, `MONITOR_REDACTED_SELECT`, `SUPPORT_SELECT`, or `ANALYTICS_SELECT`. `MZ_MCP_DATA_PRODUCTS` and `MZ_MCP_DATA_PRODUCT_DETAILS` are MCP-agent-facing views listing data products and their column/key details that are exempted from `restrict_to_user_objects` blocking. The `cluster` column in both views is null unless the session role has `USAGE` on the object's index/compute cluster (checked via `mz_show_my_cluster_privileges`); materialized views still appear regardless because they serve from persist, but plain indexed views without at least one usable index cluster are excluded. `MZ_MCP_DATA_PRODUCT_DETAILS` includes a `hydration` column (JSONB) reporting readiness across the cluster's replicas, with fields `hydrated` (bool), `replica_count` (int), and `hydrated_replica_count` (int); the hydration join uses the real cluster name even when the `cluster` column is null. `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW` (1-hour buckets, 14-day retention), `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW_24H` (5-minute buckets, 24-hour retention), and `MZ_CONSOLE_CLUSTER_UTILIZATION_OVERVIEW_3H` (un-binned raw samples, 3-hour retention) are three cluster utilization views for the Console's time-range graphs; their SQL bodies are generated by shared helpers so their fingerprints stay in sync with the Console's equivalent ad-hoc query. `MZ_BUILTIN_SOURCES` is a view in `mz_internal` listing builtin and log sources that do not appear in `mz_catalog_raw`; user sources are exposed via `mz_catalog.mz_sources`. **Connections** (`BuiltinConnection`) — System-level connection definitions. diff --git a/doc/developer/generated/cluster-controller/ctx.md b/doc/developer/generated/cluster-controller/ctx.md index 6c7f1958cd1d9..17e422045b550 100644 --- a/doc/developer/generated/cluster-controller/ctx.md +++ b/doc/developer/generated/cluster-controller/ctx.md @@ -1,6 +1,6 @@ --- source: src/cluster-controller/src/ctx.rs -revision: 74f18a3354 +revision: bbe8b6977e --- # cluster-controller::ctx @@ -11,7 +11,7 @@ The boundary between the controller and its environment. Key types: -* `ClusterControllerCtx` (trait) — six async methods: `now`, `managed_cluster_ids`, `cluster_states`, `hydrated_replicas`, `has_hydratable_objects`, `apply`. Reads are batched to bound round-trips in a separate-task deployment. `hydrated_replicas` returns the subset of given replicas that have all current collections hydrated; it is only called when a strategy declares it needs hydration via `SignalRequest`. `has_hydratable_objects` returns whether a cluster has at least one dataflow-backed object (index, materialized view, ingestion source, or sink); it is only called when a strategy declares `hydratable_objects` in its `SignalRequest`. +* `ClusterControllerCtx` (trait) — six async methods: `now`, `managed_cluster_ids`, `cluster_states`, `hydrated_replicas`, `has_hydratable_objects`, `apply`. Reads are batched to bound round-trips in a separate-task deployment. `hydrated_replicas` returns the subset of given replicas that are online and have all current collections hydrated; it is only called when a strategy declares it needs hydration via `SignalRequest`. `has_hydratable_objects` returns whether a cluster has at least one dataflow-backed object (index, materialized view, ingestion source, or sink); it is only called when a strategy declares `hydratable_objects` in its `SignalRequest`. * `ClusterState` — durable config plus observed replicas of one managed cluster for one tick. Carries `cluster_id`, `size`, `replication_factor`, `availability_zones`, `logging`, optional `auto_scaling_policy`, `reconfiguration` and `burst` records, and `replicas`. Unmanaged clusters are not represented. * `ObservedReplica` — a replica that actually exists: `replica_id`, `name`, `shape` (optional; `None` for unmanaged-location replicas), `internal`, `billed_as`, `pending`. `ObservedReplica::owned_shape` returns the replica's shape only when the controller owns it: `INTERNAL`, `BILLED AS`, and pending replicas return `None` and are excluded from the desired/actual diff, though their names still block the name generator. * `StateWrite` — the durable mutations a strategy's `update_state` requests: cut-overs (`new_size`, `new_replication_factor`, `new_availability_zones`, `new_logging`) and record writes/clears (`reconfiguration`, `burst`). `None` fields are no-ops. diff --git a/doc/developer/generated/cluster-controller/strategy.md b/doc/developer/generated/cluster-controller/strategy.md index 19c89ab2b9483..bc88a43bda19a 100644 --- a/doc/developer/generated/cluster-controller/strategy.md +++ b/doc/developer/generated/cluster-controller/strategy.md @@ -1,6 +1,6 @@ --- source: src/cluster-controller/src/strategy.rs -revision: 74f18a3354 +revision: bbe8b6977e --- # cluster-controller::strategy @@ -20,7 +20,7 @@ Key types: * `DesiredReplica` — a replica slot a strategy desires this tick, characterized by a `ReplicaShape`. * `SignalRequest` — declares which live signals a strategy needs for a cluster this tick: `hydration: bool` (probe per-replica hydration) and `hydratable_objects: bool` (check whether the cluster has at least one dataflow-backed object). The controller unions requests across strategies and fetches only what is needed. * `ConfigSignals` — environment-wide dyncfg values latched by the kernel once per tick: `burst_enabled` (the break-glass flag for the hydration-burst strategy) and `default_burst_linger` (the system-default linger duration written into new burst records when the policy omits one). Not durable state, so never witness material. -* `LiveSignals` — the fulfilled live signals for one cluster: `hydrated_replicas` is the set of replica IDs (among controller-owned replicas) that have all current collections hydrated; `has_hydratable_objects` is whether the cluster has at least one dataflow-backed object. A signal not requested by any strategy is left at its empty default. +* `LiveSignals` — the fulfilled live signals for one cluster: `hydrated_replicas` is the set of replica IDs (among controller-owned replicas) that are online and have all current collections hydrated; `has_hydratable_objects` is whether the cluster has at least one dataflow-backed object. A signal not requested by any strategy is left at its empty default. * `BaselineStrategy` — the always-present implicit strategy. Desires `replication_factor` replicas at the cluster's realized shape. With only the baseline engaged the desired set equals the realized set, so a steady-state managed cluster reconciles to no decisions. The baseline is what makes other strategies purely additive. * `BASELINE_STRATEGY_NAME` — the audit-attribution string `"baseline"`. * `GracefulReconfigurationStrategy` — engaged whenever the durable `reconfiguration` record is in progress. Desires `target.replication_factor` replicas at the target shape in addition to the baseline's realized-shape replicas. Once enough target replicas are hydrated, `update_state` cuts over (advances the realized config, marks the record finalized). On a timeout, behavior is governed by the record's `on_timeout`: `Commit` cuts over to the un-hydrated target, `Rollback` (the default) marks the record timed out and stops desiring the target replicas. diff --git a/doc/developer/generated/environmentd/_crate.md b/doc/developer/generated/environmentd/_crate.md index 90fbcb731f7a2..83b31c90fccbf 100644 --- a/doc/developer/generated/environmentd/_crate.md +++ b/doc/developer/generated/environmentd/_crate.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/lib.rs -revision: 34effa9dc0 +revision: 4e2b6c0984 --- # environmentd diff --git a/doc/developer/generated/environmentd/http/mcp.md b/doc/developer/generated/environmentd/http/mcp.md index e9b70698eb580..0d61b11019996 100644 --- a/doc/developer/generated/environmentd/http/mcp.md +++ b/doc/developer/generated/environmentd/http/mcp.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/http/mcp.rs -revision: 780d92be42 +revision: 2736da1635 --- # environmentd::http::mcp @@ -10,7 +10,8 @@ Provides two endpoints: `/api/mcp/agent` (tools: `get_data_products`, `get_data_ Each endpoint is gated by a dynamic feature flag (`ENABLE_MCP_AGENT`, `ENABLE_MCP_DEVELOPER`, `ENABLE_MCP_AGENT_QUERY_TOOL`, `ENABLE_MCP_DEVELOPER_QUERY_TOOL`, `ENABLE_MCP_AGENT_READ_DATA_PRODUCT_TOOL`); `ENABLE_MCP_AGENT_READ_DATA_PRODUCT_TOOL` gates the `read_data_product` tool independently of the per-endpoint query tool flag. Response size is bounded by `MCP_MAX_RESPONSE_SIZE`. Request execution is bounded by `MCP_REQUEST_TIMEOUT`, a dynamic config that controls how long a tool call may run before a clean JSON-RPC timeout error is returned to the caller. Each request tags the session's `application_name` with `mz_mcp_agents` or `mz_mcp_developer` (via `set_default`, so a caller-supplied value still takes precedence) to make MCP-originated sessions visible in `mz_session_history` and `mz_statement_execution_history`. Implements MCP protocol version `2025-11-25`: the `initialize` response includes a `protocolVersion` field set to `MCP_PROTOCOL_VERSION`, tool definitions carry `title`, `annotations` (`ToolAnnotations` with `readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`), and `ToolContentResult` includes an `isError` field. -Both endpoints' `initialize` responses include usage instructions (via `endpoint_instructions`). The developer endpoint guides AI agents to use `mz_internal.mz_ontology_*` tables for discovering table schemas, join paths, and column names before writing queries; when `ENABLE_MCP_DEVELOPER_QUERY_TOOL` is on, the instructions also advertise the `query` tool. The agent endpoint instructs agents to prefer indexed objects (served from memory) over unindexed materialized views, and that `read_data_product` automatically routes the read to the cluster recorded in the data product catalog so indexes are used (provided the role has `USAGE` on that cluster; if not, the call fails with a `ClusterPrivilegeMissing` error rather than silently falling back to a slower path), and to set the `cluster` parameter only when intentionally targeting a different cluster (e.g. one with larger or more replicas). The agent endpoint also instructs agents to check the `hydration` object returned by `get_data_product_details` before reading: if `hydrated` is false and `replica_count > 0` the dataflow is warming up and a read would block until it catches up; if `replica_count` is 0 the cluster has no replicas and reading cannot make progress. +Both endpoints' `initialize` responses include usage instructions (via `endpoint_instructions`). The developer endpoint guides AI agents to use `mz_internal.mz_ontology_*` tables for discovering table schemas, join paths, and column names before writing queries; when `ENABLE_MCP_DEVELOPER_QUERY_TOOL` is on, the instructions also advertise the `query` tool. The agent endpoint instructs agents to prefer indexed objects (served from memory) over unindexed materialized views, and that `read_data_product` automatically routes the read to the cluster recorded in the data product catalog so indexes are used. The `cluster` field in `mz_mcp_data_products` is null when the session role lacks `USAGE` on the object's index/compute cluster; in that case `read_data_product` without a cluster override runs on the session's default cluster (safe: only materialized views appear this way and they serve from persist). Instructions also tell agents to set the `cluster` parameter only when intentionally targeting a different cluster (e.g. one with larger or more replicas). The agent endpoint also instructs agents to check the `hydration` object returned by `get_data_product_details` before reading: if `hydrated` is false and `replica_count > 0` the dataflow is warming up and a read would block until it catches up; if `replica_count` is 0 the cluster has no replicas and reading cannot make progress. +The developer endpoint's system-catalog query validation (`validate_system_catalog_query`) accepts unqualified `pg_catalog` names in addition to `mz_`-prefixed names; both are unambiguously system under the pinned `search_path` so a user-created view named `public.pg_class` cannot be reached via an unqualified reference. Both endpoints accept GET requests with a 405 Method Not Allowed response (via `handle_mcp_method_not_allowed`) and validate the `Origin` header against the CORS allowlist (injected as `Arc>` via Axum `Extension`) to prevent DNS rebinding attacks, rejecting origins not on the allowlist with 403 Forbidden. The CORS layer alone is insufficient because DNS rebinding causes the browser to treat the request as same-origin, bypassing preflight; the server-side check via `mz_http_util::origin_is_allowed` closes this gap. SQL parsing in MCP uses `parse_with_limit` for statements and `parse_item_name_with_limit` for item names; both enforce the 1 MB `MAX_STATEMENT_BATCH_SIZE` size check before allocating, preventing oversized inputs from being parsed. Enforces read-only SQL validation and AST-based system-table access control before executing queries; the developer endpoint allows SHOW and EXPLAIN statements without table references but rejects constant SELECT queries (e.g., `SELECT 1`) to prevent misuse for arbitrary computation. @@ -18,4 +19,4 @@ Enforces read-only SQL validation and AST-based system-table access control befo `execute_sql` delegates to the private `select_single_rows(results: Vec)` function, which returns the rows of the single row-returning statement in the response. `select_single_rows` surfaces the first error encountered; a second row-returning statement is an `Internal` error rather than a silently dropped result. The framing statements around a user query (`BEGIN`, `SET`, `COMMIT`) report `Ok` with no rows, so only the user's statement contributes rows. `McpResponse` is the top-level JSON-RPC 2.0 response envelope; `McpResponse::success(id, result)` builds a response carrying `result` and `McpResponse::error(id, error)` builds one carrying `error`, with exactly one of the two fields set. `McpRequestError` maps domain errors to standard JSON-RPC error codes. -Prometheus metrics are collected via `McpMetrics` (injected as an Axum `Extension`): request counts labeled by endpoint, JSON-RPC method, and status; tool call counts and durations labeled by endpoint and tool name. `ToolCallGuard` is an RAII guard that records tool call duration and status on drop, ensuring metrics are recorded even when a future is dropped early (e.g. by a timeout). +Prometheus metrics are collected via `McpMetrics` (injected as an Axum `Extension`): request counts labeled by endpoint, JSON-RPC method, and `McpCallStatus`; tool call counts and durations labeled by endpoint and tool name. `ToolCallGuard` is an RAII guard that records tool call duration and `McpCallStatus` on drop, ensuring metrics are recorded even when a future is dropped early (e.g. by a timeout). The `call_status` helper maps a handler result to `McpCallStatus::Ok` or `McpCallStatus::Error(e.error_type())`; timeout and endpoint-disabled outcomes are mapped directly at their decision points. diff --git a/doc/developer/generated/environmentd/http/mcp_metrics.md b/doc/developer/generated/environmentd/http/mcp_metrics.md index 6408a67117ec0..1d4eb6bc85738 100644 --- a/doc/developer/generated/environmentd/http/mcp_metrics.md +++ b/doc/developer/generated/environmentd/http/mcp_metrics.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/http/mcp_metrics.rs -revision: d137bfbf58 +revision: 2736da1635 --- # environmentd::http::mcp_metrics @@ -11,12 +11,16 @@ Defines Prometheus metrics for the MCP HTTP endpoints and the RAII guard used to `McpMetrics` holds three Prometheus collectors, all registered via `MetricsRegistry`: -* `mz_mcp_requests_total` (`IntCounterVec`) — total MCP requests, labeled by `endpoint_type` (`agent` or `developer`), `method` (JSON-RPC method name), and `status` (`ok` or an `McpRequestError` variant such as `ToolNotFound` or `DataProductNotFound`). +* `mz_mcp_requests_total` (`IntCounterVec`) — total MCP requests, labeled by `endpoint_type` (`agent` or `developer`), `method` (JSON-RPC method name), and `status` (a `McpCallStatus` variant: `ok`, `cancelled`, `timeout`, `endpoint_disabled`, or an `McpRequestError::error_type()` string such as `ToolNotFound`). * `mz_mcp_tool_calls_total` (`IntCounterVec`) — total `tools/call` invocations, labeled by `endpoint_type`, `tool_name`, and `status`. * `mz_mcp_tool_call_duration_seconds` (`HistogramVec`) — duration of `tools/call` invocations, labeled by `endpoint_type` and `tool_name`, using `histogram_seconds_buckets(0.000_128, 8.0)`. `McpMetrics` implements `Clone`; Prometheus collector handles are `Arc`-shared internally, so cloning is cheap. The struct is stored and passed as an Axum `Extension`. +## McpCallStatus + +`McpCallStatus` is a closed enum of outcomes recorded in the MCP `status` metric label. Using an enum rather than free-form strings at call sites pins the label cardinality and prevents typos from silently creating new label values. Variants: `Ok`, `Cancelled`, `Timeout`, `EndpointDisabled`, and `Error(&'static str)` (carrying the `McpRequestError::error_type()` string). `as_str()` converts each variant to its wire label string. + ## ToolCallGuard -`ToolCallGuard` is an RAII guard for a single `tools/call` invocation. On construction it starts a `HistogramTimer` for `mz_mcp_tool_call_duration_seconds`. On drop it increments `mz_mcp_tool_calls_total` with the current status label. The default status is `"cancelled"`, so if the surrounding future is dropped before completion (e.g. by `tokio::time::timeout`), the metric records as cancelled rather than being silently lost. Callers set the status to the actual outcome via `set_status` before the guard goes out of scope. +`ToolCallGuard` is an RAII guard for a single `tools/call` invocation. On construction it starts a `HistogramTimer` for `mz_mcp_tool_call_duration_seconds`. On drop it increments `mz_mcp_tool_calls_total` with the current `McpCallStatus`. The default status is `McpCallStatus::Cancelled`, so if the surrounding future is dropped before completion (e.g. by `tokio::time::timeout`), the metric records as cancelled rather than being silently lost. Callers set the status to the actual outcome via `set_status` before the guard goes out of scope. `McpMetrics::record_request` encapsulates the label ordering and `McpCallStatus` to `&str` conversion so call sites never touch raw label strings. diff --git a/doc/developer/generated/environmentd/telemetry.md b/doc/developer/generated/environmentd/telemetry.md index 2b91d7e7b7600..c1cb247bc9a3d 100644 --- a/doc/developer/generated/environmentd/telemetry.md +++ b/doc/developer/generated/environmentd/telemetry.md @@ -1,9 +1,10 @@ --- source: src/environmentd/src/telemetry.rs -revision: 61212870d6 +revision: 4e2b6c0984 --- # environmentd::telemetry Runs a periodic reporting loop that queries the adapter for environment statistics (active sources, sinks, views, clusters, etc.) and sends them to Segment via `group` (latest state) and `track` ("Environment Rolled Up" event with delta statistics). The loop starts via `start_reporting` and fires at a configurable `report_interval`; the first tick only sends traits, subsequent ticks also compute and report delta counters for DML operations. +`Config` carries a `license_key: ValidatedLicenseKey` field. Each reporting interval, the loop merges the license key's organization ID, environment ID, key ID, expiration timestamp, expired flag, and expiration behavior into the traits map before the Segment `group` and `track` calls. Expiry is recomputed against the wall clock each interval so a key that lapses while environmentd keeps running is reported as expired. diff --git a/doc/developer/generated/sql/plan/statement/ddl/connection.md b/doc/developer/generated/sql/plan/statement/ddl/connection.md index 75ed3d825e5a0..7d12ee2996a3c 100644 --- a/doc/developer/generated/sql/plan/statement/ddl/connection.md +++ b/doc/developer/generated/sql/plan/statement/ddl/connection.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/statement/ddl/connection.rs -revision: 5173c50671 +revision: 31fb870f20 --- # mz-sql::plan::statement::ddl::connection @@ -9,7 +9,7 @@ Contains DDL planning logic specific to `CONNECTION` objects: parsing `CREATE CO Uses `generate_extracted_config!` for `ConnectionOption` extraction and validates connection-type-specific constraints. The `build_tunnel_definition` function (also used by CSR, Postgres, MySQL, and SQL Server connection planning) constructs a `Tunnel` from optional SSH tunnel, AWS PrivateLink, and `KafkaMatchingBrokerRule` inputs: no options yields `Tunnel::Direct`; an SSH tunnel yields `Tunnel::Ssh`; a default `aws_privatelink` yields `Tunnel::AwsPrivatelink`; and a non-empty list of matching rules yields `Tunnel::AwsPrivatelinks` (rule-based PrivateLink, supported for Kafka only). Kafka connection planning calls `get_brokers_and_rules` to split `BROKERS` entries into static brokers and `MATCHING` rules, then passes the rules to `build_tunnel_definition`; use of matching rules requires the `enable_kafka_broker_matching_rules` feature flag and at least one static broker address. -`CreateConnectionType::GlueSchemaRegistry` is planned by resolving the `AwsConnection` reference and the `Registry` name string; planning is guarded by `ENABLE_GLUE_SCHEMA_REGISTRY` and rejects an empty registry name. +`CreateConnectionType::GlueSchemaRegistry` is planned by resolving the `AwsConnection` reference and the `Registry` name string; it rejects an empty registry name. `CreateConnectionType::Gcp` is planned by extracting the `SERVICE ACCOUNT KEY` secret option (required) and building a `GcpConnection { credentials_json }`. For `IcebergCatalogType::Rest`, the catalog auth is expressed via `IcebergCatalogAuth`: `OAuth { credential, scope }` when a `CREDENTIAL` is supplied, or `Gcp(GcpConnectionReference)` when a `GCP CONNECTION` is supplied (mutually exclusive; at least one is required). GCP auth is only valid with the BigLake Iceberg REST Catalog URI (`https://biglake.googleapis.com/iceberg/v1/restcatalog`). `IcebergCatalogType::S3TablesRest` does not support a GCP connection. The `get_gcp_connection_reference` helper resolves a `GCP CONNECTION` option to a `GcpConnectionReference`, erroring if the referenced item is not a GCP connection. diff --git a/doc/developer/generated/sql/plan/transform_ast.md b/doc/developer/generated/sql/plan/transform_ast.md index 0f2a8233a49c2..be60a60a0c6aa 100644 --- a/doc/developer/generated/sql/plan/transform_ast.md +++ b/doc/developer/generated/sql/plan/transform_ast.md @@ -1,9 +1,10 @@ --- source: src/sql/src/plan/transform_ast.rs -revision: ddc1ff8d2d +revision: 434f6b82c0 --- # mz-sql::plan::transform_ast Applies SQL-level rewrites to the AST before planning: `FuncRewriter` rewrites functions like `mod`, `nullif`, `avg`, variance/stddev, and others into canonical forms; `Desugarer` expands syntactic sugar (e.g. `EXISTS`, lateral joins, implicit `GROUP BY`, `VALUES` with subqueries). The public entry point `transform` applies both passes in sequence. +Variance and standard-deviation rewrites with `DISTINCT` split the `sum(DISTINCT x²)` component into two sign-class aggregates (`sum(DISTINCT CASE WHEN x >= 0 THEN x² END)` and `sum(DISTINCT CASE WHEN x < 0 THEN x² END)`) to ensure deduplication happens on `x`, not on `x²`. Squaring is not injective over all reals (e.g. `-2` and `2` both square to `4`), so a single `sum(DISTINCT x²)` would incorrectly collapse values that differ only in sign. diff --git a/doc/developer/generated/sql/rbac.md b/doc/developer/generated/sql/rbac.md index b5b93f6571b23..bc962ebd929a9 100644 --- a/doc/developer/generated/sql/rbac.md +++ b/doc/developer/generated/sql/rbac.md @@ -1,6 +1,6 @@ --- source: src/sql/src/rbac.rs -revision: e6fbabee58 +revision: 2ba315025c --- # mz-sql::rbac @@ -10,7 +10,7 @@ Implements role-based access control (RBAC) checks for SQL statements. `check_plan` accepts a `target_conn_role: Option` parameter that carries the authenticated role of the connection targeted by a `Plan::SideEffectingFunc` (specifically `PgCancelBackend`); this role is used directly in RBAC membership checks rather than being looked up lazily via a callback. System users (`mz_system`, `mz_support`) bypass most checks, and individual checks can be disabled via feature flags. `check_plan` accepts both `resolved_ids` (the statement's main dependency set) and a separate `sql_impl_resolved_ids` (IDs from SQL-implemented function bodies, which are implementation details rather than real statement dependencies). `check_restrict_to_user_objects` is called once for `sql_impl_resolved_ids` before the main RBAC requirements are generated, so that SQL-impl function body IDs are checked for `restrict_to_user_objects` without affecting regular dependency tracking. -`check_restrict_to_user_objects` runs when `session.restrict_to_user_objects()` is true: it iterates over the provided `resolved_ids`, and for each system-owned item rejects non-`Func`/non-`Type` catalog items unless their OID appears in `RESTRICT_TO_USER_OBJECTS_ALLOWED_OIDS` (currently `mz_mcp_data_products`, `mz_mcp_data_product_details`, and `mz_show_my_cluster_privileges`). `mz_show_my_cluster_privileges` is exempted so that `read_data_product` can check cluster USAGE without triggering the restriction. The error variant is `UnauthorizedError::RestrictedSystemObject`. +`check_restrict_to_user_objects` runs when `session.restrict_to_user_objects()` is true: it iterates over the provided `resolved_ids`, and for each system-owned item rejects non-`Func`/non-`Type` catalog items unless their OID appears in `RESTRICT_TO_USER_OBJECTS_ALLOWED_OIDS` (currently `mz_mcp_data_products`, `mz_mcp_data_product_details`, and `mz_show_my_cluster_privileges`). `mz_show_my_cluster_privileges` is exempted because `mz_mcp_data_products` and `mz_mcp_data_product_details` reference it to null the advertised cluster when the session role lacks USAGE on it; it uses `mz_session_role_memberships()` rather than a `has_cluster_privilege` body that would reference `mz_roles`. The error variant is `UnauthorizedError::RestrictedSystemObject`. `generate_read_privileges_inner` uses an iterative worklist traversal rather than recursion; view dependency chains are user-controlled and can be arbitrarily deep, so recursion risks a stack overflow. `ALTER ROLE ... SET restrict_to_user_objects` requires superuser: the `PlannedAlterRoleOption::Variable` match arm in `generate_rbac_requirements` enforces this with a case-insensitive name comparison. `Plan::CreateMaterializedView` in `generate_rbac_requirements` requires ownership of both `replace` (the `CREATE OR REPLACE` target, if set) and `materialized_view.replacement_target` (the `FOR ` replacement target, if set). Both are independent and optional; ownership of whichever are present is required, mirroring the ownership check for `ALTER ... APPLY REPLACEMENT` and `CREATE INDEX`. diff --git a/doc/developer/generated/sql/session/vars/definitions.md b/doc/developer/generated/sql/session/vars/definitions.md index 7ab27ea1309d4..1671f244e7509 100644 --- a/doc/developer/generated/sql/session/vars/definitions.md +++ b/doc/developer/generated/sql/session/vars/definitions.md @@ -1,6 +1,6 @@ --- source: src/sql/src/session/vars/definitions.rs -revision: a7b4b4f70f +revision: 31fb870f20 --- # mz-sql::session::vars::definitions @@ -11,5 +11,5 @@ The `feature_flags!` macro accepts an optional `scope: ,` field per flag e The `lazy_value!` and `value!` macros (from `polyfill`) are used extensively to express default values that cannot be computed at compile time. This file is the authoritative source of truth for which variables exist and their defaults. `RESTRICT_TO_USER_OBJECTS` is a read-only `bool` session variable (default `false`) that restricts queries from accessing system catalog objects; it is designed to be set only via `ALTER ROLE ... SET` by superusers. -Feature flags include `enable_repeat_row_non_negative` (guards the `repeat_row_non_negative` table function), `enable_storage_introspection_logs` (guards forwarding storage timely logging events into the compute introspection dataflow), `enable_kafka_broker_matching_rules` (guards `MATCHING` broker rules in `BROKERS` for Kafka PrivateLink connections), `enable_glue_schema_registry` (guards `CREATE CONNECTION ... TO AWS GLUE SCHEMA REGISTRY`; defaults to `true`), `enable_bounded_staleness_isolation` (guards the `bounded staleness ` transaction isolation level; defaults to `true`), `enable_fixed_correlated_cte_lowering` (gates the corrected HIR-to-MIR lowering path that uses CTE-aware branch keys when decorrelating CTEs referenced from nested correlated scopes; defaults to `true`), `enable_statement_arrival_logging` (when enabled, SQL frontends log incoming statements at info level as they arrive, before processing, with literals redacted; intended as an emergency diagnostic for incidents where statements crash the process before reaching the statement log; defaults to `false`), `unsafe_enable_unbounded_custom_type_resolution` (default `false`, `enable_for_item_parsing: true`; when enabled, custom type resolution proceeds without the nesting-depth and total-node limits that ordinarily bound resolution work), and `enable_auto_scaling_strategy` (guards the `AUTO SCALING STRATEGY` cluster option; default `true`, `enable_for_item_parsing: true`; `RESET (AUTO SCALING STRATEGY)` is intentionally not gated by this flag so clusters can shed an autoscaling policy after a flag rollback). +Feature flags include `enable_repeat_row_non_negative` (guards the `repeat_row_non_negative` table function), `enable_storage_introspection_logs` (guards forwarding storage timely logging events into the compute introspection dataflow), `enable_kafka_broker_matching_rules` (guards `MATCHING` broker rules in `BROKERS` for Kafka PrivateLink connections), `enable_bounded_staleness_isolation` (guards the `bounded staleness ` transaction isolation level; defaults to `true`), `enable_fixed_correlated_cte_lowering` (gates the corrected HIR-to-MIR lowering path that uses CTE-aware branch keys when decorrelating CTEs referenced from nested correlated scopes; defaults to `true`), `enable_statement_arrival_logging` (when enabled, SQL frontends log incoming statements at info level as they arrive, before processing, with literals redacted; intended as an emergency diagnostic for incidents where statements crash the process before reaching the statement log; defaults to `false`), `unsafe_enable_unbounded_custom_type_resolution` (default `false`, `enable_for_item_parsing: true`; when enabled, custom type resolution proceeds without the nesting-depth and total-node limits that ordinarily bound resolution work), and `enable_auto_scaling_strategy` (guards the `AUTO SCALING STRATEGY` cluster option; default `true`, `enable_for_item_parsing: true`; `RESET (AUTO SCALING STRATEGY)` is intentionally not gated by this flag so clusters can shed an autoscaling policy after a flag rollback). `DEFAULT_TIMESTAMP_INTERVAL`, `CLUSTER_CHECK_SCHEDULING_POLICIES_INTERVAL`, `SSH_CHECK_INTERVAL`, and `STORAGE_STATISTICS_INTERVAL` each carry a `NON_ZERO_DURATION` constraint, so setting any of them to zero raises an `InvalidParameterValue` error. diff --git a/doc/developer/generated/storage-client/controller.md b/doc/developer/generated/storage-client/controller.md index 5938302c445d7..e10af4e24e57a 100644 --- a/doc/developer/generated/storage-client/controller.md +++ b/doc/developer/generated/storage-client/controller.md @@ -1,6 +1,6 @@ --- source: src/storage-client/src/controller.rs -revision: 55e361b0a6 +revision: bbe8b6977e --- # storage-client::controller @@ -11,3 +11,4 @@ Provides key supporting types: `CollectionDescription`, `DataSource`, `ExportDes `StorageWriteOp` represents high-level write operations (`Append` and `Delete`) used to update differential introspection collections. `WallclockLagHistogramPeriod` represents a `[start, end)` time range for wallclock lag histogram bucketing. The `StorageTxn` trait abstracts durable metadata persistence for shard-to-collection mappings, unfinalized shard tracking, and the txn WAL shard. `remove_unfinalized_shards` removes entries from the unfinalized shard set without finalizing the underlying Persist shards; callers are responsible for reconciling the set against active collection metadata before calling it, since stale entries can refer to active collections. +`collections_hydrated_on_replicas` skips ingestions whose scheduled replica set is disjoint from the target replicas: a single-replica source stays scheduled on its current replica and cannot hydrate on a replica it is not scheduled on, so such ingestions must not count against the target's hydration readiness. Replica health (online/offline status) is a caller concern and is checked separately outside this method. diff --git a/doc/developer/generated/storage-controller/_crate.md b/doc/developer/generated/storage-controller/_crate.md index 612993543b19b..35646113e45bd 100644 --- a/doc/developer/generated/storage-controller/_crate.md +++ b/doc/developer/generated/storage-controller/_crate.md @@ -1,6 +1,6 @@ --- source: src/storage-controller/src/lib.rs -revision: 1de0314a65 +revision: bbe8b6977e --- # storage-controller diff --git a/doc/developer/generated/storage-controller/instance.md b/doc/developer/generated/storage-controller/instance.md index a026e3623e789..6efc156e71ac9 100644 --- a/doc/developer/generated/storage-controller/instance.md +++ b/doc/developer/generated/storage-controller/instance.md @@ -1,11 +1,12 @@ --- source: src/storage-controller/src/instance.rs -revision: 00cc513fa5 +revision: bbe8b6977e --- # storage-controller::instance Defines `Instance`, which manages communication with all replicas of one storage instance (cluster), and `Replica`, which handles the connection to a single replica over gRPC. `Instance` maintains per-object scheduling decisions — single-replica objects (most sinks and some sources) run on the lowest-ID replica, while multi-replica objects run on all replicas — and replays the command history to newly added or reconnected replicas via `CommandHistory`. +`Instance::active_replica_ids` is a shared helper that resolves an object ID to its scheduled replicas, returning `ActiveReplicas::Scheduled(&BTreeSet)` for objects with per-replica scheduling (ingestions and exports) or `ActiveReplicas::All` for objects without scheduling (tables, webhooks). `active_replicas` and `is_active_replica` project this result into their respective mutable-iterator and bool forms. The `collections_hydrated_on_replicas` check uses `active_replica_ids` to skip ingestions that are not scheduled on any of the target replicas: a single-replica source keeps running on its existing replica and cannot hydrate on a pending replica before cut-over, so it must not count against the target's hydration readiness. `Replica` runs a background `ReplicaTask` that connects (with retry) and runs a bidirectional message loop forwarding commands from the controller and responses back up. `ReplicaConfig` bundles the build info, network location, and gRPC parameters needed to connect. diff --git a/doc/developer/generated/storage-types/sources/_module.md b/doc/developer/generated/storage-types/sources/_module.md index 0c96dd481ce48..4ea8d779673b3 100644 --- a/doc/developer/generated/storage-types/sources/_module.md +++ b/doc/developer/generated/storage-types/sources/_module.md @@ -1,6 +1,6 @@ --- source: src/storage-types/src/sources.rs -revision: f2a5b6012b +revision: bbe8b6977e --- # storage-types::sources @@ -10,3 +10,4 @@ The `SourceConnection` and `SourceTimestamp` traits abstract over the five concr `SourceData` implements `mz_persist_types::Codec` using a custom columnar Arrow-based encoding that stores data rows and error rows in separate columns for efficient filter pushdown. Submodules `casts`, `encoding`, `envelope`, `kafka`, `load_generator`, `mysql`, `postgres`, and `sql_server` each define the connection-specific structs for their respective source types; `casts` defines `StorageScalarExpr` for source cast expressions. `SourceExportStatementDetails::Postgres` carries a `cast_oid_full_range: bool` field. Exports purified before the OID cast was widened decode this as `false` and use the legacy `i32`-range cast; newly purified exports set it to `true` and use the full `u32`-range cast. +For old-syntax sources, the `source_exports` field in `IngestionDescription` includes the primary source's ID in addition to explicit export IDs; callers that need only ingestion exports must filter it out. New-syntax sources (with source tables) list only their explicit exports in `source_exports`, so the primary collection ID is not present there. From 7a1c8b49ac67f3474ef1aedbc7135c375437a3ce Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Wed, 22 Jul 2026 07:00:38 +0000 Subject: [PATCH 3/6] doc: update generated developer documentation Automated daily refresh of doc/developer/generated/ to reflect source changes since each doc was last written. Co-Authored-By: Claude Opus 4.6 (1M context) --- doc/developer/generated/adapter/coord/_module.md | 4 +++- doc/developer/generated/adapter/coord/introspection.md | 3 ++- doc/developer/generated/adapter/coord/message_handler.md | 3 ++- doc/developer/generated/adapter/metrics.md | 2 +- doc/developer/generated/catalog/builtin/mz_internal.md | 2 +- doc/developer/generated/mz-deploy/cli/commands/dev.md | 2 +- doc/developer/generated/mz-deploy/client/validation.md | 3 ++- doc/developer/generated/mz-deploy/lsp/diagnostics.md | 3 ++- .../project/resolve/normalize/overlay_transformer.md | 3 ++- .../generated/mz-deploy/project/resolve/normalize/visitor.md | 2 +- 10 files changed, 17 insertions(+), 10 deletions(-) diff --git a/doc/developer/generated/adapter/coord/_module.md b/doc/developer/generated/adapter/coord/_module.md index 478e9701497d3..2a8b5a7f7ce91 100644 --- a/doc/developer/generated/adapter/coord/_module.md +++ b/doc/developer/generated/adapter/coord/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord.rs -revision: 55e361b0a6 +revision: 7f6c52776d --- # adapter::coord @@ -8,12 +8,14 @@ revision: 55e361b0a6 The core coordinator: `coord.rs` defines the `Coordinator` struct (the central state machine), `ExecuteContext` (the per-statement execution handle), the internal `Message` enum, and the `serve` / `Config` entry points. The coordinator owns the `Catalog`, active compute sinks, pending peeks, read-policy manager, timeline oracles, and all inter-subsystem handles (controller, storage collections, secrets, orchestrator). The `Coordinator` struct holds a `reconcile_now: Arc` field that wakes the cluster controller task to reconcile immediately after catalog transactions that change durable cluster state, rather than waiting out the tick interval. The file also defines `IdPool`, a pre-allocated pool of user `GlobalId` integers that amortizes per-DDL persist writes by reserving batches of IDs at once; the pool is owned by the coordinator and access is serialized through its single-threaded event loop. +`ArrangementSizeRecord` is a struct for rows destined for `mz_object_arrangement_size_history`, carrying `replica_id`, `object_id`, `size`, and `hydration_complete`; records are prepared off-thread by the arrangement sizes snapshot task and stamped with a collection timestamp on the coordinator loop at write time. Child modules partition the coordinator's responsibilities: `command_handler` handles external `Command` messages; `message_handler` handles internal async `Message` responses; `sequencer` executes SQL plans; `appends` manages table and builtin-table writes; `catalog_implications` derives and applies downstream effects from catalog state changes, including pushing replica-scoped dyncfg override layers to the compute controller via `push_replica_dyncfg_overrides`; `ddl` wraps catalog transactions; `peek` and `read_policy` manage query execution and compaction; `read_then_write` implements the read-then-write protocol for DML (INSERT, UPDATE, DELETE) including dependency validation; `group_sync` computes the diff between sync-managed and manually-granted role memberships for SSO group synchronization; `timestamp_selection` and `timeline` handle temporal reasoning; `catalog_serving` serves catalog snapshots; `info_metrics` owns the catalog `*_info` Prometheus series background task; `cluster_controller` scaffolds the cluster controller task that reconciles durable cluster state with the live controller; and supporting modules cover cluster scheduling, unified compute introspection subscribes, consistency checking, index management, and statement logging. The `Coordinator` struct holds a `scoped_frontend: Option>` field populated by `Command::InstallScopedSystemParameterFrontend`. `reconcile_scoped_system_parameters` persists the diff between the current durable working copy and the desired `ScopedParameters` via `Op::UpdateScopedSystemParameters`, which also updates the in-memory working copy and introspection relations. `scoped_overrides_create_op` evaluates scoped overrides for freshly-created clusters and replicas from explicit `ClusterEvalContext` / `ReplicaEvalContext` values and returns an `Op::UpdateScopedSystemParameters` to fold into the same transaction that creates them, so the committed diff drives the replica-scoped controller push before `create_replica`. `push_replica_dyncfg_overrides` propagates the replica-scoped overrides from the catalog working copy to the compute controller's per-replica dyncfg layer. `cluster_scoped_optimizer_overrides` retrieves the cluster-coherent `OptimizerFeatureOverrides` from the catalog working copy for use at plan time. The `Coordinator` struct holds a `catalog_info_metrics_registry: MetricsRegistry` field used to hand the metrics registry to the catalog info-metrics background task spawned at bootstrap. Bootstrap handles derived builtin storage collections (builtin MVs) separately: after registering input-less collections in dependency order, it bumps their sinces based on transitive dependency frontiers to satisfy as-of selection invariants. Bootstrap restores compaction policies for all MV version IDs (via `global_ids()`, not just the write ID) so that no version blocks capability propagation through the `primary` ownership chain and pins compaction. When applying replacement MVs, each `CollectionDescription` receives a `primary` field pointing to its predecessor's latest collection ID, chaining shard ownership from the oldest version through each replacement in order. Bootstrap also calls `push_replica_dyncfg_overrides` once before dataflows are rendered so that replica-scoped dyncfg flags take effect on the first configuration of each replica, and calls `spawn_cluster_controller_task` to start the cluster controller background task. The `clusters_caught_up_check` built during `serve` excludes both migrated MVs and new builtin MVs (and their transitive dependents) from the caught-up frontier check: migrated MV dataflows do not advance their write frontier in read-only mode, and new builtin MVs have no writer until the deployment promotes, so both sets would otherwise stall the check. The `Message` enum includes a `ClusterControllerRequest(cluster_controller::ClusterControllerRequest)` variant, carrying one pull/apply call from the cluster controller task to be answered on the main coordinator message loop. +`Message::ArrangementSizesWrite(Vec)` carries the records prepared by the off-thread arrangement sizes snapshot task back to the coordinator loop for stamping and appending to `mz_object_arrangement_size_history`. `ship_dataflow`, `try_ship_dataflow`, and `ship_dataflow_and_notice_builtin_table_updates` accept `DataflowDescription` (previously `DataflowDescription`). `ExecuteContextInner` carries a `response_barriers: Vec` field (Debug-ignored). `ExecuteContext::from_parts` is a convenience wrapper around `from_parts_with_response_barriers`, which accepts an explicit barrier list. `into_parts` returns the barrier list as a fifth element alongside `tx`, `internal_cmd_tx`, `session`, and `extra`; callers that repack an `ExecuteContext` must preserve and forward these barriers. `delay_response_until` appends a `BuiltinTableAppendCompletion` barrier; on retirement, the response is held until all barriers resolve. `ExplainTimestampStage` has four variants in order: `Optimize`, `RealTimeRecency`, `LinearizeTimestamp` (carries `ExplainTimestampLinearizeTimestamp`: validity, format, optimized_plan, cluster_id, source_ids, when, real_time_recency_ts), and `Finish` (carries `ExplainTimestampFinish`: validity, format, cluster_id, source_ids, when, real_time_recency_ts, timeline_context, oracle_read_ts). The `LinearizeTimestamp` stage reads the oracle timestamp off the coordinator loop and passes it forward as `oracle_read_ts` in `ExplainTimestampFinish`. diff --git a/doc/developer/generated/adapter/coord/introspection.md b/doc/developer/generated/adapter/coord/introspection.md index 3e43a002ae090..60953ddcfbb5d 100644 --- a/doc/developer/generated/adapter/coord/introspection.md +++ b/doc/developer/generated/adapter/coord/introspection.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/introspection.rs -revision: 277b33e9c0 +revision: 7f6c52776d --- # adapter::coord::introspection @@ -8,3 +8,4 @@ revision: 277b33e9c0 Implements unified compute introspection: the process of collecting introspection data exported by individual replicas through their logging indexes and writing that data, tagged with the respective replica ID, to unified storage collections. `install_introspection_subscribes` installs all defined introspection subscribes on a given replica (and `bootstrap_introspection_subscribes` calls it for all existing replicas during coordinator startup); `handle_introspection_subscribe_batch` processes each batch response, writing updates to the corresponding storage-managed collection and reinstalling failed subscribes on disconnect; `drop_introspection_subscribes` removes all subscribes installed on a replica before it is dropped. Each introspection subscribe is sequenced through a multi-stage pipeline (`OptimizeMir` → `TimestampOptimizeLir` → `Finish`) using the `sequence_staged` driver. The optimizer config for introspection subscribes includes cluster-coherent scoped overrides via `Coordinator::cluster_scoped_optimizer_overrides`. +`IntrospectionSubscribe` tracks a `first_data_at: Option` field recording when the subscribe first appended data to its target storage collection in the current process. Rows in that collection before that point may describe a previous environmentd process or a prior replica incarnation and must not be trusted. `invalidate_introspection_freshness(replica_id)` clears `first_data_at` for all subscribes targeting a given replica when a cluster event reports the replica offline or restarted. `fresh_introspection_replicas(introspection_type, margin)` returns the string replica IDs whose subscribe of the given type delivered data at least `margin` ago; consumers such as the arrangement sizes snapshot use this to exclude stale replicas. diff --git a/doc/developer/generated/adapter/coord/message_handler.md b/doc/developer/generated/adapter/coord/message_handler.md index cb8a8cdddde8f..690f6725bb0e8 100644 --- a/doc/developer/generated/adapter/coord/message_handler.md +++ b/doc/developer/generated/adapter/coord/message_handler.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/message_handler.rs -revision: 1c17d34993 +revision: 7f6c52776d --- # adapter::coord::message_handler @@ -10,3 +10,4 @@ Handles controller responses (compute peek results, subscribe batches, copy-to r This is the heart of the coordinator's reactive loop; every asynchronous response from the storage/compute layers arrives here. `storage_usage_update` obtains a write timestamp from the oracle, allocates a single durable batch id via `Catalog::allocate_storage_usage_id`, builds `BuiltinTableUpdate` rows via `pack_storage_usage_update` for each shard, and submits them via `builtin_table_update().execute()` without going through `catalog_transact_inner`. In read-only mode, `storage_usage_fetch` logs an info message and reschedules via `Message::StorageUsageSchedule` without performing any shard scan or writes. +`ARRANGEMENT_SIZES_FRESHNESS_MARGIN` (10 seconds) is the minimum age of a replica's introspection-subscribe delivery before `arrangement_sizes_snapshot` trusts its data; the margin covers the collection manager's batched write flush and the oracle read timestamp trailing the wall clock. `arrangement_sizes_snapshot` resolves catalog IDs on the main loop, then spawns a task that takes a read timestamp and snapshots `mz_object_arrangement_sizes_unified` and `mz_compute_hydration_times`; the task calls `arrangement_sizes_records` to prepare `Vec` and sends them back as `Message::ArrangementSizesWrite` (or reschedules directly if empty or on failure). `arrangement_sizes_write` receives the prepared records, re-validates replica freshness (cluster events between snapshot and write can invalidate previously-fresh replicas), obtains a write timestamp, packs and appends the rows to `mz_object_arrangement_size_history`, and reschedules the next collection. In read-only mode, `arrangement_sizes_snapshot` reschedules without spawning a task. diff --git a/doc/developer/generated/adapter/metrics.md b/doc/developer/generated/adapter/metrics.md index ba5434a726c3c..a6a3b3bed2f99 100644 --- a/doc/developer/generated/adapter/metrics.md +++ b/doc/developer/generated/adapter/metrics.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/metrics.rs -revision: dbd2c3fc06 +revision: 7f6c52776d --- # adapter::metrics diff --git a/doc/developer/generated/catalog/builtin/mz_internal.md b/doc/developer/generated/catalog/builtin/mz_internal.md index 92ff22d054016..6b65c95b1151e 100644 --- a/doc/developer/generated/catalog/builtin/mz_internal.md +++ b/doc/developer/generated/catalog/builtin/mz_internal.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/builtin/mz_internal.rs -revision: 2ba315025c +revision: 7f6c52776d --- # catalog::builtin::mz_internal diff --git a/doc/developer/generated/mz-deploy/cli/commands/dev.md b/doc/developer/generated/mz-deploy/cli/commands/dev.md index cfd37f342a120..685fd484e555d 100644 --- a/doc/developer/generated/mz-deploy/cli/commands/dev.md +++ b/doc/developer/generated/mz-deploy/cli/commands/dev.md @@ -1,6 +1,6 @@ --- source: src/mz-deploy/src/cli/commands/dev.rs -revision: 8ee3def844 +revision: 5cc5e835e7 --- # mz-deploy::cli::commands::dev diff --git a/doc/developer/generated/mz-deploy/client/validation.md b/doc/developer/generated/mz-deploy/client/validation.md index f0ac58c0c3bc3..7e279867dbcb8 100644 --- a/doc/developer/generated/mz-deploy/client/validation.md +++ b/doc/developer/generated/mz-deploy/client/validation.md @@ -1,8 +1,9 @@ --- source: src/mz-deploy/src/client/validation.rs -revision: a647094cc4 +revision: d8759e6fea --- # mz-deploy::client::validation Database validation operations. +System-catalog dependencies (those with no database component, such as `mz_catalog` schema objects) are excluded from external-dependency existence checks because they are always present and their 2-part name never matches the 3-part FQN the existence query builds. diff --git a/doc/developer/generated/mz-deploy/lsp/diagnostics.md b/doc/developer/generated/mz-deploy/lsp/diagnostics.md index e0bf3db442da3..6eb7f83bce302 100644 --- a/doc/developer/generated/mz-deploy/lsp/diagnostics.md +++ b/doc/developer/generated/mz-deploy/lsp/diagnostics.md @@ -1,8 +1,9 @@ --- source: src/mz-deploy/src/lsp/diagnostics.rs -revision: c3db2feba9 +revision: 24424b9efb --- # mz-deploy::lsp::diagnostics LSP-specific diagnostic emission. +The internal `position_to_offset` helper converts an LSP `Position` (line, character) to a byte offset in the rope, returning `None` if the position's line lies past the end of the document to prevent a panic on out-of-range access. diff --git a/doc/developer/generated/mz-deploy/project/resolve/normalize/overlay_transformer.md b/doc/developer/generated/mz-deploy/project/resolve/normalize/overlay_transformer.md index f7b9e5809ea91..95b90003c1b62 100644 --- a/doc/developer/generated/mz-deploy/project/resolve/normalize/overlay_transformer.md +++ b/doc/developer/generated/mz-deploy/project/resolve/normalize/overlay_transformer.md @@ -1,8 +1,9 @@ --- source: src/mz-deploy/src/project/resolve/normalize/overlay_transformer.rs -revision: a647094cc4 +revision: 5cc5e835e7 --- # mz-deploy::project::resolve::normalize::overlay_transformer Name transformation for `mz-deploy dev` overlay compilation. +`OverlayTransformer` applies a two-step rule: external references (database not in `in_project_databases`) are left verbatim; in-project references are rewritten to the overlay database (`__`) only when the fully-qualified object is in `overlay_objects: &BTreeSet`. Routing by object rather than by schema means a non-overlaid object that shares a schema with an overlaid one continues to resolve to its production address. diff --git a/doc/developer/generated/mz-deploy/project/resolve/normalize/visitor.md b/doc/developer/generated/mz-deploy/project/resolve/normalize/visitor.md index 15bc5d2f43765..9b71b767040b2 100644 --- a/doc/developer/generated/mz-deploy/project/resolve/normalize/visitor.md +++ b/doc/developer/generated/mz-deploy/project/resolve/normalize/visitor.md @@ -1,6 +1,6 @@ --- source: src/mz-deploy/src/project/resolve/normalize/visitor.rs -revision: a647094cc4 +revision: 5cc5e835e7 --- # mz-deploy::project::resolve::normalize::visitor From 0faba3b047dd6a0d992b71f13f56b60e88c9bc07 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 23 Jul 2026 07:01:45 +0000 Subject: [PATCH 4/6] doc: update generated developer documentation Automated daily refresh of doc/developer/generated/ to reflect source changes since each doc was last written. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../generated/adapter-types/cluster_state.md | 2 +- .../generated/adapter/catalog/_module.md | 3 +- .../generated/adapter/catalog/apply.md | 2 +- .../adapter/catalog/cluster_state.md | 2 +- .../generated/adapter/catalog/open.md | 2 +- .../generated/adapter/catalog/open/_module.md | 2 +- .../generated/adapter/catalog/transact.md | 2 +- .../generated/adapter/coord/_module.md | 2 +- .../generated/adapter/coord/appends.md | 3 +- .../adapter/coord/catalog_implications.md | 2 +- .../adapter/coord/cluster_controller.md | 2 +- doc/developer/generated/adapter/coord/ddl.md | 2 +- .../generated/adapter/coord/info_metrics.md | 2 +- .../adapter/coord/message_handler.md | 2 +- .../adapter/coord/sequencer/inner.md | 2 +- .../adapter/coord/sequencer/inner/_module.md | 2 +- .../adapter/coord/sequencer/inner/cluster.md | 2 +- .../coord/sequencer/inner/create_index.md | 2 +- .../inner/create_materialized_view.md | 2 +- doc/developer/generated/adapter/coord/sql.md | 2 +- .../adapter/coord/statement_logging.md | 2 +- .../generated/adapter/coord/timeline.md | 2 +- doc/developer/generated/adapter/metrics.md | 3 +- .../generated/adapter/optimize/_module.md | 2 +- doc/developer/generated/adapter/session.md | 2 +- doc/developer/generated/adapter/util.md | 2 +- .../generated/catalog-protos/_crate.md | 6 ++-- .../generated/catalog-protos/objects.md | 4 +-- .../generated/catalog/durable/_module.md | 2 +- .../generated/catalog/durable/error.md | 4 +-- .../generated/catalog/durable/initialize.md | 2 +- .../catalog/durable/objects/_module.md | 2 +- .../catalog/durable/objects/serialization.md | 2 +- .../catalog/durable/objects/state_update.md | 2 +- .../generated/catalog/durable/persist.md | 2 +- .../generated/catalog/durable/transaction.md | 2 +- .../catalog/durable/upgrade/_module.md | 7 +++-- .../generated/catalog/memory/objects.md | 2 +- .../generated/cluster-controller/_crate.md | 2 +- .../generated/cluster-controller/ctx.md | 2 +- .../generated/cluster-controller/strategy.md | 2 +- .../compute-client/controller/_module.md | 2 +- .../generated/compute-types/config.md | 2 +- .../generated/compute-types/dyncfgs.md | 2 +- .../generated/compute-types/explain/text.md | 2 +- .../generated/compute-types/plan/_module.md | 2 +- .../compute-types/plan/join/delta_join.md | 4 +-- .../generated/compute-types/plan/lowering.md | 2 +- .../generated/compute-types/plan/scalar.md | 2 +- .../generated/compute/render/columnar.md | 18 +++++++++++ .../compute/render/join/delta_join.md | 2 +- .../environmentd/deployment/preflight.md | 4 ++- .../generated/interchange/envelopes.md | 10 +++++-- .../mz-deploy/cli/commands/clusters.md | 10 +++++-- .../generated/mz-deploy/cli/commands/stage.md | 8 +++-- .../generated/mz-deploy/client/_module.md | 7 +++-- .../mz-deploy/client/auto_scaling.md | 20 +++++++++++++ .../generated/mz-deploy/client/connection.md | 8 ++++- .../mz-deploy/client/introspection.md | 8 ++++- .../generated/mz-deploy/client/models.md | 12 +++++++- .../mz-deploy/client/provisioning.md | 8 ++++- .../generated/mz-deploy/project/clusters.md | 10 ++++++- doc/developer/generated/ore/_crate.md | 4 ++- doc/developer/generated/ore/cgroup.md | 2 +- doc/developer/generated/ore/memory.md | 12 ++++++++ doc/developer/generated/ore/pool/_module.md | 30 +++++++++++++++++++ doc/developer/generated/ore/pool/extent.md | 16 ++++++++++ doc/developer/generated/ore/pool/region.md | 19 ++++++++++++ .../generated/persist-client/_crate.md | 2 +- doc/developer/generated/repr/optimize.md | 2 +- .../sql-parser/ast/defs/statement.md | 2 +- doc/developer/generated/sql-parser/parser.md | 2 +- doc/developer/generated/sql/plan.md | 6 ++-- doc/developer/generated/sql/plan/_module.md | 4 +-- doc/developer/generated/sql/plan/lowering.md | 2 +- .../generated/sql/plan/lowering/_module.md | 2 +- .../generated/sql/plan/statement/ddl.md | 2 +- .../sql/plan/statement/ddl/_module.md | 2 +- .../generated/sql/plan/statement/dml.md | 2 +- .../generated/sql/plan/transform_hir.md | 2 +- .../generated/sql/session/vars/definitions.md | 2 +- .../generated/storage-client/controller.md | 2 +- .../generated/storage-controller/_crate.md | 2 +- .../persist_handles/_module.md | 4 +-- .../persist_handles/read_only_table_worker.md | 2 +- 85 files changed, 271 insertions(+), 95 deletions(-) create mode 100644 doc/developer/generated/compute/render/columnar.md create mode 100644 doc/developer/generated/mz-deploy/client/auto_scaling.md create mode 100644 doc/developer/generated/ore/memory.md create mode 100644 doc/developer/generated/ore/pool/_module.md create mode 100644 doc/developer/generated/ore/pool/extent.md create mode 100644 doc/developer/generated/ore/pool/region.md diff --git a/doc/developer/generated/adapter-types/cluster_state.md b/doc/developer/generated/adapter-types/cluster_state.md index 056bc75a62a03..c07783cf8510c 100644 --- a/doc/developer/generated/adapter-types/cluster_state.md +++ b/doc/developer/generated/adapter-types/cluster_state.md @@ -1,6 +1,6 @@ --- source: src/adapter-types/src/cluster_state.rs -revision: 74f18a3354 +revision: fca741734d --- # adapter-types::cluster_state diff --git a/doc/developer/generated/adapter/catalog/_module.md b/doc/developer/generated/adapter/catalog/_module.md index 203d21cf73b6c..f05a1c38f46d6 100644 --- a/doc/developer/generated/adapter/catalog/_module.md +++ b/doc/developer/generated/adapter/catalog/_module.md @@ -1,12 +1,13 @@ --- source: src/adapter/src/catalog.rs -revision: 605cc6fd1a +revision: a60edac7f1 --- # adapter::catalog The coordinator's catalog layer: wraps the `mz_catalog` crate's durable store and in-memory objects with the additional adapter-specific logic needed to drive DDL and serve catalog queries. `Catalog` is the top-level struct exposing the full catalog API: it holds a `CatalogState` (in-memory view), an `ExpressionCacheHandle`, a `transient_revision` counter, a `shared_transient_revision: Arc` shared across all clones, and delegates persistence to the durable store. `Catalog::transient_revision_is_current` compares the clone's frozen revision against the shared latest, letting a snapshot holder detect staleness from off-thread without a Coordinator round-trip. +`CatalogUpperHandle` is a handle for advancing the durable catalog upper off the coordinator loop; its `advance_upper` method advances the upper to at least a given timestamp. Child modules divide responsibilities: `state` owns the in-memory data structures and `SessionCatalog` impl; `transact` executes atomic DDL operations; `apply` reconciles durable diffs into in-memory state; `open` bootstraps the catalog at startup; `migrate` rewrites stored SQL for syntax changes; `builtin_table_updates` maintains system-table row diffs; `consistency` provides invariant checking; `timeline` resolves timeline contexts for timestamp selection; and `cluster_state` projects live cluster config into plain-data `ExpectedClusterState` structs for conditional catalog writes. Re-exports include `InjectedAuditEvent` from `transact`. Builtin materialized views are handled alongside tables and views in descriptor tests. `is_reserved_role_name` returns true for names that match `is_reserved_name`, `is_public_role`, or the `RESERVED_ROLE_SPECIFICATION_NAMES` list. `RESERVED_ROLE_SPECIFICATION_NAMES` contains the five lowercase PostgreSQL role-specification keywords (`current_user`, `current_role`, `session_user`, `user`, `none`) that would be ambiguous in statements like `GRANT ... TO CURRENT_USER`; only the lowercase spellings are reserved, matching what unquoted identifiers normalize to. diff --git a/doc/developer/generated/adapter/catalog/apply.md b/doc/developer/generated/adapter/catalog/apply.md index b00b1d93dd36a..b888012aea3f3 100644 --- a/doc/developer/generated/adapter/catalog/apply.md +++ b/doc/developer/generated/adapter/catalog/apply.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/apply.rs -revision: 584bb9030c +revision: fca741734d --- # adapter::catalog::apply diff --git a/doc/developer/generated/adapter/catalog/cluster_state.md b/doc/developer/generated/adapter/catalog/cluster_state.md index e3de982434d58..467d6c454065b 100644 --- a/doc/developer/generated/adapter/catalog/cluster_state.md +++ b/doc/developer/generated/adapter/catalog/cluster_state.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/cluster_state.rs -revision: 74f18a3354 +revision: fca741734d --- # adapter::catalog::cluster_state diff --git a/doc/developer/generated/adapter/catalog/open.md b/doc/developer/generated/adapter/catalog/open.md index 85bfdf20a659d..2245ee04b37f8 100644 --- a/doc/developer/generated/adapter/catalog/open.md +++ b/doc/developer/generated/adapter/catalog/open.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/open.rs -revision: dbd2c3fc06 +revision: fca741734d --- # adapter::catalog::open diff --git a/doc/developer/generated/adapter/catalog/open/_module.md b/doc/developer/generated/adapter/catalog/open/_module.md index 72f0feb241982..b59913fd7d8fb 100644 --- a/doc/developer/generated/adapter/catalog/open/_module.md +++ b/doc/developer/generated/adapter/catalog/open/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/open.rs -revision: dbd2c3fc06 +revision: fca741734d --- # adapter::catalog::open diff --git a/doc/developer/generated/adapter/catalog/transact.md b/doc/developer/generated/adapter/catalog/transact.md index 07f611759471a..57bc40e33b455 100644 --- a/doc/developer/generated/adapter/catalog/transact.md +++ b/doc/developer/generated/adapter/catalog/transact.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/transact.rs -revision: 74f18a3354 +revision: a60edac7f1 --- # adapter::catalog::transact diff --git a/doc/developer/generated/adapter/coord/_module.md b/doc/developer/generated/adapter/coord/_module.md index 2a8b5a7f7ce91..950f9fbe4bf45 100644 --- a/doc/developer/generated/adapter/coord/_module.md +++ b/doc/developer/generated/adapter/coord/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord.rs -revision: 7f6c52776d +revision: a60edac7f1 --- # adapter::coord diff --git a/doc/developer/generated/adapter/coord/appends.md b/doc/developer/generated/adapter/coord/appends.md index 95528aad1d7e0..a131d1e4d0d10 100644 --- a/doc/developer/generated/adapter/coord/appends.md +++ b/doc/developer/generated/adapter/coord/appends.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/appends.rs -revision: 1c17d34993 +revision: a60edac7f1 --- # adapter::coord::appends @@ -10,3 +10,4 @@ Implements all append operations executed by the coordinator: group-commit of us `BuiltinTableAppendNotify` is a pinned future that resolves when a builtin-table write completes. `BuiltinTableAppendCompletion` is a completion handle wrapping a `BuiltinTableAppendNotify`; callers obtain one to use as a response barrier and convert it to a notify via `into_notify`. `GroupCommitWriteLocks` manages the locking primitives used to serialise group commits. `group_commit` always performs the append call even when there are no user writes, relying on the storage layer to periodically advance the upper of all tables; it does not iterate catalog entries to inject empty advancement entries per table. `UserWriteResponder` is an enum with a single `Session(PendingTxn)` variant that wraps the per-session `PendingTxn` for `PendingWriteTxn::User`; the `User` variant's `pending_txn` field is replaced by `responder: UserWriteResponder`. The write timestamp for `User` writes is picked by the oracle during group commit; the write lock is either handed off from the submitting session (`write_locks: Some(..)`) or acquired during group commit (`write_locks: None`). +`GroupCommitter` serializes runtime txns-shard writes off the coordinator loop via `TableWriteCmd` variants: `GroupCommit`, `Register`, and `Forget`. `DeferredOp` represents an operation awaiting a resource; `DeferredOp::Plan` wraps a `DeferredPlan` that must uniquely hold write locks (e.g. UPDATE), while `DeferredOp::Write` wraps a `DeferredWrite` for blind inserts that can be optimistically retried. `GroupCommitRequest` accumulates appends, responses, statement-logging IDs, notifies, write locks, and permits for a single commit batch. diff --git a/doc/developer/generated/adapter/coord/catalog_implications.md b/doc/developer/generated/adapter/coord/catalog_implications.md index 8708da0f67d4d..db867c6f56458 100644 --- a/doc/developer/generated/adapter/coord/catalog_implications.md +++ b/doc/developer/generated/adapter/coord/catalog_implications.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/catalog_implications.rs -revision: 1c17d34993 +revision: a60edac7f1 --- # adapter::coord::catalog_implications diff --git a/doc/developer/generated/adapter/coord/cluster_controller.md b/doc/developer/generated/adapter/coord/cluster_controller.md index 22fb8e70eadd9..e093286d800a3 100644 --- a/doc/developer/generated/adapter/coord/cluster_controller.md +++ b/doc/developer/generated/adapter/coord/cluster_controller.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/cluster_controller.rs -revision: bbe8b6977e +revision: a60edac7f1 --- # adapter::coord::cluster_controller diff --git a/doc/developer/generated/adapter/coord/ddl.md b/doc/developer/generated/adapter/coord/ddl.md index a63b54ca47548..07fdba1e71768 100644 --- a/doc/developer/generated/adapter/coord/ddl.md +++ b/doc/developer/generated/adapter/coord/ddl.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/ddl.rs -revision: 8598d82c1c +revision: a60edac7f1 --- # adapter::coord::ddl diff --git a/doc/developer/generated/adapter/coord/info_metrics.md b/doc/developer/generated/adapter/coord/info_metrics.md index d1c520936ff1d..e3db10b6f70ef 100644 --- a/doc/developer/generated/adapter/coord/info_metrics.md +++ b/doc/developer/generated/adapter/coord/info_metrics.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/info_metrics.rs -revision: dbd2c3fc06 +revision: fca741734d --- # adapter::coord::info_metrics diff --git a/doc/developer/generated/adapter/coord/message_handler.md b/doc/developer/generated/adapter/coord/message_handler.md index 690f6725bb0e8..20baf071f7374 100644 --- a/doc/developer/generated/adapter/coord/message_handler.md +++ b/doc/developer/generated/adapter/coord/message_handler.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/message_handler.rs -revision: 7f6c52776d +revision: a60edac7f1 --- # adapter::coord::message_handler diff --git a/doc/developer/generated/adapter/coord/sequencer/inner.md b/doc/developer/generated/adapter/coord/sequencer/inner.md index 48edb8174b7f5..7c7354e93427d 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner.rs -revision: 3713eed996 +revision: a60edac7f1 --- # adapter::coord::sequencer::inner diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/_module.md b/doc/developer/generated/adapter/coord/sequencer/inner/_module.md index 03826fe7e747f..78c605e7cf6a6 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/_module.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner.rs -revision: 3713eed996 +revision: a60edac7f1 --- # adapter::coord::sequencer::inner diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md index 34c92564ebaae..06793ec70cc28 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/cluster.rs -revision: bbe8b6977e +revision: a60edac7f1 --- # adapter::coord::sequencer::inner::cluster diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/create_index.md b/doc/developer/generated/adapter/coord/sequencer/inner/create_index.md index b3d4995ac090f..b20f3f0dbfa4b 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/create_index.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/create_index.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/create_index.rs -revision: 277b33e9c0 +revision: a60edac7f1 --- # adapter::coord::sequencer::inner::create_index diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/create_materialized_view.md b/doc/developer/generated/adapter/coord/sequencer/inner/create_materialized_view.md index 10d3624083720..c235755f4efc4 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/create_materialized_view.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/create_materialized_view.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/create_materialized_view.rs -revision: 277b33e9c0 +revision: a60edac7f1 --- # adapter::coord::sequencer::inner::create_materialized_view diff --git a/doc/developer/generated/adapter/coord/sql.md b/doc/developer/generated/adapter/coord/sql.md index 8337f6154c58a..17a8d5f78d43b 100644 --- a/doc/developer/generated/adapter/coord/sql.md +++ b/doc/developer/generated/adapter/coord/sql.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sql.rs -revision: 1c17d34993 +revision: a60edac7f1 --- # adapter::coord::sql diff --git a/doc/developer/generated/adapter/coord/statement_logging.md b/doc/developer/generated/adapter/coord/statement_logging.md index 8ec44d8840567..68c2ecd294ba5 100644 --- a/doc/developer/generated/adapter/coord/statement_logging.md +++ b/doc/developer/generated/adapter/coord/statement_logging.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/statement_logging.rs -revision: 770c31e9f8 +revision: a60edac7f1 --- # adapter::coord::statement_logging diff --git a/doc/developer/generated/adapter/coord/timeline.md b/doc/developer/generated/adapter/coord/timeline.md index 58bce3b9aaa0a..52d2b85a2426e 100644 --- a/doc/developer/generated/adapter/coord/timeline.md +++ b/doc/developer/generated/adapter/coord/timeline.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/timeline.rs -revision: 4002e6039c +revision: a60edac7f1 --- # adapter::coord::timeline diff --git a/doc/developer/generated/adapter/metrics.md b/doc/developer/generated/adapter/metrics.md index a6a3b3bed2f99..f61b9ec8ce7bc 100644 --- a/doc/developer/generated/adapter/metrics.md +++ b/doc/developer/generated/adapter/metrics.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/metrics.rs -revision: 7f6c52776d +revision: a60edac7f1 --- # adapter::metrics @@ -10,5 +10,6 @@ Registers and vends all Prometheus metrics for the adapter and coordinator. `Metrics` includes a `timestamp_difference_for_bounded_staleness_ms` histogram (per-compute-instance label) that records how much older bounded-staleness timestamps are compared to serializable, measuring the actual staleness incurred. `SessionMetrics` exposes this via `timestamp_difference_for_bounded_staleness_ms`. The `mz_time_to_first_row_seconds` histogram carries an `application_name` label in addition to `instance_id`, `isolation_level`, and `strategy`. `Metrics` tracks catalog snapshot cache behavior via `catalog_snapshot_seconds` (a `HistogramVec` labeled by `context`, observed only on cache misses) and `catalog_snapshot_cache` (an `IntCounterVec` labeled by `context` and `result`, counting hits and misses). `catalog_arc_strong_count` and `catalog_arc_weak_count` are `UIntGauge` metrics tracking the number of strong and weak references to the current catalog snapshot `Arc`, respectively. +`Metrics` includes `catalog_transact_seconds` (a `HistogramVec` labeled by `method`) for timing catalog transact methods, `apply_catalog_implications_seconds` for timing catalog implication application, and `group_commit_catalog_upper_seconds` for timing catalog shard upper advances during group commits and table register/forget operations. Several public metrics carry `MetricTag` annotations for categorization: `mz_query_total`, `mz_active_sessions`, `mz_active_subscribes`, and `mz_adapter_commands` carry `MetricTag::Environment`. Helper functions `session_type_label_value`, `statement_type_label_value`, and `subscribe_output_label_value` produce the label strings used for partitioning these metrics. diff --git a/doc/developer/generated/adapter/optimize/_module.md b/doc/developer/generated/adapter/optimize/_module.md index 07fe83f29c978..f3927829ebbc5 100644 --- a/doc/developer/generated/adapter/optimize/_module.md +++ b/doc/developer/generated/adapter/optimize/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/optimize.rs -revision: 3d7eb1c1da +revision: b6b85cfcef --- # adapter::optimize diff --git a/doc/developer/generated/adapter/session.md b/doc/developer/generated/adapter/session.md index 5b25e809610c4..dd613841c15f0 100644 --- a/doc/developer/generated/adapter/session.md +++ b/doc/developer/generated/adapter/session.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/session.rs -revision: fd5141bb8c +revision: a60edac7f1 --- # adapter::session diff --git a/doc/developer/generated/adapter/util.md b/doc/developer/generated/adapter/util.md index 1ab224ca4a1e7..defe7db551055 100644 --- a/doc/developer/generated/adapter/util.md +++ b/doc/developer/generated/adapter/util.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/util.rs -revision: 4002e6039c +revision: a60edac7f1 --- # adapter::util diff --git a/doc/developer/generated/catalog-protos/_crate.md b/doc/developer/generated/catalog-protos/_crate.md index c774cceca0bf2..cf855b97b2361 100644 --- a/doc/developer/generated/catalog-protos/_crate.md +++ b/doc/developer/generated/catalog-protos/_crate.md @@ -1,12 +1,12 @@ --- source: src/catalog-protos/src/lib.rs -revision: 8598d82c1c +revision: fca741734d --- # mz-catalog-protos Provides all Rust types durably persisted in the Materialize catalog, along with `RustType` conversion impls bridging those types to protobuf. -The crate exposes the current schema as `objects` (currently v89) plus frozen snapshots `objects_v74` through `objects_v89` used for migrations. -`CATALOG_VERSION` (89) and `MIN_CATALOG_VERSION` (74) constants bound the supported migration range; the build script validates file hashes to prevent accidental mutation of snapshots. +The crate exposes the current schema as `objects` (currently v90) plus frozen snapshots `objects_v74` through `objects_v90` used for migrations. +`CATALOG_VERSION` (90) and `MIN_CATALOG_VERSION` (74) constants bound the supported migration range; the build script validates file hashes to prevent accidental mutation of snapshots. The crate has an optional `proptest` feature; `derive(Arbitrary)` on catalog types is compiled only when the `test` cfg or the `proptest` feature is enabled. Key dependencies are `mz-proto`, `mz-repr`, `mz-sql`, `mz-audit-log`, `mz-compute-types`, and `mz-storage-types`; the primary consumer is `mz-catalog`. diff --git a/doc/developer/generated/catalog-protos/objects.md b/doc/developer/generated/catalog-protos/objects.md index 89f5c9c83876e..c41de25cfb3b3 100644 --- a/doc/developer/generated/catalog-protos/objects.md +++ b/doc/developer/generated/catalog-protos/objects.md @@ -1,11 +1,11 @@ --- source: src/catalog-protos/src/objects.rs -revision: 6c07c975be +revision: fca741734d --- # mz-catalog-protos::objects -Defines the current (v89) set of Rust structs and enums that represent all durably persisted catalog objects, generated from protobuf definitions. +Defines the current (v90) set of Rust structs and enums that represent all durably persisted catalog objects, generated from protobuf definitions. This file is the canonical snapshot of the current catalog schema; `objects_v.rs` files are frozen snapshots used as migration sources. Key types include `ConfigKey`, `ConfigValue`, `SettingKey`, `SettingValue`, `IdAllocKey`, `RoleId`, `DatabaseId`, `SchemaId`, `AutoProvisionSource`, `RoleAttributes`, and many more covering every catalog entity. Scoped system configuration is represented by `ClusterSystemConfigurationKey`/`ClusterSystemConfigurationValue` (keyed by `ClusterId` and parameter name) and `ReplicaSystemConfigurationKey`/`ReplicaSystemConfigurationValue` (keyed by `ReplicaId` and parameter name), along with their corresponding `ClusterSystemConfiguration` and `ReplicaSystemConfiguration` wrapper structs and variants in `StateUpdateKind`. The `ManagedCluster` struct carries durable cluster autoscaling state: `auto_scaling_strategy` (`Option`), `reconfiguration` (`Option`), and `burst` (`Option`). `AutoScalingStrategy` holds an optional `OnHydration` sub-policy. `ReconfigurationState` records an in-flight graceful reconfiguration with a `ReconfigurationTarget`, deadline, and `OnTimeoutAction`. `BurstState` records an active hydration burst. The managed `ReplicaLocation` (`ManagedLocation`) stores `availability_zones` as a list rather than a single optional zone. The `audit_log_event_v1` submodule includes `CreateRoleV1`, `AlterAddColumnV1`, `AlterSourceTimestampIntervalV1`, `AlterClusterReconfigurationV1` (with `ReconfigurationLifecycleV1` variants `Started`, `Finalized`, `TimedOut`, `Cancelled`, `ResourceExhausted` and `ClusterReplicaLoggingV1`), and `ClusterHydrationBurstV1` (with `HydrationBurstLifecycleV1` and `BurstFinishCauseV1` variants `LingerElapsed`, `NoLongerWarranted`) for audit-logging corresponding catalog events. `CreateOrDropClusterReplicaReasonV1` carries reasons `Manual`, `Schedule`, `System`, `Reconfiguration`, `HydrationBurst`, and `Retired`. diff --git a/doc/developer/generated/catalog/durable/_module.md b/doc/developer/generated/catalog/durable/_module.md index 0b1fcaa95f010..22efee414577b 100644 --- a/doc/developer/generated/catalog/durable/_module.md +++ b/doc/developer/generated/catalog/durable/_module.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable.rs -revision: 8598d82c1c +revision: a60edac7f1 --- # catalog::durable diff --git a/doc/developer/generated/catalog/durable/error.md b/doc/developer/generated/catalog/durable/error.md index 35ffdd431b882..918b87fe8df28 100644 --- a/doc/developer/generated/catalog/durable/error.md +++ b/doc/developer/generated/catalog/durable/error.md @@ -1,11 +1,11 @@ --- source: src/catalog/src/durable/error.rs -revision: 00cc513fa5 +revision: a60edac7f1 --- # catalog::durable::error Defines the error hierarchy for durable catalog operations. `CatalogError` wraps either a `SqlCatalogError` (logical catalog errors) or a `DurableCatalogError` (storage-level errors). -`DurableCatalogError` covers fencing (`FenceError`), incompatible data versions (`IncompatibleDataVersion`, `IncompatiblePersistVersion`), uninitialized catalog, not-writable catalog, protobuf deserialization errors, duplicate key, uniqueness violations, storage errors, and internal errors. The method `can_recover_with_write_mode` reports whether the error can be recovered by reopening the catalog in writable mode. +`DurableCatalogError` covers fencing (`FenceError`), incompatible data versions (`IncompatibleDataVersion`, `IncompatiblePersistVersion`), uninitialized catalog, not-writable catalog, dry-run transaction commit attempts (`DryRunTransaction`), protobuf deserialization errors, duplicate key, uniqueness violations, storage errors, catalog out-of-sync (`CatalogOutOfSync`, reported when the durable catalog has advanced past what this process has applied), and internal errors. The method `can_recover_with_write_mode` reports whether the error can be recovered by reopening the catalog in writable mode. `FenceError` has three variants: `DeployGeneration` (fenced by a newer deployment generation), `Epoch` (fenced by a newer epoch), and `MigrationUpper` (fenced during 0dt builtin table migration). The enum is ordered from most to least informative. diff --git a/doc/developer/generated/catalog/durable/initialize.md b/doc/developer/generated/catalog/durable/initialize.md index b31009f9615c9..9eae6e45d457c 100644 --- a/doc/developer/generated/catalog/durable/initialize.md +++ b/doc/developer/generated/catalog/durable/initialize.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/initialize.rs -revision: 80f8711523 +revision: fca741734d --- # catalog::durable::initialize diff --git a/doc/developer/generated/catalog/durable/objects/_module.md b/doc/developer/generated/catalog/durable/objects/_module.md index 973b88aba6087..95a1881d1ff7e 100644 --- a/doc/developer/generated/catalog/durable/objects/_module.md +++ b/doc/developer/generated/catalog/durable/objects/_module.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/objects.rs -revision: 8598d82c1c +revision: fca741734d --- # catalog::durable::objects diff --git a/doc/developer/generated/catalog/durable/objects/serialization.md b/doc/developer/generated/catalog/durable/objects/serialization.md index 27093c4163ac3..68494502b8c76 100644 --- a/doc/developer/generated/catalog/durable/objects/serialization.md +++ b/doc/developer/generated/catalog/durable/objects/serialization.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/objects/serialization.rs -revision: 8598d82c1c +revision: fca741734d --- # catalog::durable::objects::serialization diff --git a/doc/developer/generated/catalog/durable/objects/state_update.md b/doc/developer/generated/catalog/durable/objects/state_update.md index 43f50ce421e2b..94244394f21ac 100644 --- a/doc/developer/generated/catalog/durable/objects/state_update.md +++ b/doc/developer/generated/catalog/durable/objects/state_update.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/objects/state_update.rs -revision: 584bb9030c +revision: a60edac7f1 --- # catalog::durable::objects::state_update diff --git a/doc/developer/generated/catalog/durable/persist.md b/doc/developer/generated/catalog/durable/persist.md index 27c2e74a537af..7568e98eb38dd 100644 --- a/doc/developer/generated/catalog/durable/persist.md +++ b/doc/developer/generated/catalog/durable/persist.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/persist.rs -revision: 584bb9030c +revision: a60edac7f1 --- # catalog::durable::persist diff --git a/doc/developer/generated/catalog/durable/transaction.md b/doc/developer/generated/catalog/durable/transaction.md index 27791c578e4d0..99814634473af 100644 --- a/doc/developer/generated/catalog/durable/transaction.md +++ b/doc/developer/generated/catalog/durable/transaction.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/durable/transaction.rs -revision: 55e361b0a6 +revision: a60edac7f1 --- # catalog::durable::transaction diff --git a/doc/developer/generated/catalog/durable/upgrade/_module.md b/doc/developer/generated/catalog/durable/upgrade/_module.md index 5bb24ebf235d6..91b31eb2ec829 100644 --- a/doc/developer/generated/catalog/durable/upgrade/_module.md +++ b/doc/developer/generated/catalog/durable/upgrade/_module.md @@ -1,14 +1,14 @@ --- source: src/catalog/src/durable/upgrade.rs -revision: 8598d82c1c +revision: fca741734d --- # catalog::durable::upgrade Orchestrates catalog schema migrations by replaying version-specific upgrade functions over a sequence of protobuf snapshots. -`CATALOG_VERSION` is the current schema version (89); `run_upgrade` chains individual `v{N}_to_v{N+1}` functions (v74 through v89) until the stored data matches the current version. +`CATALOG_VERSION` is the current schema version (90); `run_upgrade` chains individual `v{N}_to_v{N+1}` functions (v74 through v90) until the stored data matches the current version. Each version-specific submodule operates entirely on frozen `objects_v{N}` types, ensuring that future code changes cannot retroactively break old migrations. -The `objects\!` macro generates per-version support code, handling both old protobuf-based snapshots (v74--v78) and newer serde-based snapshots (v79+). +The `objects!` macro generates per-version support code, handling both old protobuf-based snapshots (v74--v78) and newer serde-based snapshots (v79+). The `json_compatible` submodule provides helpers for reading old JSON-encoded state when proto types have changed. The v82→v83 migration is a byte-level repair pass that plugs directly into `run_upgrade` rather than going through `run_versioned_upgrade`, because it requires raw access to snapshot diffs. The v83→v84 migration similarly calls `v83_to_v84::upgrade` directly from `run_upgrade`. @@ -17,3 +17,4 @@ The v85→v86 migration runs through `run_versioned_upgrade` and is a no-op that The v86→v87 migration runs through `run_versioned_upgrade` and adds durable cluster autoscaling state: `ManagedCluster` gains `auto_scaling_strategy`, `reconfiguration`, and `burst` (all defaulted to `None`), and the managed `ReplicaLocation`'s single `availability_zone` user-pin becomes an `availability_zones` list recording the zones the replica was provisioned under. The v87→v88 migration runs through `run_versioned_upgrade` and is a no-op: new `CreateOrDropClusterReplicaReasonV1` reasons (`Reconfiguration`, `HydrationBurst`, `Retired`) and new `AlterClusterReconfigurationV1` / `ClusterHydrationBurstV1` event details are additive and confined to the append-only audit log; no existing record changes shape, so a v87-serialized record is already valid v88. The v88→v89 migration runs through `run_versioned_upgrade` and adds `ReconfigurationState::status`, backfilling any in-flight reconfiguration records as `InProgress`. All other v88→v89 changes (new `ReconfigurationLifecycleV1::ResourceExhausted` variant, `ClusterReplicaLoggingV1`, `BurstFinishCauseV1`, updated `AlterClusterReconfigurationV1` and `ClusterHydrationBurstV1` audit event fields) are additive and confined to the append-only audit log; no existing record changes shape beyond the reconfiguration status backfill. +The v89→v90 migration runs through `run_versioned_upgrade` and adds the `arrangement_compression` flag to managed clusters and cluster replicas, backfilling it as `false`. Managed `Cluster` and `ClusterReplica` records are rewritten to include the new field; unmanaged clusters and all other records pass through unchanged. The flag also appears on a managed cluster's in-flight `reconfiguration` target, backfilled the same way. diff --git a/doc/developer/generated/catalog/memory/objects.md b/doc/developer/generated/catalog/memory/objects.md index 98a7483cb08d6..0465b9d3840bc 100644 --- a/doc/developer/generated/catalog/memory/objects.md +++ b/doc/developer/generated/catalog/memory/objects.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/memory/objects.rs -revision: 74f18a3354 +revision: fca741734d --- # catalog::memory::objects diff --git a/doc/developer/generated/cluster-controller/_crate.md b/doc/developer/generated/cluster-controller/_crate.md index 0e609c9e5a21d..2edb8aaf8c056 100644 --- a/doc/developer/generated/cluster-controller/_crate.md +++ b/doc/developer/generated/cluster-controller/_crate.md @@ -1,6 +1,6 @@ --- source: src/cluster-controller/src/lib.rs -revision: 74f18a3354 +revision: fca741734d --- # mz-cluster-controller diff --git a/doc/developer/generated/cluster-controller/ctx.md b/doc/developer/generated/cluster-controller/ctx.md index 17e422045b550..54cf4aa0857a5 100644 --- a/doc/developer/generated/cluster-controller/ctx.md +++ b/doc/developer/generated/cluster-controller/ctx.md @@ -1,6 +1,6 @@ --- source: src/cluster-controller/src/ctx.rs -revision: bbe8b6977e +revision: fca741734d --- # cluster-controller::ctx diff --git a/doc/developer/generated/cluster-controller/strategy.md b/doc/developer/generated/cluster-controller/strategy.md index bc88a43bda19a..13595d5c46722 100644 --- a/doc/developer/generated/cluster-controller/strategy.md +++ b/doc/developer/generated/cluster-controller/strategy.md @@ -1,6 +1,6 @@ --- source: src/cluster-controller/src/strategy.rs -revision: bbe8b6977e +revision: fca741734d --- # cluster-controller::strategy diff --git a/doc/developer/generated/compute-client/controller/_module.md b/doc/developer/generated/compute-client/controller/_module.md index bce3dd538ad16..08494aae3962e 100644 --- a/doc/developer/generated/compute-client/controller/_module.md +++ b/doc/developer/generated/compute-client/controller/_module.md @@ -1,6 +1,6 @@ --- source: src/compute-client/src/controller.rs -revision: e926ec3a86 +revision: fca741734d --- # mz-compute-client::controller diff --git a/doc/developer/generated/compute-types/config.md b/doc/developer/generated/compute-types/config.md index 9b709a58810eb..0b50ac80bbea9 100644 --- a/doc/developer/generated/compute-types/config.md +++ b/doc/developer/generated/compute-types/config.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/config.rs -revision: 4267863081 +revision: fca741734d --- # compute-types::config diff --git a/doc/developer/generated/compute-types/dyncfgs.md b/doc/developer/generated/compute-types/dyncfgs.md index 107598f1486bf..3bdebbac6522b 100644 --- a/doc/developer/generated/compute-types/dyncfgs.md +++ b/doc/developer/generated/compute-types/dyncfgs.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/dyncfgs.rs -revision: 0d67fec095 +revision: 18d4571699 --- # compute-types::dyncfgs diff --git a/doc/developer/generated/compute-types/explain/text.md b/doc/developer/generated/compute-types/explain/text.md index ed5bc1f6be38d..8d4cb88a45a01 100644 --- a/doc/developer/generated/compute-types/explain/text.md +++ b/doc/developer/generated/compute-types/explain/text.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/explain/text.rs -revision: e926ec3a86 +revision: 58b5ff70c8 --- # compute-types::explain::text diff --git a/doc/developer/generated/compute-types/plan/_module.md b/doc/developer/generated/compute-types/plan/_module.md index 8dddab97ba1d5..716bb13623f2f 100644 --- a/doc/developer/generated/compute-types/plan/_module.md +++ b/doc/developer/generated/compute-types/plan/_module.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/plan.rs -revision: 92047d0776 +revision: 481d4de5cb --- # compute-types::plan diff --git a/doc/developer/generated/compute-types/plan/join/delta_join.md b/doc/developer/generated/compute-types/plan/join/delta_join.md index bba1e6dbf3f5c..9a37c6d933d24 100644 --- a/doc/developer/generated/compute-types/plan/join/delta_join.md +++ b/doc/developer/generated/compute-types/plan/join/delta_join.md @@ -1,10 +1,10 @@ --- source: src/compute-types/src/plan/join/delta_join.rs -revision: e926ec3a86 +revision: 58b5ff70c8 --- # compute-types::plan::join::delta_join Plans delta joins: a join over multiple inputs implemented as one independent dataflow path per input relation. -`DeltaJoinPlan` contains a `DeltaPathPlan` for each source relation; each path specifies a `source_key: Vec` and a sequence of `DeltaStagePlan` lookup stages, each carrying `stream_key` and `lookup_key` as `Vec`. +`DeltaJoinPlan` contains a `DeltaPathPlan` for each source relation; each path specifies a `source_key: Option>` (the source arrangement key, or `None` when the source is consumed as a raw unarranged collection in single-time dataflows) and a sequence of `DeltaStagePlan` lookup stages, each carrying `stream_key` and `lookup_key` as `Vec`. Delta joins re-use existing arrangements and create no new stateful operators, making them efficient when all input arrangements already exist. diff --git a/doc/developer/generated/compute-types/plan/lowering.md b/doc/developer/generated/compute-types/plan/lowering.md index bc52539d4bc0e..1205e52203b29 100644 --- a/doc/developer/generated/compute-types/plan/lowering.md +++ b/doc/developer/generated/compute-types/plan/lowering.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/plan/lowering.rs -revision: f74a121770 +revision: 380c8dd1a1 --- # compute-types::plan::lowering diff --git a/doc/developer/generated/compute-types/plan/scalar.md b/doc/developer/generated/compute-types/plan/scalar.md index 4e74ca1e36733..6017ef4188f46 100644 --- a/doc/developer/generated/compute-types/plan/scalar.md +++ b/doc/developer/generated/compute-types/plan/scalar.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/plan/scalar.rs -revision: 822256a77d +revision: 380c8dd1a1 --- # compute-types::plan::scalar diff --git a/doc/developer/generated/compute/render/columnar.md b/doc/developer/generated/compute/render/columnar.md new file mode 100644 index 0000000000000..543950fd64d1d --- /dev/null +++ b/doc/developer/generated/compute/render/columnar.md @@ -0,0 +1,18 @@ +--- +source: src/compute/src/render/columnar.rs +revision: d43cd78803 +--- + +# mz-compute::render::columnar + +Columnar dataflow edge support. + +Defines `CollectionEdge`, a wrapper that lets dataflow edges between Plan nodes carry either row-based (`VecCollection`) or columnar (`ColumnarCollection`) batches of `(D, T, R)` updates. + +`ColumnarCollection` mirrors differential's `VecCollection` with `Column<(D, T, R)>` as the container instead of `Vec<(D, T, R)>`. + +`CollectionEdge` is an enum with two variants: +- `Vec` — row-formatted collection; the current default for all producers. +- `Columnar` — columnar collection; reserved for producers once the migration completes. + +The migration is consumer-first: every Plan-node consumer learns to accept both variants before any producer emits the columnar variant. Consumers that have not yet learned the columnar form fall back to `CollectionEdge::into_vec`, which decodes through a named `ColumnarToVec` operator. These repack seams remain visible in dataflow introspection so they can be found and retired. Pure passthrough consumers (Negate, Union) round-trip the columnar variant without decoding. diff --git a/doc/developer/generated/compute/render/join/delta_join.md b/doc/developer/generated/compute/render/join/delta_join.md index 5f72c25fc9f71..1e04070e23cd9 100644 --- a/doc/developer/generated/compute/render/join/delta_join.md +++ b/doc/developer/generated/compute/render/join/delta_join.md @@ -1,6 +1,6 @@ --- source: src/compute/src/render/join/delta_join.rs -revision: 1c55de49eb +revision: 58b5ff70c8 --- # mz-compute::render::join::delta_join diff --git a/doc/developer/generated/environmentd/deployment/preflight.md b/doc/developer/generated/environmentd/deployment/preflight.md index 5f6408505fd51..1be88b8918c85 100644 --- a/doc/developer/generated/environmentd/deployment/preflight.md +++ b/doc/developer/generated/environmentd/deployment/preflight.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/deployment/preflight.rs -revision: 584bb9030c +revision: a60edac7f1 --- # environmentd::deployment::preflight @@ -8,3 +8,5 @@ revision: 584bb9030c Implements zero-downtime (0dt) preflight checks for new `environmentd` deployments. Compares the catalog's deploy generation against the incoming generation to determine whether to boot in read-only mode, and spawns a background task that waits for the deployment to catch up before fencing out the old environment and rebooting as leader. Also periodically checks for DDL changes on the old environment during the catch-up period, restarting the new process in read-only mode if new objects or replicas appear that need to be hydrated first. + +The DDL check (`check_ddl_changes`) computes baseline IDs from `get_next_ids`, which derives the next user item ID and replica ID from the maximum existing IDs in the catalog rather than from the allocator counter. This avoids false negatives when batch ID allocation (`IdPool`) has advanced the counter ahead of actually-committed items. diff --git a/doc/developer/generated/interchange/envelopes.md b/doc/developer/generated/interchange/envelopes.md index 572fd9cd34ffd..bf171c4beef8d 100644 --- a/doc/developer/generated/interchange/envelopes.md +++ b/doc/developer/generated/interchange/envelopes.md @@ -1,10 +1,14 @@ --- source: src/interchange/src/envelopes.rs -revision: 1c55de49eb +revision: f543f7ba5c --- # interchange::envelopes -Provides `for_each_diff_pair`, a function that walks a single arrangement batch and invokes a callback for each `DiffPair` at each `(key, timestamp)`. -The function is generic over both a batch type `B` (bounded by `BatchReader