Ma/prom metrics/sql min#37877
Draft
mtabebe wants to merge 4 commits into
Draft
Conversation
…g item) Adds the minimal SQL front for metric sinks, adapted from the reference PR MaterializeInc#37560 but scoped to the minimum needed to benchmark the dataflow: - Grammar: CREATE METRIC SINK <name> IN CLUSTER <c> FROM <rel>, minimal DROP METRIC SINK. No SHOW (deferred). - Plan: plan_create_metric_sink resolves FROM and validates the canonical metric-sink columns (metric_name, metric_type, labels, value, help). - Feature flag enable_metric_sink (default off, forced on for item parsing). - Durable CatalogItem::MetricSink: persisted as an ordinary item keyed on create_sql, item type re-derived by re-parsing on boot. No proto/catalog version bump: the audit event maps to the generic IdFullNameV1 detail and reuses mz_audit_log::ObjectType::Sink, and the three catalog-proto enum conversions (CatalogItemType/ObjectType/CommentObjectId) are unreachable for metric sinks. Compiles: mz-sql-parser, mz-sql, mz-catalog. Parser snapshot tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ep 2) Gives shape_metric_sink_source its first caller and wires the sink end to end: - optimize::metric_sink::Optimizer: assembly pipeline (import FROM, MIR-shape the rows, export a MetricSink compute sink, MIR->LIR finalize). Modeled on optimize::index. - Staged sequencer sequence_create_metric_sink (validate -> optimize -> finish): allocates a durable item id + transient view id, optimizes, creates the durable CatalogItem::MetricSink, and ships the dataflow with a read hold across shipping (ship_new_dataflow). No persist/allow_writes: metric sinks write to the in-process registry only. - ExecuteResponse::CreatedMetricSink + Message/Stage plumbing. - Catalog lifecycle: catalog_implications drops the dataflow on DROP; set_optimized_plan/set_physical_plan/set_dataflow_metainfo accept MetricSink; coord.rs bootstrap loops skip metric sinks (don't panic on a durable one at boot; full restart rehydration deferred). - controller doc: metric sinks are sink exports, so replica-targeting passes. cargo check -p mz-adapter is green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Creates a metric sink over a canonical-shaped view, asserts the user series and companion health gauges reach the replica's Prometheus registry (via mz_introspection.mz_cluster_prometheus_metrics), checks the null-value gap semantics, and asserts DROP retracts both. Also covers plan-time rejection of a FROM relation missing/mistyping a canonical column. Does not depend on the deferred mz_metric_sinks catalog view; reads the sink id from the gauge label. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deserialize_item re-parses a durable item's create_sql and converts the Plan back into a CatalogItem to verify it round-trips. This match lacked a Plan::CreateMetricSink arm, so creating (or booting) a durable metric sink panicked with "catalog entry generated inappropriate plan". Add the arm, mirroring CreateIndex. Caught by test/testdrive/metric-sink.td. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remove these sections if your commit already has a good description!
Motivation
Why does this change exist? Link to a GitHub issue, design doc, Slack
thread, or explain the problem in a sentence or two. A reviewer who has
no context should understand why after reading this section.
If this implements or addresses an existing issue, it's enough to link to that:
Closes
Fixes
etc.
Description
What does this PR actually do? Focus on the approach and any non-obvious
decisions. The diff shows the code --- use this space to explain what the
diff can't tell a reviewer.
Verification
How do you know this change is correct? Describe new or existing automated
tests, or manual steps you took.