mz-deploy: support array types in the offline type-checker#37876
Open
sjwiesman wants to merge 1 commit into
Open
mz-deploy: support array types in the offline type-checker#37876sjwiesman wants to merge 1 commit into
sjwiesman wants to merge 1 commit into
Conversation
Array types are not resolved structurally. When name resolution sees `text[]` it looks up the element type `text`, then follows that item's `type_details.array_id` to reach the `_text` array type. Builtin type definitions declare `array_id: None`, and the adapter catalog fills it in during bootstrap: as each array type is inserted, it points its element type back at itself. The offline catalog copied `array_id` from the builtin definition verbatim, so it was always `None`. The array types themselves were registered correctly, but nothing could reach them, and resolution failed with `type "text[]" does not exist`. The consequence was that a project could not be compiled or staged at all if any declared dependency had an array-typed column. `lock` records `type = "text[]"` faithfully, so the lock and compile halves of the round-trip disagreed. Mirror the adapter's back-patch when seeding builtin types. The ordering this needs, element types before the array types referencing them, is the same ordering `resolve_builtin_type_references` already relies on. Adds `test_resolve_array_types`, asserting each element type points at its array type, and `test_stub_table_with_array_column`, which builds a stub table with `text[]` and `int4[]` columns and type-checks a view over it using `array_length`, `unnest` and `= ANY(...)`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
mz-deploy compilefailed to type-check any project whose declared dependencies included an array-typed column, reporting:Because
stagerunscompilefirst, this blocked deployment outright rather than just local checks. Any project referencing an array-typed source column was undeployable.Root cause
Array types are not resolved structurally. When name resolution encounters
text[], it resolves the element typetextand then follows that item'stype_details.array_idto reach the_textarray type (src/sql/src/names.rs). Whenarray_idisNone, resolution bails with the error above.Builtin type definitions declare
array_id: Nonestatically. The adapter catalog populates it during bootstrap: as eachBuiltin::Typethat is aCatalogType::Arrayis inserted, it reaches back into its element type and setsarray_idto its own id (src/adapter/src/catalog/apply.rs).mz-deploy's offline catalog copied
array_idfrom the builtin definition verbatim, so it was alwaysNone. The array types themselves were registered correctly all along. Nothing could ever reach them.This also explains why hand-editing
types.lockto retype the column astextgot past the load and instead failed on the model's array operations. The type existed. Only the element-to-array link was missing.It also left the two halves of the round-trip inconsistent:
lockrecordedtype = "text[]"faithfully, andcompilethen rejected whatlockhad written.Changes
Mirror the adapter's back-patch when seeding builtin types in the offline catalog. The ordering this depends on, element types registered before the array types referencing them, is the same ordering
resolve_builtin_type_referencesalready relies on, so no new assumption is introduced. The fix is generic over all array builtins rather than special-cased totext[].The
TaskCatalogoverlay needs no change, as builtin types live only in the shared base catalog.Tests
test_resolve_array_typesasserts that each of a set of element types points at its corresponding array type viaarray_id.test_stub_table_with_array_columnbuilds a stub table withtext[]andint4[]columns and then type-checks a view over it that usesarray_length,unnestand= ANY(...), covering the array operations that previously produced follow-onfunction ... does not existerrors.Both were confirmed to fail before the fix, reproducing the reported error exactly.
I separately verified the wider lock/compile round-trip over the other composite types
lockcan emit:text[][],uint4[],"char"[],mz_timestamp[],mz_aclitem[],text list,text list list,map[text=>text],map[text=>int4],int4range, and the parameterizednumeric(5),varchar(10),char(3),timestamp(3). All resolve correctly.One narrower gap remains and is left for separate work. A dependency with an anonymous record column causes
lockto writetype = "record", and compile then fails withcannot reference pseudo type pg_catalog.record. Unlike arrays this is not a missing link.recordis a pseudo-type with no nameable SQL spelling, so Materialize itself also rejects it as a column declaration. Addressing it needs a different approach, either a structural stub or an explicit unsupported-type diagnostic raised atlocktime so the failure surfaces early with a clear message.Release notes
This release will fix
mz-deploy compileandmz-deploy stagefailing withtype "text[]" does not existfor projects whose dependencies have array-typed columns.🤖 Generated with Claude Code