Skip to content

mz-deploy: support array types in the offline type-checker#37876

Open
sjwiesman wants to merge 1 commit into
MaterializeInc:mainfrom
sjwiesman:mz-deploy-array-types
Open

mz-deploy: support array types in the offline type-checker#37876
sjwiesman wants to merge 1 commit into
MaterializeInc:mainfrom
sjwiesman:mz-deploy-array-types

Conversation

@sjwiesman

Copy link
Copy Markdown
Contributor

Motivation

mz-deploy compile failed to type-check any project whose declared dependencies included an array-typed column, reporting:

error: type "text[]" does not exist

Because stage runs compile first, this blocked deployment outright rather than just local checks. Any project referencing an array-typed source column was undeployable.

Root cause

Array types are not resolved structurally. When name resolution encounters text[], it resolves the element type text and then follows that item's type_details.array_id to reach the _text array type (src/sql/src/names.rs). When array_id is None, resolution bails with the error above.

Builtin type definitions declare array_id: None statically. The adapter catalog populates it during bootstrap: as each Builtin::Type that is a CatalogType::Array is inserted, it reaches back into its element type and sets array_id to its own id (src/adapter/src/catalog/apply.rs).

mz-deploy's offline catalog copied array_id from the builtin definition verbatim, so it was always None. The array types themselves were registered correctly all along. Nothing could ever reach them.

This also explains why hand-editing types.lock to retype the column as text got past the load and instead failed on the model's array operations. The type existed. Only the element-to-array link was missing.

It also left the two halves of the round-trip inconsistent: lock recorded type = "text[]" faithfully, and compile then rejected what lock had written.

Changes

Mirror the adapter's back-patch when seeding builtin types in the offline catalog. The ordering this depends on, element types registered before the array types referencing them, is the same ordering resolve_builtin_type_references already relies on, so no new assumption is introduced. The fix is generic over all array builtins rather than special-cased to text[].

The TaskCatalog overlay needs no change, as builtin types live only in the shared base catalog.

Tests

  • test_resolve_array_types asserts that each of a set of element types points at its corresponding array type via array_id.
  • test_stub_table_with_array_column builds a stub table with text[] and int4[] columns and then type-checks a view over it that uses array_length, unnest and = ANY(...), covering the array operations that previously produced follow-on function ... does not exist errors.

Both were confirmed to fail before the fix, reproducing the reported error exactly.

I separately verified the wider lock/compile round-trip over the other composite types lock can emit: text[][], uint4[], "char"[], mz_timestamp[], mz_aclitem[], text list, text list list, map[text=>text], map[text=>int4], int4range, and the parameterized numeric(5), varchar(10), char(3), timestamp(3). All resolve correctly.

One narrower gap remains and is left for separate work. A dependency with an anonymous record column causes lock to write type = "record", and compile then fails with cannot reference pseudo type pg_catalog.record. Unlike arrays this is not a missing link. record is a pseudo-type with no nameable SQL spelling, so Materialize itself also rejects it as a column declaration. Addressing it needs a different approach, either a structural stub or an explicit unsupported-type diagnostic raised at lock time so the failure surfaces early with a clear message.

Release notes

This release will fix mz-deploy compile and mz-deploy stage failing with type "text[]" does not exist for projects whose dependencies have array-typed columns.

🤖 Generated with Claude Code

Array types are not resolved structurally. When name resolution sees
`text[]` it looks up the element type `text`, then follows that item's
`type_details.array_id` to reach the `_text` array type. Builtin type
definitions declare `array_id: None`, and the adapter catalog fills it in
during bootstrap: as each array type is inserted, it points its element
type back at itself.

The offline catalog copied `array_id` from the builtin definition
verbatim, so it was always `None`. The array types themselves were
registered correctly, but nothing could reach them, and resolution failed
with `type "text[]" does not exist`.

The consequence was that a project could not be compiled or staged at all
if any declared dependency had an array-typed column. `lock` records
`type = "text[]"` faithfully, so the lock and compile halves of the
round-trip disagreed.

Mirror the adapter's back-patch when seeding builtin types. The ordering
this needs, element types before the array types referencing them, is the
same ordering `resolve_builtin_type_references` already relies on.

Adds `test_resolve_array_types`, asserting each element type points at its
array type, and `test_stub_table_with_array_column`, which builds a stub
table with `text[]` and `int4[]` columns and type-checks a view over it
using `array_length`, `unnest` and `= ANY(...)`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant