Add definitely_all_null() for cheap all-null detection

[joseph-isaacs]

Summary

This PR introduces a new definitely_all_null() method to ArrayRef that performs cheap, static detection of entirely-null arrays without executing compute. This enables short-circuiting in compute kernels to avoid unnecessary work and canonicalization when processing all-null inputs.

The method returns true only when all-null-ness can be proven without computation:

• Constant-null arrays
• Arrays with Validity::AllInvalid
• Arrays with a constant-false validity array

A false result means "not provably all-null" (conservative), not "contains valid values", so callers must fall back to their normal path.

Changes

1. New definitely_all_null() method (vortex-array/src/array/erased.rs):

   • Added public method with comprehensive documentation
   • Checks for constant-null scalars and static validity patterns
   • Includes unit tests covering constant-null detection and validity-based detection

2. Removed unused Validity::not() method (vortex-array/src/validity.rs):

   • Cleaned up dead code that was not being used

3. Updated compute kernels to use the new method:

   • is_null.rs: Short-circuit entirely-null inputs to return all-true
   • is_not_null.rs: Short-circuit entirely-null inputs to return all-false
   • struct/compute/take.rs: Return canonical all-null constant array instead of manually constructing with Validity::AllInvalid
   • list/compute/filter.rs: Return canonical all-null constant array instead of manually constructing

4. Added ConstantArray::null() helper (vortex-array/src/arrays/constant/array.rs):

   • Convenience method to construct the canonical all-null representation
   • Documented as the standard representation for all-null arrays

Testing

Added comprehensive unit tests in vortex-array/src/array/erased.rs:

• definitely_all_null_detects_constant_null: Verifies detection of constant-null and constant-non-null arrays
• definitely_all_null_via_validity: Verifies detection via AllInvalid validity, constant-false validity arrays, and non-nullable/all-valid cases

Existing tests pass with the updated compute kernels using the new short-circuit path.

https://claude.ai/code/session_01Q8K741TL4zABgsL1N4kLWw

[codspeed-hq]

Merging this PR will degrade performance by 29.83%

⚡ 1 improved benchmark
❌ 15 regressed benchmarks
✅ 1493 untouched benchmarks
⏩ 83 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	slice_empty_vortex	368.3 ns	2,628.6 ns	-85.99%
❌	Simulation	chunked_bool_canonical_into[(1000, 10)]	20.3 µs	35.8 µs	-43.32%
❌	Simulation	slice_vortex_buffer[1024]	871.4 ns	1,335 ns	-34.73%
❌	Simulation	slice_vortex_buffer[16384]	871.4 ns	1,335 ns	-34.73%
❌	Simulation	slice_vortex_buffer[2048]	871.4 ns	1,335 ns	-34.73%
❌	Simulation	slice_vortex_buffer[128]	871.4 ns	1,335 ns	-34.73%
❌	Simulation	slice_vortex_buffer[65536]	871.4 ns	1,335 ns	-34.73%
❌	Simulation	chunked_varbinview_canonical_into[(1000, 10)]	162.2 µs	198.2 µs	-18.15%
❌	Simulation	chunked_varbinview_into_canonical[(1000, 10)]	177.7 µs	214.2 µs	-17.04%
❌	Simulation	search_index_below_min_chunked	1.3 ms	1.5 ms	-13.61%
❌	Simulation	search_index_mixed_out_of_range_chunked	1.3 ms	1.5 ms	-13.32%
❌	Simulation	count_i32_clustered_nulls	47 µs	54 µs	-12.97%
❌	Simulation	search_index_full_range_random_chunked	1.4 ms	1.6 ms	-12.08%
❌	Simulation	chunked_varbinview_canonical_into[(100, 100)]	273.2 µs	308.1 µs	-11.33%
❌	Simulation	chunked_varbinview_into_canonical[(100, 100)]	330.8 µs	367.7 µs	-10.02%
⚡	Simulation	chunked_varbinview_opt_into_canonical[(1000, 10)]	229.1 µs	193.6 µs	+18.32%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/epic-hawking-3ixvms-phase1 (803e530) with develop (85aad72)²}

Footnotes

1. 83 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on develop (0ed06b3) during the generation of this report, so 85aad72 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩