Switch ERA5 source to earthmover-public/era5 (full archive)

[aaronspring]

Summary

Swaps the Arraylake data source from earthmover-public/era5-surface-aws to earthmover-public/era5.

Why:

• Longer record: 1940-01-01 to 2025-12-31 (vs 1975-2024 previously) — 50 more years of hourly reanalysis.
• Fresher data: the new archive is updated quarterly, instead of being a static snapshot.
• More variables: 38 surface variables vs 22 previously (adds wind gusts, friction velocity, cloud cover by level, snowfall, several radiation/heat-flux terms, and soil temperature levels 2-4). tp (total precipitation) is now available directly in the store instead of being derived as cp + lsp.

Adaptation required (the new repo restructures the store):

• Group path changed from "{query_type}" to "single/{query_type}" for surface-level variables.
• Time coordinate renamed from time to valid_time — normalized back to time right after opening the dataset so the rest of the retrieval pipeline (and cached filenames/output variable names) is unaffected.
• Three variables were renamed in the store: t2→t2m, d2→d2m, mslp→msl. Added a zarr_name field on ERA5Variable so our catalog's stable short names (t2, d2, mslp) keep mapping correctly to the new store array names.
• Removed the now-unnecessary compute_tp = cp + lsp fallback since tp is a native variable in the new store.
• The new archive compresses several variables with the pcodec codec — added numcodecs[pcodec] as a core dependency, otherwise retrieval fails with codec not available: ''pcodec''. earthmover-public/era5 uses pcodec filter
• The store declares an lsm land-sea-mask coordinate on every variable but ships it unwritten — reads return 721×1440 of pure NaN. It is dropped from downloads rather than propagated as an empty array that looks like a usable mask.

Why the Python floor moves to 3.11

requires-python said >=3.10, but the project could not be installed on 3.10 at all — uv lock fails outright with an unsatisfiable-requirements error:

Because the requested Python version (>=3.10) does not satisfy Python>=3.11
and zarr>=3.0.8,<=3.1.6 depends on Python>=3.11, we can conclude that
zarr>=3.0.8,<=3.1.6 cannot be used. ... your project's requirements are unsatisfiable.

Two dependencies force this:

Dependency	Requires-Python	Why we need it
zarr>=3.0.0	>=3.11 (3.1.x), >=3.12 (3.2.x)	Zarr v3 / Icechunk store access — already a dependency before this PR
numcodecs[pcodec]>=0.16.0	>=3.11	New: earthmover-public/era5 compresses with the pcodec codec

So >=3.10 was already wrong on main (zarr alone made it unsatisfiable); this PR's numcodecs pin makes it wrong for a second, independent reason. Since this PR is what introduces that pin, it's the right place to fix the declared floor rather than leave pyproject.toml advertising an environment nobody can install into. The Programming Language :: Python :: 3.10 classifier, the [tool.mypy] python_version, and black's target-version were all saying 3.10 too, and are synced.

Why 3.11 and not 3.12? >=3.12 also resolves, but 3.11 is the lowest floor that actually works, so it's the least restrictive honest choice. Resolution succeeding isn't sufficient evidence on its own — on 3.11, zarr pins back to 3.1.6, so the pcodec codec (the entire reason numcodecs is pinned) has to register against that older zarr. Verified directly rather than assumed:

$ python3.11 -c "from zarr.registry import get_codec_class; print(get_codec_class('numcodecs.pcodec'))"
zarr 3.1.6 | numcodecs 0.16.5
<class 'zarr.codecs.numcodecs._codecs.PCodec'>

3.11 is genuinely supported, not merely resolvable. The Dockerfile (3.12) and CI are unaffected.

Not yet in scope: the new archive also has pressure-level variables (temperature, wind, humidity, geopotential, etc. at 13 levels from 1000-50 hPa, under a separate pressure/ group). retrieve_era5_data only supports the single/ (surface) group for now — wiring up pressure levels is flagged as follow-up work in the system prompt. AgentConfig.data_group now makes that group reachable by configuration.

Review fixes

Subsequent commits address findings from a review of the initial diff:

• zarr_name had no test coverage — the three store renames were the most breakage-prone part of this change and only failed at runtime, against the network. Now pinned against the store's 38 array names; mutation-verified (retyping t2m, or dropping d2's mapping, each fails four tests).
• Cache key omitted the data source — a warm data/ directory would keep serving files fetched from era5-surface-aws forever. Names are now prefixed with the source repo. This matters beyond staleness: old spatial cp/lsp/ssr/ssrd files have dims (latitude, longitude, time) where new ones are (time, latitude, longitude).
• Empty lsm coordinate dropped from downloads (see above).
• Stale ERA5T 5-day-lag guard removed. That lag describes the near-real-time ERA5T stream, not this quarterly archive; the guard was both too strict and useless (the real limit is months earlier). Date validation now runs before the credential check, since it needs neither credentials nor network — which also fixes test_future_date_returns_error, red for anyone without ARRAYLAKE_API_KEY.
• fg10 (wind gust, ≥ 0) no longer uses a diverging colormap; invariant pinned by a test.
• The "reproduce this download yourself" snippet now actually reproduces it (it selected on valid_time and never renamed).
• "single/" group prefix moved out of retrieval.py into AgentConfig.data_group, next to the data_source it must travel with.

Test plan

• Full suite passes without ARRAYLAKE_API_KEY — 206 passed, 5 skipped. (Previously test_future_date_returns_error failed in this configuration, on main too.)
• Full suite passes with a real key, including live e2e tests — 217 passed, 2 skipped.
• Verified the new repo's group/dimension/variable structure directly via Arraylake (single/spatial, single/temporal, pressure/spatial; 38 vars, valid_time dim, t2m/d2m/msl renames).
• Live end-to-end retrieval against earthmover-public/era5, covering:
  • Temporal query (t2) and spatial query (sst, different group path)
  • Renamed variables d2 → d2m and mslp → msl, output correctly named d2/mslp
  • mslp combined with prime-meridian-crossing longitude bounds (lon range −6° → 10°)
  • Native tp (no longer computed from cp + lsp)
  • Download → cache-hit round trip on the new source-tagged filename
  • Uncovered the missing pcodec codec dependency and the all-NaN lsm array in the process
• Generated snippet extracted and executed verbatim; output diffed against the tool's own — same dims, same coords, np.allclose(..., equal_nan=True) on values.
• pcodec codec registration confirmed on Python 3.11 (see above).

🤖 Generated with Claude Code

[aaronspring]

not sure if these make sense. how did you come up with typical_range so far? gut feeling or programmatically?