Skip to content

[NNX] Emergency checkpoint: build the manager with a Linen-layout abstract#4372

Draft
ecnal-cienet wants to merge 2 commits into
mainfrom
fix/nnx-emc-linen-abstract
Draft

[NNX] Emergency checkpoint: build the manager with a Linen-layout abstract#4372
ecnal-cienet wants to merge 2 commits into
mainfrom
fix/nnx-emc-linen-abstract

Conversation

@ecnal-cienet

@ecnal-cienet ecnal-cienet commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Description

The EMC resume-from-GCS test (maxtext_emc_resume_from_gcs) fails when a pure_nnx run restores from step 99:

ValueError: User-provided restore item and on-disk value metadata tree structures do not match
  model:     - Source: {...}         - Target: MISSING
  opt_state: - Source: MISSING       - Target: [...]

What's happening: pure_nnx saves checkpoints in the on-disk checkpoint layout (params/opt_state/step plus an nnx_aux subtree, via to_checkpoint_dict) so old Linen checkpoints stay compatible. Both the regular and the emergency manager save this way. The difference is that when we build the emergency manager we hand it the NNX abstract state (model/optimizer), and the emergency manager keeps that abstract and restores against it — it does not use the item we pass at restore time. So it ends up matching a checkpoint-layout tree on disk against an NNX tree, and Orbax complains.

This is also why the earlier fix (0da53c3) didn't help: it converts the restore-time item, but the emergency manager never looks at that item.

Fix: convert the abstract with to_checkpoint_dict inside create_orbax_emergency_checkpoint_manager when it's an nnx.State, so what the manager holds matches what's on disk. Restore still reshapes back to NNX afterwards, so the state we train on is NNX. On-disk format is unchanged, so regular and EMC checkpoints are still interchangeable. 5 lines, in the constructor.

Tests

Reproduced and verified on our v6e-8 (jax 0.10.0, same as the failing job): before the fix the save→restore hits the exact same ValueError, after the fix it restores fine.

  • TestEmergencyManagerAbstractLayout — mocks the Orbax manager and checks it gets a Linen abstract (params/opt_state/step), not the NNX model/optimizer. Runs anywhere, so CI catches it if someone breaks this again.
  • TestEmergencySaveRestoreRoundTrip — full save→restore through the real emergency manager, checks it comes back as NNX with the right weights. Skips if the EMC manager can't be built.
  • Both fail if I revert the fix. 3 pass on the TPU, existing checkpoint tests still pass.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@ecnal-cienet ecnal-cienet force-pushed the fix/nnx-emc-linen-abstract branch from 0e7124b to 03d1b36 Compare July 7, 2026 20:32
@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.28571% with 9 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/utils/maxtext_utils.py 58.82% 4 Missing and 3 partials ⚠️
src/maxtext/common/checkpointing.py 96.42% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@ecnal-cienet ecnal-cienet force-pushed the fix/nnx-emc-linen-abstract branch 3 times, most recently from fc2d7ec to 8cf016e Compare July 9, 2026 17:33
The NNX-only rngs/dropout state is stripped from the Linen on-disk layout, so on
resume it was refilled from a base key(0) -- every preemption reset the dropout
RNG stream even though the dataset iterator was restored. Persist that state as a
separate nnx_aux composite item (the items payload stays byte-identical to the
Linen layout, preserving interchangeability) and restore it when present, falling
back to deterministic defaults for Linen-trained or pre-fix checkpoints. Emergency
managers keep the default-fill path.

Also add checkpoint_missing_param_policy for a weight the checkpoint doesn't carry.
Weights are filtered by Variable type (nnx.Param via nnx.split_state) so only real
weights are checked -- rngs/dropout/batch stats are never flagged. "warn" (default)
logs a warning naming the weight and keeps its init value; "error" raises.
…t abstract

pure_nnx saves in the on-disk checkpoint layout (Linen params/opt_state/step plus an
nnx_aux subtree) for both the regular and emergency managers. But the emergency manager
is constructed with the NNX-native abstract (model/optimizer) from get_abstract_state,
and unlike the regular manager it bakes that abstract in and restores against it,
ignoring the restore-time item. So on resume Orbax compared a checkpoint-layout tree on
disk against an NNX target and raised 'User-provided restore item and on-disk value
metadata tree structures do not match', failing the EMC resume-from-GCS job.

Convert the abstract with to_checkpoint_dict in create_orbax_emergency_checkpoint_manager
so the baked-in blueprint matches what maybe_save_checkpoint writes; the restore path
reshapes it back to NNX. Adds a mock guard (asserts the manager gets the checkpoint
layout, not NNX roots) plus an end-to-end save/restore round trip through the real
emergency manager.

Stacked on the to_checkpoint_dict / nnx_aux format change.
@ecnal-cienet ecnal-cienet force-pushed the fix/nnx-emc-linen-abstract branch from 8cf016e to 1c96f5c Compare July 9, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant