docs(training-guides): add checkpointing and resuming TrainJobs guide

[typhoonzero]

What

Adds a new Checkpointing and Resuming TrainJobs guide to the training guides, plus three runnable assets and a routing row in the index.

Kubeflow Trainer v2 has no checkpoint API of its own — the platform provides durable storage, pod restarts (failurePolicy.maxRestarts), and pause/resume (spec.suspend), while the training script must write checkpoints and auto-detect the newest one on start. Get either half wrong and the failure is silent: the job restarts and trains from step 0. This guide documents that split and the non-obvious API behaviour around it.

Contents

• checkpointing-and-resuming.mdx (weight 33 — between the main Trainer v2 guide and the preemption guide that builds on it):
  • What a resumable checkpoint must contain (weights + optimizer + scheduler + step + RNG, not just weights)
  • Storage access modes — RWO vs RWX and when each is actually required
  • A checkpoint-aware TrainingRuntime, with the resume auto-detect explained
  • How each interruption resumes: crash / node failure, preemption, deliberate suspend
  • Per-framework knobs (HF Trainer, LlamaFactory, training_hub, MindSpeed, hand-rolled)
  • Multi-node runs and torch.distributed.checkpoint (DCP) — the repo's first DCP mention
  • A verify drill and a troubleshooting table
• Assets under assets/checkpointing/: checkpoint-pvc.yaml, checkpoint-trainingruntime.yaml, trainjob-resume.yaml
• index.mdx: one new routing row

It deliberately does not restate the HuggingFace save_steps/resume_from_checkpoint recipe that preemptible-trainjobs-with-kueue.mdx already owns; instead it's the general mechanics doc that guide now links to (and vice-versa).

Verification

Behaviour was validated live against Kubeflow Trainer v2.1.0 with a CPU-only TrainJob driven through crash, suspend, and cross-node resume. Findings folded into the doc:

• The trainer.kubeflow.org/trainjob-ancestor-step: trainer label on the trainer replicatedJob is required — without it, every spec.trainer.* override (env, resourcesPerNode, image, command, numNodes) is dropped silently, no error or event.
• podTemplateOverrides can mount volumes but the validating webhook rejects env on trainer/initializer containers — env must go in spec.trainer.env.
• A ReadWriteOnce PVC (Ceph RBD) follows the pod across nodes, so single-node resume does not require RWX; RWX is only needed for numNodes > 1.

All three assets pass kubectl apply --dry-run=server, and doom lint is clean.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying alauda-ai with    Cloudflare Pages

Latest commit:	465de44
Status:	✅  Deploy successful!
Preview URL:	https://0037dd34.alauda-ai.pages.dev
Branch Preview URL:	https://docs-trainer-v2-checkpointin.alauda-ai.pages.dev

View logs