refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep#2630
Conversation
…ridep enable_deepep was a deprecated BackendConfig flag whose __post_init__ conversion silently rewrote experts/dispatcher -- and because it ran last, it overrode any explicitly-set dispatcher (so `dispatcher: hybridep` next to a stale `enable_deepep: true` was forced back to deepep). Remove the conversion. enable_deepep is now ignored with a logged warning if a config still sets it; the field is retained only so an old config does not crash the kw_only dataclass. All in-repo configs/tests are migrated to explicit fields, preserving prior runtime behavior: - enable_deepep: true -> experts: gmm, dispatcher: deepep - enable_deepep: false -> dispatcher: torch - 5 configs wrote `experts: te` alongside `enable_deepep: true`; the flag was silently forcing experts=gmm at runtime, so they migrate to `experts: gmm`. Also switch the recipes hitting the 26.06 release-testing DeepEP internode dispatch faults (recv counters stuck at -1 / CPU-dispatch timeout; AM-450, AM-473, AM-488, AM-490, all ep_size>8) to the HybridEP dispatcher, mirroring #2614 (glm_4.5_air) and the gpt_oss_120b / ling_1t_sft / *_gb200 recipes already on it: minimax_m2.{1,5,7}_hellaswag_pp, qwen3_next_te_deepep (benchmark+finetune), glm_4.7_te_deepep (benchmark+finetune), qwen3_vl_moe_235b, step_3.5_flash_hellaswag_pp. Tests: rewrite the BackendConfig enable_deepep cases to assert ignore+warn, and migrate model-test BackendConfig kwargs from enable_deepep=False to dispatcher="torch". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
| rms_norm: torch_fp32 | ||
| rope_fusion: false | ||
| enable_deepep: false | ||
| dispatcher: torch |
There was a problem hiding this comment.
Good catch. enable_deepep: false had migrated 1:1 to dispatcher: torch, but this example is ep_size: 8 (intra-node), so deepep is appropriate. Changed to experts: gmm + dispatcher: deepep in 8c83a5b.
Rule applied across these: ep_size > 8 (internode) → hybridep, ep_size <= 8 → deepep, paired with experts: gmm to match the real examples/vlm_finetune/nemotron_omni/* configs.
| rms_norm: torch_fp32 | ||
| rope_fusion: false | ||
| enable_deepep: false | ||
| dispatcher: torch |
There was a problem hiding this comment.
if the original one is torch can we change it to deepep/hybridep
There was a problem hiding this comment.
Yes — done in 8c83a5b. This example is ep_size: 8 (intra-node), so it now uses experts: gmm + dispatcher: deepep. The rule: ep_size <= 8 → deepep, ep_size > 8 (internode) → hybridep.
| backend: | ||
| _target_: nemo_automodel.components.models.common.utils.BackendConfig | ||
| enable_deepep: false | ||
| dispatcher: torch |
There was a problem hiding this comment.
Done (8c83a5b): switched to experts: gmm + dispatcher: deepep — this config is ep_size: 8 (intra-node). Applied the same to the other customizer_gpt_oss_*_full_sft* and customizer_nemotron_nano_* configs that have EP.
The single-GPU PEFT variants (customizer_gpt_oss_peft{,_packing}, ep_size: null, nproc_per_node: 1) stay on torch — no expert parallelism, so deepep/hybridep (multi-rank all-to-all) would not apply.
The enable_deepep: false -> dispatcher: torch migrations left configs that have real expert parallelism on the slow torch reference path. Switch them to a performant dispatcher per ep_size: ep_size > 8 (internode) -> hybridep, ep_size <= 8 (intra-node) -> deepep, each paired with experts: gmm to match the sibling deepep configs for these models. Single-GPU PEFT configs (ep_size=null) and the 4-layer functional test keep dispatcher: torch (no/minimal expert parallelism). Addresses review feedback from HuiyingLi on #2630. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Pipeline 55187333 (main-mirror, eos/H100) shows ep_size>8 deepep recipes faulting in DeepEP internode dispatch (RuntimeError: DeepEP error: timeout (dispatch CPU) at internode_dispatch). On H100 (8 GPUs/node) any ep_size>8 spans nodes, so deepep internode dispatch faults; hybridep is the validated internode path (mirrors #2614). Switch deepep -> hybridep: - glm_5_hellaswag_pp, glm_5.1_hellaswag_pp (ep64; set to deepep earlier in this PR, but ep64 > 8 so they must be hybridep) - glm_5.1_lora, ling_flash_2_0_sft, minimax_m2.7_hellaswag_lora (ep32) - deepseek_v4_flash_hellaswag{,_lora,_packed_sequence} (ep32), deepseek_v4_pro_*_ep64 (ep64), hy3_preview_deepep_lora (ep64), mimo_v2_flash_hellaswag (ep32), nemotron_ultra_v3_hellaswag_peft (ep32) nemotron_ultra_v3_squad (ep64) is left as-is: it fails an unrelated check (non_pp_size must be a multiple of ep_size) from running a 16-node recipe on 1 node, not a dispatch fault. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Addressed @HuiyingLi's review + a follow-up
|
Without a ci section the harness defaults to 1 node (8 GPUs), which fails the "non_pp_size must be a multiple of ep_size" check (non_pp_size=8 vs ep_size=64). Add a ci section requesting 8 nodes (pp_size(1) * ep_size(64) = 64 GPUs) and assign recipe_owner adil-a, mirroring nemotron_ultra_v3_hellaswag_peft.yaml. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Without a ci section the harness defaults to 1 node, which fails the "non_pp_size must be a multiple of ep_size" check for ep_size>8 recipes. Add ci sections (recipe_owner HuiyingLi), sizing nodes to pp_size*ep_size: - nemotron_ultra_v3_hellaswag_peft_gb200 (ep16, 4 GB200/node): 4 nodes - minimax_m3_vl_sft_ep32pp4 (ep32, pp4): 16 nodes - deepseekv3_pretrain (ep16, pp8): 16 nodes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
step3p7_medpix_200b_ep32pp4 (ep32; surfaced failing in pipeline 55187316) and minimax_m3_vl_sft_ep32pp4 (ep32) are internode on H100 (8 GPUs/node), where DeepEP dispatch faults. hybridep is the validated internode path (mirrors #2614). Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
validate-nightly-recipes (tools/lint_example_yamls.py) requires the top-level ci section to be the last section; the ci block I added landed before the wandb section. Move it after wandb so ci is the last top-level key. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
| backend: | ||
| _target_: nemo_automodel.components.models.common.utils.BackendConfig | ||
| enable_deepep: false | ||
| dispatcher: torch |
There was a problem hiding this comment.
Yes, intentional. This PEFT recipe is single-GPU (ep_size: null, dp_size: 1, nproc_per_node: 1), so there is no expert parallelism — deepep/hybridep are multi-rank all-to-all dispatchers and do not apply, so torch is correct here. (The *_full_sft* variants are ep_size: 8 and did move to deepep.)
| backend: | ||
| _target_: nemo_automodel.components.models.common.utils.BackendConfig | ||
| enable_deepep: false | ||
| dispatcher: torch |
There was a problem hiding this comment.
Same — customizer_gpt_oss_peft_packing is also single-GPU (ep_size: null, nproc_per_node: 1), so torch is intentional (no EP to dispatch).
|
|
||
| ci: | ||
| # pp_size(1) * ep_size(16) = 16 GPUs => 4 nodes (4 GB200/node). | ||
| recipe_owner: HuiyingLi |
There was a problem hiding this comment.
Yes — fixed to adil-a in 70fb43e (matches nemotron_ultra_v3_squad and nemotron_ultra_v3_hellaswag_peft).
|
|
||
| ci: | ||
| # pp_size(8) * ep_size(16) = 128 GPUs => 16 nodes (8 H100/node). | ||
| recipe_owner: HuiyingLi |
There was a problem hiding this comment.
Yes — fixed to hemildesai in 70fb43e (matches the deepseek_v4_* recipes).
|
|
||
| ci: | ||
| # pp_size(4) * ep_size(32) = 128 GPUs => 16 nodes (8 H100/node). | ||
| recipe_owner: HuiyingLi |
There was a problem hiding this comment.
Yes — fixed to athitten (Abhishree) in 70fb43e.
Per @HuiyingLi review on #2630, the ci sections should name the actual model owners rather than a single placeholder: - nemotron_ultra_v3_hellaswag_peft_gb200: -> adil-a (matches nemotron_ultra_v3_squad / _hellaswag_peft) - deepseekv3_pretrain: -> hemildesai (matches the deepseek_v4 recipes) - minimax_m3_vl_sft_ep32pp4: -> athitten Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
|
/ok to test 70fb43e |
…o hybridep (2630)` into `r0.5.0` Cherry-pick of #2630 (50a12b7) onto r0.5.0. Release-branch resolutions: - Dropped 4 files modified by #2630 that do not exist on r0.5.0 (the models were not backported): examples/vlm_finetune/minimax_m3/minimax_m3_vl_sft_ep32pp4.yaml, tests/unit_tests/models/minimax_m3_vl/{conftest.py,test_minimax_rope_fp32.py}, tests/unit_tests/models/qwen3_5/test_qwen3_5_dense_backbone.py. - Applied #2630's enable_deepep=False -> dispatcher="torch" migration to tests/unit_tests/models/qwen3_5/test_qwen3_5_pp.py as well. #2630 did not touch this file (main migrated it earlier in #2557, not yet on r0.5.0); the stale enable_deepep=False would otherwise become a silently-ignored no-op, letting the backend default to dispatcher="deepep" on the GPU runner. The post-merge CI failures on main for 50a12b7 were pre-existing L0 unit-test suite timeouts (CPU 600s, GPU 1800s) seen on the parent commit too, not a regression introduced by this change. --- refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep (#2630) enable_deepep was a deprecated BackendConfig flag whose __post_init__ conversion silently rewrote experts/dispatcher -- and because it ran last, it overrode any explicitly-set dispatcher (so `dispatcher: hybridep` next to a stale `enable_deepep: true` was forced back to deepep). Remove the conversion. enable_deepep is now ignored with a logged warning if a config still sets it; the field is retained only so an old config does not crash the kw_only dataclass. All in-repo configs/tests are migrated to explicit fields, preserving prior runtime behavior: - enable_deepep: true -> experts: gmm, dispatcher: deepep - enable_deepep: false -> dispatcher: torch - 5 configs wrote `experts: te` alongside `enable_deepep: true`; the flag was silently forcing experts=gmm at runtime, so they migrate to `experts: gmm`. Also switch the recipes hitting the 26.06 release-testing DeepEP internode dispatch faults (recv counters stuck at -1 / CPU-dispatch timeout; AM-450, AM-473, AM-488, AM-490, all ep_size>8) to the HybridEP dispatcher, mirroring #2614 (glm_4.5_air) and the gpt_oss_120b / ling_1t_sft / *_gb200 recipes already on it. Tests: rewrite the BackendConfig enable_deepep cases to assert ignore+warn, and migrate model-test BackendConfig kwargs from enable_deepep=False to dispatcher="torch". (cherry picked from commit 50a12b7) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…o hybridep (2630)` into `r0.5.0` Cherry-pick of #2630 (50a12b7) onto r0.5.0. Release-branch resolutions: - Dropped 4 files modified by #2630 that do not exist on r0.5.0 (the models were not backported): examples/vlm_finetune/minimax_m3/minimax_m3_vl_sft_ep32pp4.yaml, tests/unit_tests/models/minimax_m3_vl/{conftest.py,test_minimax_rope_fp32.py}, tests/unit_tests/models/qwen3_5/test_qwen3_5_dense_backbone.py. - Applied #2630's enable_deepep=False -> dispatcher="torch" migration to tests/unit_tests/models/qwen3_5/test_qwen3_5_pp.py as well. #2630 did not touch this file (main migrated it earlier in #2557, not yet on r0.5.0); the stale enable_deepep=False would otherwise become a silently-ignored no-op, letting the backend default to dispatcher="deepep" on the GPU runner. The post-merge CI failures on main for 50a12b7 were pre-existing L0 unit-test suite timeouts (CPU 600s, GPU 1800s) seen on the parent commit too, not a regression introduced by this change. --- refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep (#2630) enable_deepep was a deprecated BackendConfig flag whose __post_init__ conversion silently rewrote experts/dispatcher -- and because it ran last, it overrode any explicitly-set dispatcher (so `dispatcher: hybridep` next to a stale `enable_deepep: true` was forced back to deepep). Remove the conversion. enable_deepep is now ignored with a logged warning if a config still sets it; the field is retained only so an old config does not crash the kw_only dataclass. All in-repo configs/tests are migrated to explicit fields, preserving prior runtime behavior: - enable_deepep: true -> experts: gmm, dispatcher: deepep - enable_deepep: false -> dispatcher: torch - 5 configs wrote `experts: te` alongside `enable_deepep: true`; the flag was silently forcing experts=gmm at runtime, so they migrate to `experts: gmm`. Also switch the recipes hitting the 26.06 release-testing DeepEP internode dispatch faults (recv counters stuck at -1 / CPU-dispatch timeout; AM-450, AM-473, AM-488, AM-490, all ep_size>8) to the HybridEP dispatcher, mirroring #2614 (glm_4.5_air) and the gpt_oss_120b / ling_1t_sft / *_gb200 recipes already on it. Tests: rewrite the BackendConfig enable_deepep cases to assert ignore+warn, and migrate model-test BackendConfig kwargs from enable_deepep=False to dispatcher="torch". (cherry picked from commit 50a12b7) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
…o hybridep (2630)` into `r0.5.0` (#2644) Cherry-pick of #2630 (50a12b7) onto r0.5.0. Release-branch resolutions: - Dropped 4 files modified by #2630 that do not exist on r0.5.0 (the models were not backported): examples/vlm_finetune/minimax_m3/minimax_m3_vl_sft_ep32pp4.yaml, tests/unit_tests/models/minimax_m3_vl/{conftest.py,test_minimax_rope_fp32.py}, tests/unit_tests/models/qwen3_5/test_qwen3_5_dense_backbone.py. - Applied #2630's enable_deepep=False -> dispatcher="torch" migration to tests/unit_tests/models/qwen3_5/test_qwen3_5_pp.py as well. #2630 did not touch this file (main migrated it earlier in #2557, not yet on r0.5.0); the stale enable_deepep=False would otherwise become a silently-ignored no-op, letting the backend default to dispatcher="deepep" on the GPU runner. The post-merge CI failures on main for 50a12b7 were pre-existing L0 unit-test suite timeouts (CPU 600s, GPU 1800s) seen on the parent commit too, not a regression introduced by this change. --- refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep (#2630) enable_deepep was a deprecated BackendConfig flag whose __post_init__ conversion silently rewrote experts/dispatcher -- and because it ran last, it overrode any explicitly-set dispatcher (so `dispatcher: hybridep` next to a stale `enable_deepep: true` was forced back to deepep). Remove the conversion. enable_deepep is now ignored with a logged warning if a config still sets it; the field is retained only so an old config does not crash the kw_only dataclass. All in-repo configs/tests are migrated to explicit fields, preserving prior runtime behavior: - enable_deepep: true -> experts: gmm, dispatcher: deepep - enable_deepep: false -> dispatcher: torch - 5 configs wrote `experts: te` alongside `enable_deepep: true`; the flag was silently forcing experts=gmm at runtime, so they migrate to `experts: gmm`. Also switch the recipes hitting the 26.06 release-testing DeepEP internode dispatch faults (recv counters stuck at -1 / CPU-dispatch timeout; AM-450, AM-473, AM-488, AM-490, all ep_size>8) to the HybridEP dispatcher, mirroring #2614 (glm_4.5_air) and the gpt_oss_120b / ling_1t_sft / *_gb200 recipes already on it. Tests: rewrite the BackendConfig enable_deepep cases to assert ignore+warn, and migrate model-test BackendConfig kwargs from enable_deepep=False to dispatcher="torch". (cherry picked from commit 50a12b7) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
What does this PR do ?
Removes the deprecated
BackendConfig.enable_deepepflag and switches the MoE recipes failing the 26.06 release-testing DeepEP internode-dispatch faults to the HybridEP dispatcher.enable_deepepsilently rewroteexperts/dispatcherin__post_init__, and since that ran last it would override an explicitly-setdispatcher(a staleenable_deepep: truenext todispatcher: hybridepwas forced back todeepep). It is now ignored with a logged warning if set — the field is retained only so an old config does not crash thekw_onlydataclass — and all in-repo configs/tests are migrated to explicitdispatcher/experts.Changelog
BackendConfig: stop honoringenable_deepep; log a warning (rank 0) if set, and no longer alterdispatcher/experts.enable_deepep, preserving prior runtime behavior:true → experts: gmm + dispatcher: deepep,false → dispatcher: torch.experts: tealongsideenable_deepep: true; the flag was silently forcingexperts=gmmat runtime, so they migrate toexperts: gmm(flag iftewas actually intended).ep_size>8recipes todispatcher: hybridep(Linear AM-450/473/488/490):minimax_m2.{1,5,7}_hellaswag_pp,qwen3_next_te_deepep(benchmark+finetune),glm_4.7_te_deepep(benchmark+finetune),qwen3_vl_moe_235b,step_3.5_flash_hellaswag_pp. Mirrors fix(docker): build DeepEP against the NVSHMEM wheel matching the apt runtime #2614.enable_deepeptest cases to assert the ignore+warn behavior; drop one pre-existing unused-variable (F841) in a touched test.Before your PR is "Ready for review"
Pre checks:
tests/unit_tests/moe/test_backend_config.pyenable_deepep cases (49 passed locally)docs/guides/vlm/nemotron-omniAdditional Information
ep_size>8DeepEP internode dispatch faults).