cp: refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep (2630) into r0.5.0#2644
Merged
Merged
Conversation
…o hybridep (2630)` into `r0.5.0` Cherry-pick of #2630 (50a12b7) onto r0.5.0. Release-branch resolutions: - Dropped 4 files modified by #2630 that do not exist on r0.5.0 (the models were not backported): examples/vlm_finetune/minimax_m3/minimax_m3_vl_sft_ep32pp4.yaml, tests/unit_tests/models/minimax_m3_vl/{conftest.py,test_minimax_rope_fp32.py}, tests/unit_tests/models/qwen3_5/test_qwen3_5_dense_backbone.py. - Applied #2630's enable_deepep=False -> dispatcher="torch" migration to tests/unit_tests/models/qwen3_5/test_qwen3_5_pp.py as well. #2630 did not touch this file (main migrated it earlier in #2557, not yet on r0.5.0); the stale enable_deepep=False would otherwise become a silently-ignored no-op, letting the backend default to dispatcher="deepep" on the GPU runner. The post-merge CI failures on main for 50a12b7 were pre-existing L0 unit-test suite timeouts (CPU 600s, GPU 1800s) seen on the parent commit too, not a regression introduced by this change. --- refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep (#2630) enable_deepep was a deprecated BackendConfig flag whose __post_init__ conversion silently rewrote experts/dispatcher -- and because it ran last, it overrode any explicitly-set dispatcher (so `dispatcher: hybridep` next to a stale `enable_deepep: true` was forced back to deepep). Remove the conversion. enable_deepep is now ignored with a logged warning if a config still sets it; the field is retained only so an old config does not crash the kw_only dataclass. All in-repo configs/tests are migrated to explicit fields, preserving prior runtime behavior: - enable_deepep: true -> experts: gmm, dispatcher: deepep - enable_deepep: false -> dispatcher: torch - 5 configs wrote `experts: te` alongside `enable_deepep: true`; the flag was silently forcing experts=gmm at runtime, so they migrate to `experts: gmm`. Also switch the recipes hitting the 26.06 release-testing DeepEP internode dispatch faults (recv counters stuck at -1 / CPU-dispatch timeout; AM-450, AM-473, AM-488, AM-490, all ep_size>8) to the HybridEP dispatcher, mirroring #2614 (glm_4.5_air) and the gpt_oss_120b / ling_1t_sft / *_gb200 recipes already on it. Tests: rewrite the BackendConfig enable_deepep cases to assert ignore+warn, and migrate model-test BackendConfig kwargs from enable_deepep=False to dispatcher="torch". (cherry picked from commit 50a12b7) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
akoumpa
force-pushed
the
cherry-pick-2630-r0.5.0
branch
from
15d3323 to
56ffcde
Compare
Contributor
Author
|
/ok to test 56ffcde |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #2630 (
refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep) intor0.5.0.CI investigation (the two failing jobs on main's merge)
The two failing jobs on the post-merge
mainrun for this commit —L0_Unit_Tests_CPUand
L0_Unit_Tests_GPU—are pre-existing L0 unit-test suite timeouts, not a regression from #2630:
600000ms, GPU1800000ms) on all 3 retries, with no test FAILED/ERROR reported — the suite is killed mid-run before pytest prints a summary. Each retry stops at a different test file, i.e. the suite simply doesn't finish in time (timing nondeterminism), not a deterministic hang or assertion failure.2a6491e4, fix(parallelizer): resolve NemotronH decoder blocks for Nemotron-V3 #2638) GPU run timed out the same way (1800000ms× 3), so the GPU timeout predates this change.BackendConfig.__post_init__(swap an auto-conversion for a one-time warning log); the test delta is net −34 LOC (test_backend_config.py). Nothing here adds a slow code path or new heavy tests.Cherry-pick resolutions for r0.5.0
The commit applied with conflicts because r0.5.0 is behind main; resolved as:
examples/vlm_finetune/minimax_m3/minimax_m3_vl_sft_ep32pp4.yaml,tests/unit_tests/models/minimax_m3_vl/{conftest.py,test_minimax_rope_fp32.py},tests/unit_tests/models/qwen3_5/test_qwen3_5_dense_backbone.py.tests/unit_tests/models/qwen3_5/test_qwen3_5_pp.py. refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep #2630 didn'ttouch this file (main migrated it earlier in feat(qwen3_5): port dense Qwen3.5 to a native custom-model implementation #2557, not yet on r0.5.0). Its stale
enable_deepep=Falsewould otherwise become a silently-ignored no-op after this change, letting thebackend default to
dispatcher="deepep"on the DeepEP-enabled GPU runner. Applied the sameenable_deepep=False → dispatcher="torch"migration refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep #2630 uses everywhere to preserve behavior.nemotron_v3/test_nemotron_v3_model.pyandqwen3_5/test_qwen3_5_mtp.pyresolved by applying refactor(moe): remove enable_deepep, switch failing ep recipes to hybridep #2630's exact
enable_deepep=False → dispatcher="torch"transformation to r0.5.0's versions.After the cherry-pick, no stale
enable_deepep=True/Falseauto-conversion usage remains in r0.5.0.Validation
Directly-affected unit tests pass locally against the cherry-picked tree: