fix(model): apply dict config overrides with custom dispatch#2657
Merged
Conversation
…fig.* is a dict NVBugs: 6259955 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Contributor
Author
|
/ok to test 6e9fdb5 |
HuiyingLi
approved these changes
Jun 20, 2026
akoumpa
added a commit
that referenced
this pull request
Jun 20, 2026
fix(model): apply dict config overrides with custom dispatch (#2657) fix(model): keep custom dispatch and apply overrides when --model.config.* is a dict NVBugs: 6259955 Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NVBug: 6259955
What
get_hf_config/__init_model(nemo_automodel/_transformers/model_init.py) now treat a plain-dictconfigkwarg — how the config loader materializes--model.config.*CLI overrides (and YAMLmodel.config:maps) — as config overrides rather than a finished config object.get_hf_config: adictconfigis popped and its keys folded into the kwargsAutoConfig.from_pretrainedapplies as overrides, so a real config (e.g.
GlmMoeDsaConfigwithnum_hidden_layers=N) is built. Itsarchitecturessurvive, so custom-model dispatch is preserved.__init_model: the now-consumed dictconfigis dropped from kwargs so it isn't forwarded into the modelconstructor (which receives the config positionally).
Adds two regression tests in
tests/unit_tests/_transformers/test_model_init.py.Why
Fixes NVBugs 6259955 Defect 2. Passing
--model.config.num_hidden_layers 16(the natural shrink knob forGLM-5.1 /
GlmMoeDsaForCausalLM) crashed with:Root cause: the override was materialized as a dict and returned verbatim by
get_hf_config.get_architectures(dict)returns[], so_resolve_custom_model_cls_for_configreturnedNoneanddispatch flipped from the custom GlmMoeDsa implementation to the stock-HF path — which forwards the
custom-only
backendkwarg into the built-in transformersGlmMoeDsaForCausalLM.__init__and rejects it.(PR #1784 coerced dict→BackendConfig only on the custom path, so it didn't cover this stock-HF fallback.)
This made it impossible to shrink GLM-5.1 below its 256-H100 footprint via the supported CLI. The fix is
the report's preferred remedy:
--model.config.*overrides no longer change which implementation isdispatched, and they are correctly applied. Affects every registered custom model that accepts custom-only
constructor kwargs, not just GlmMoeDsa.
How tested
On cw-dfw (H100) in the rc6 container, latest main
2b96596a, transformers 5.8.1:NeMoAutoModelForCausalLM.from_pretrainedon a tiny GlmMoeDsa config (noGLM-5.1 download): before → control custom path OK, override →
TypeError: ... 'backend'(exact bugtraceback); after → control unchanged, override keeps the custom path and applies the override (1 layer
built vs the file's 2), exit 0.
pytest tests/unit_tests/_transformers/test_model_init.py→ 39 passed (37 existing + 2 new), 0 regressions.Notes
(recipe switched deepep→hybridep); no change needed here.
GLM-5.1 for a small-footprint smoke.