Skip to content

fix(model): apply dict config overrides with custom dispatch#2657

Merged
HuiyingLi merged 1 commit into
mainfrom
akoumparouli/nvbug6259955-glm5_1-config-override
Jun 20, 2026
Merged

fix(model): apply dict config overrides with custom dispatch#2657
HuiyingLi merged 1 commit into
mainfrom
akoumparouli/nvbug6259955-glm5_1-config-override

Conversation

@akoumpa

@akoumpa akoumpa commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

NVBug: 6259955

What

get_hf_config / __init_model (nemo_automodel/_transformers/model_init.py) now treat a plain-dict
config kwarg — how the config loader materializes --model.config.* CLI overrides (and YAML
model.config: maps) — as config overrides rather than a finished config object.

  • get_hf_config: a dict config is popped and its keys folded into the kwargs AutoConfig.from_pretrained
    applies as overrides, so a real config (e.g. GlmMoeDsaConfig with num_hidden_layers=N) is built. Its
    architectures survive, so custom-model dispatch is preserved.
  • __init_model: the now-consumed dict config is dropped from kwargs so it isn't forwarded into the model
    constructor (which receives the config positionally).
    Adds two regression tests in tests/unit_tests/_transformers/test_model_init.py.

Why

Fixes NVBugs 6259955 Defect 2. Passing --model.config.num_hidden_layers 16 (the natural shrink knob for
GLM-5.1 / GlmMoeDsaForCausalLM) crashed with:

TypeError: GlmMoeDsaForCausalLM.__init__() got an unexpected keyword argument 'backend'

Root cause: the override was materialized as a dict and returned verbatim by get_hf_config.
get_architectures(dict) returns [], so _resolve_custom_model_cls_for_config returned None and
dispatch flipped from the custom GlmMoeDsa implementation to the stock-HF path — which forwards the
custom-only backend kwarg into the built-in transformers GlmMoeDsaForCausalLM.__init__ and rejects it.
(PR #1784 coerced dict→BackendConfig only on the custom path, so it didn't cover this stock-HF fallback.)
This made it impossible to shrink GLM-5.1 below its 256-H100 footprint via the supported CLI. The fix is
the report's preferred remedy: --model.config.* overrides no longer change which implementation is
dispatched, and they are correctly applied. Affects every registered custom model that accepts custom-only
constructor kwargs, not just GlmMoeDsa.

How tested

On cw-dfw (H100) in the rc6 container, latest main 2b96596a, transformers 5.8.1:

  • Weightless repro driving NeMoAutoModelForCausalLM.from_pretrained on a tiny GlmMoeDsa config (no
    GLM-5.1 download): before → control custom path OK, override → TypeError: ... 'backend' (exact bug
    traceback); after → control unchanged, override keeps the custom path and applies the override (1 layer
    built vs the file's 2), exit 0.
  • pytest tests/unit_tests/_transformers/test_model_init.py → 39 passed (37 existing + 2 new), 0 regressions.

Notes

…fig.* is a dict

NVBugs: 6259955
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa akoumpa requested a review from a team as a code owner June 20, 2026 02:06
@akoumpa akoumpa added the r0.5.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. label Jun 20, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jun 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@akoumpa

akoumpa commented Jun 20, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 6e9fdb5

@HuiyingLi HuiyingLi changed the title fix(model): keep custom dispatch and apply overrides when --model.config.* is a dict fix(model): keep custom dispatch and apply overrides when --model.config.* is dict Jun 20, 2026
@HuiyingLi HuiyingLi changed the title fix(model): keep custom dispatch and apply overrides when --model.config.* is dict fix(model): keep custom dispatch and apply overrides when model.config.* is dict Jun 20, 2026
@HuiyingLi HuiyingLi enabled auto-merge (squash) June 20, 2026 04:07
@HuiyingLi HuiyingLi changed the title fix(model): keep custom dispatch and apply overrides when model.config.* is dict fix(model): apply dict config overrides with custom dispatch Jun 20, 2026
@HuiyingLi HuiyingLi merged commit de2e0e6 into main Jun 20, 2026
82 of 86 checks passed
@HuiyingLi HuiyingLi deleted the akoumparouli/nvbug6259955-glm5_1-config-override branch June 20, 2026 05:43
akoumpa added a commit that referenced this pull request Jun 20, 2026
fix(model): apply dict config overrides with custom dispatch (#2657)

fix(model): keep custom dispatch and apply overrides when --model.config.* is a dict

NVBugs: 6259955

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

r0.5.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants