feat: video + audio understanding GRPO training recipe #2823

Merged

yuki-97 merged 40 commits into

NVIDIA-NeMo:mainfrom

yuekaizhang:audio_video

Jul 2, 2026

Commits on Jun 10, 2026

feat(eval): support Daily-Omni + Qwen2.5-Omni eval
yuekaizhang
committed
feat(grpo): add audio+video Intent GRPO recipe for Qwen2.5-Omni-3B
yuekaizhang
committed
fix(grpo-intent): use two-step processor call for audio+video samples
yuekaizhang
committed
fix(grpo-intent): pass prompt_token_ids to vLLM for audio+video samples
yuekaizhang
committed
fix(grpo-intent): pass audio + video as independent streams (no use_audio_in_video)
yuekaizhang
committed
docs(grpo-intent): align comments + tests with verified independent-streams path
yuekaizhang
committed
docs(grpo-intent): match guide to verified independent-streams smoke run
yuekaizhang
committed
feat(grpo): expose per-component reward metrics in VLM validation
yuekaizhang
committed
docs(grpo-intent): match Results section to per-component validation metrics
yuekaizhang
committed
fix(grpo-intent): explicit think+answer prompt + fallback reward to bootstrap GRPO
yuekaizhang
committed

Commits on Jun 15, 2026

Commits on Jun 23, 2026

Merge branch 'main' into audio_video
yuekaizhang
authored

Commits on Jun 24, 2026

Merge branch 'main' into audio_video
yuekaizhang
authored

Commits on Jun 26, 2026

Commits on Jun 29, 2026

Commits on Jun 30, 2026