Skip to content

Fix RL LR schedule default#4392

Open
AntonyMei wants to merge 1 commit into
mainfrom
fix-rl-lr-schedule-default
Open

Fix RL LR schedule default#4392
AntonyMei wants to merge 1 commit into
mainfrom
fix-rl-lr-schedule-default

Conversation

@AntonyMei

Copy link
Copy Markdown
Collaborator

RL computes its real training length from num_batches, rl.num_iterations, train_fraction, and num_epoch, but rl.yml inherited base.yml steps=150001. Config normalization then converted learning_rate_schedule_steps=-1 into that inherited step count before the RL optimizer saw max_train_steps, causing warmup/decay to use a much longer horizon than the actual RL run.

Set both steps and learning_rate_schedule_steps to -1 in the RL base config so the optimizer falls back to the computed max_train_steps unless a recipe explicitly pins the schedule horizon. Also warn when a positive learning_rate_schedule_steps differs from the computed RL training length, since that may be intentional but is easy to inherit accidentally.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

RL computes its real training length from num_batches, rl.num_iterations, train_fraction, and num_epoch, but rl.yml inherited base.yml steps=150001. Config normalization then converted learning_rate_schedule_steps=-1 into that inherited step count before the RL optimizer saw max_train_steps, causing warmup/decay to use a much longer horizon than the actual RL run.

Set both steps and learning_rate_schedule_steps to -1 in the RL base config so the optimizer falls back to the computed max_train_steps unless a recipe explicitly pins the schedule horizon. Also warn when a positive learning_rate_schedule_steps differs from the computed RL training length, since that may be intentional but is easy to inherit accidentally.
@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant