Learning Rate Schedules

The learning rate controls how strongly an optimizer updates model parameters using gradient information. A learning rate schedule changes that rate over time instead of keeping it constant throughout training. Schedules may depend on optimizer steps, completed epochs, or a monitored metric such as validation loss. They are useful because a learning rate that enables rapid progress early in training may be too large for stable refinement later.

Common approaches include step decay, exponential decay, cosine decay, and reductions triggered by a metric plateau. Step decay lowers the learning rate at specified milestones, while cosine decay changes it smoothly over a defined training horizon. Warmup gradually increases the learning rate at the beginning, often before a decay phase. Cyclical schedules and warm restarts intentionally increase the rate again, so scheduling does not necessarily mean continuously decreasing it.

When debugging convergence, inspect training loss, validation loss, the actual learning rate, and, when available, gradient norms together. Strong oscillations or rapidly increasing loss can indicate an excessive learning rate, although bad inputs, exploding gradients, and numerical overflow can produce similar symptoms. Very slow improvement can indicate a learning rate that is too small or a schedule that decays too early. If training loss improves while validation loss worsens, investigate overfitting rather than assuming that a lower learning rate will solve the problem.

For example, a transformer fine-tuning job might increase the learning rate over the first 5% of optimizer steps and then decay it over the remaining steps. This is a starting configuration, not a universal prescription. A smaller dataset, different optimizer, or partially frozen model can require a different peak rate and warmup duration. For a classification model whose validation loss has stopped improving, a plateau-based scheduler can reduce the rate after a configured patience interval. Its improvement threshold and patience should tolerate ordinary metric noise.

In cloud training, the schedule usually runs inside the training framework or training container, while the managed service provides compute, job orchestration, metrics, and artifact storage. Expose schedule settings as job hyperparameters and compare runs using the same data split and evaluation metric. Log the learning rate explicitly; a configured schedule is not proof that the training loop applied it correctly. Hyperparameter search can compare peak rates and decay settings, but each trial consumes compute and must have enough budget to reveal meaningful behavior.

Implementation details frequently explain apparent convergence failures. A scheduler intended to advance once per epoch behaves very differently if advanced after every minibatch. With gradient accumulation, define step-based schedules in terms of optimizer updates rather than every forward pass. Follow the framework’s documented scheduler call order. When resuming an interrupted job, restore model, optimizer, scheduler, and global-step state together so that the learning rate does not unintentionally restart.

Adaptive optimizers such as Adam adjust parameter-wise update scaling, but they can still benefit from a scheduled base learning rate. Likewise, gradient clipping and mixed-precision loss scaling address different problems and do not replace schedule selection. Changing the global batch size or training duration changes the optimization context, so reassess the schedule instead of copying it unchanged. Evaluate convergence speed, held-out quality, and compute cost together rather than judging a schedule only by the lowest training loss.