Learning Rate Schedules

By the end of this lesson, you will be able to...

• read loss and validation evidence to identify a learning-rate phase problem;

• compare schedule families by the training behavior they assume;

• choose a bounded schedule experiment rather than applying decay by ritual.

Idea in one sentence: A learning-rate schedule changes how aggressively an optimizer uses valid gradients as training moves from unstable exploration toward local refinement.

Core Insight

A classifier is training for 30 epochs with Adam and a fixed learning rate of 0.01. Its training loss drops from 0.90 to 0.34 in the first ten epochs. From epoch 11 onward it alternates between 0.28 and 0.36; validation loss also stops improving. The gradient check passed and update norms are nonzero. The question is no longer whether gradients exist, but whether the same step size still fits the current phase.

The tempting model is to pick one learning rate at the start and call it a property of the model. That works when the run is short and the same step size remains useful. It becomes a compromise when early progress needs movement but late progress needs precision.

A schedule is a policy for eta(t), the learning rate at step or epoch t. It does not change the optimizer's momentum or adaptive state; it changes the scale applied to its update. The trace here is illustrative evidence from a controlled run, not a universal recipe.

The Fixed-Rate Compromise

With a fixed rate, the same eta must solve two different jobs:

early: move through an unfamiliar region and make visible progress
late:  avoid jumping past a locally useful region

For the classifier above, 0.01 was useful early: loss fell quickly. Later, the repeated loss swings suggest that the updates are still too large to settle on this batch-and-model combination. Reducing eta at epoch zero might make the later phase calmer, but it would also make the first ten epochs slower. Keeping it fixed preserves speed but may waste the remaining budget in oscillation.

This is not proof that a schedule is the only explanation. Oscillation can also come from data noise, a too-small validation set, a bad loss, or unstable normalization. Inspect the training and validation curves together. A schedule is justified when a controlled comparison supports the claim that the training phase—not a broken upstream mechanism—is the pressure.

Four Policies for One Training Budget

Assume a 30-epoch budget and a peak rate of 0.01. Each policy encodes a different belief about what should happen next.

Warmup is not merely another way to decay. It starts below the target rate and ramps upward first:

epochs 0–2:  0.001 -> 0.005 -> 0.01
epochs 3–30: use the selected decay policy

Warmup is a response to a different symptom: instability at the beginning, such as loss spikes or invalid values immediately after training starts. It protects early optimizer state and parameter updates from a full-strength rate before the run has settled. It does not automatically fix late oscillation.

Make the Choice from Evidence

Return to the 30-epoch classifier. The evidence is: fast initial loss reduction, later oscillation, no gradient-check failure, and no early loss spike. A compact first experiment is step decay at epoch 10:

epochs 0–9:   eta = 0.01
epochs 10–29: eta = 0.001

Why this design? It preserves the observed useful early rate, then tests whether a tenfold smaller late update reduces oscillation. Do not call it successful merely because training loss becomes smoother. Compare validation loss, accuracy or task metric, and the size of parameter updates under the same seed, data split, optimizer, and epoch budget.

If the curve remains noisy but validation improves, the noise may be acceptable minibatch variation rather than harmful instability. If both losses flatten immediately after epoch 10, decay may be too early or too severe. If training diverges before epoch 2, step decay at epoch 10 is irrelevant; lower the initial rate or test warmup first.

This is the design loop:

observed phase signal
-> smallest schedule change that targets it
-> controlled comparison
-> validation evidence
-> keep, revise, or reject the policy

Trace a Schedule, Not Just Its Name

The schedule must be applied at a defined time. For an epoch-level step policy:

for epoch in range(30):
    if epoch == 10:
        optimizer.learning_rate = 0.001
    train_one_epoch()
    evaluate_validation()

For a continuous policy, calculate the rate from the current step instead. A simple cosine decay from eta_max toward eta_min over T steps is:

eta(t) = eta_min + 0.5 * (eta_max - eta_min) *
         (1 + cos(pi * t / T))

At t = 0, the cosine term is one, so the rate is eta_max. At t = T, the cosine term is minus one, so it reaches eta_min. The formula is not decoration: it gives a smooth change with no abrupt epoch boundary. It is a better fit when the budget is known and a gradual refinement policy is desired; it is not inherently superior to a step change earned by a visible phase transition.

The trade-off is clear. Rich schedules can match a known budget or an observed phase transition more closely, but they add parameters, timing decisions, and more ways to overfit a single run. A constant rate or one step drop is often the better first experiment when interpretability and limited tuning budget matter.

Boundaries and Signals

A schedule cannot repair incorrect gradients. If the loss becomes NaN, update norms explode from the first steps, or the gradient check fails, return to the earlier debugging loop before adding schedule complexity.

A lower rate can hide a problem. It may make loss curves look calm while the model barely learns. Watch both loss reduction and validation evidence; a smooth flat curve is not a success signal.

Scheduler timing is part of the contract. Applying a scheduler per batch rather than per epoch changes the policy substantially. Record the unit, peak rate, final rate, warmup length, decay rule, and total step budget with the run.

Validation is the boundary on training loss. A schedule that lowers training loss but worsens validation may be fitting the training set more aggressively, not improving the task. Later lessons on regularization will make that distinction more explicit.

When comparing schedules, keep the total number of optimizer steps constant. Otherwise a schedule can appear better merely because it received more updates or a longer warmup, rather than because its rate policy matched the observed phase.

Check Your Understanding

Check: A run has loss spikes in the first 50 steps but settles afterward. Which schedule element is the first plausible experiment?

Think first, then reveal.

Answer: Warmup or a lower initial rate. The pressure is early instability, so a late decay does not target it.

Check: A run improves through epoch 20, then stops improving after a decay at epoch 10. What is one plausible explanation?

Answer: The decay may have been too early or too aggressive. Compare the same run with a later or gentler decay while holding other conditions fixed.

Check: Why is a scheduler call's timing important?

Answer: A rate reduced once per epoch is very different from the same reduction applied every batch. The schedule is defined over time units, not only by its name.

Design a Small Schedule Experiment

You have a 40-epoch run. Training loss falls rapidly for 12 epochs, then oscillates; validation loss improves until epoch 14 and stays flat afterward. There are no NaN values and the gradient check already passed.

Propose a first schedule experiment. State the schedule, the evidence that motivates it, what you will hold fixed, and what result would make you reject it.

Model answer: Use a step decay from 0.01 to 0.001 around epoch 12–14, because the training curve begins oscillating while validation improvement slows. Hold seed, data split, optimizer, model, batch size, and 40-epoch budget fixed. Compare validation loss or metric, not training loss alone. Reject the schedule if validation does not improve, if learning stalls immediately after decay, or if a simpler constant-rate baseline performs as well. This is a bounded experiment, not a claim that step decay is universally best.

Resources

• [DOCUMENTATION] PyTorch: how to adjust learning rate — Focus: scheduler timing and optimizer integration.
• [PAPER] SGDR: Stochastic Gradient Descent with Warm Restarts — Focus: cosine-style scheduling and restart motivation.
• [DOCUMENTATION] PyTorch: OneCycleLR — Focus: a phase-based schedule with explicit step timing.
• [BOOK] Deep Learning, Chapter 8 — Focus: step size and optimization behavior.

Key Takeaways

• A schedule is a time policy for the size of valid optimizer updates, not a replacement for gradient correctness.
• Warmup addresses early instability; decay addresses a need for smaller late-stage movement.
• Choose the simplest schedule that targets an observed failure signal, then compare validation evidence under controlled conditions.
• Record scheduler timing and budget, because changing rates per batch or per epoch produces different training policies.