Gradient Descent Fundamentals

By the end of this lesson, you will be able to...

• trace how a gradient and a learning rate change one model parameter;

• predict whether a parameter update moves toward or away from lower loss;

• explain how batch size changes the evidence used for each update without changing the goal of training.

Idea in one sentence: Gradient descent repeatedly measures which small parameter change would lower the current loss, then takes a controlled step in that direction.

Core Insight

Lesson 002 gave us two candidate lines and a way to compare them. Candidate A had a lower squared-error total than candidate B. That leaves a practical question:

Instead of guessing many possible weights by hand, how can training find a better weight?

The tempting answer is “try a random new number whenever the prediction is wrong.” That works only as a vague hope. A parameter can be wrong in many ways, and changing it in the wrong direction makes the loss larger.

Gradient descent adds local guidance. At the current parameter value, it asks how the loss would change if the parameter increased a little. That local rate of change is the gradient. Training then moves in the opposite direction, because the gradient points uphill and we want lower loss.

current parameters
-> predictions
-> loss
-> gradient: which way is uphill here?
-> small step downhill
-> repeat

This is a teaching model of training, not a promise that every model will become good. It explains how parameters move. Whether the data, loss, representation, and evaluation are appropriate remains a separate question.

One Weight, One Prediction, One Clear Pressure

Keep the same quiz-score setting from the preceding lesson. To isolate the update mechanism, make one deliberately small assumption:

• the model intercept is fixed at 48;
• one student completed 3 practice sessions;
• that student's observed quiz score is 63;
• only the weight w may change.

The model is:

ŷ = 48 + 3w

If w = 0, the prediction is 48. It misses the observed score by 15 points. Use squared error as the loss for this one example:

L(w) = (ŷ − y)²
     = (48 + 3w − 63)²
     = 9(w − 5)²

The loss is smallest when w = 5. We can see that algebraically in this toy case, but imagine we did not solve it directly. Training needs a rule for improving its current guess.

The loss has a bowl shape:

loss
 ^
 |  \                 /
 |   \               /
 |    \             /
 |     \_____*_____/
 +----------------------> weight w
             5

To the left of 5, increasing w lowers loss. To the right, decreasing w lowers loss. The gradient tells us which of those situations we are in.

The Update Rule, Read One Piece at a Time

For this loss, the derivative with respect to w is:

dL/dw = 18(w − 5)

This derivative is the one-parameter gradient. Do not treat it as a formula to memorize. It earns its place because its sign answers a concrete question:

• if dL/dw is negative, increasing w lowers the loss locally;
• if dL/dw is positive, decreasing w lowers the loss locally;
• if it is zero, this toy loss is at its bottom.

The update rule is:

w_next = w_current − η × dL/dw

η (eta) is the learning rate. It scales the size of the change.

The minus sign is essential. A positive gradient means “loss rises as w rises,” so subtracting it moves w down. A negative gradient means “loss falls as w rises,” so subtracting a negative number moves w up.

In models with many parameters, the same rule applies to a vector of parameters:

parameters_next = parameters_current − learning_rate × gradient

The vector form is the same mechanism in more directions. We will keep one weight so that every update stays inspectable.

A Three-Update Trace

Set the learning rate to η = 0.05 and start at w = 0. All numbers in this trace are calculated from the toy one-example model above.

Let’s unpack the first row.

1. At w = 0, the model predicts 48; its squared error is 225.
2. The gradient is -90. The negative sign says that a larger weight will move downhill from this starting point.
3. The update is 0 − 0.05 × (-90) = 4.5.
4. At w = 4.5, the prediction becomes 61.5 and the loss drops to 2.25.

The next two updates are smaller because the gradient is smaller near the bottom. This is not the optimizer “getting tired.” It is seeing less local slope, so the same learning rate produces a smaller change.

So far, we have a real mechanism rather than a slogan: predictions create a loss; the loss creates a gradient; the gradient and learning rate create the next parameter value. The loop repeats until improvement becomes small enough under a chosen stopping rule.

Why the Learning Rate Can Help or Ruin the Trace

The gradient gives direction, not permission to take an arbitrary-sized jump. The learning rate decides how far the update travels.

With the same starting point, try η = 0.20 instead of 0.05:

start:  w = 0,    gradient = -90,  next w = 18
next:   w = 18,   gradient = 234,  next w = -28.8

The first step jumps past the bottom at w = 5. The second jumps far back in the other direction. The loss grows rather than settling. This is overshooting.

The opposite failure is a learning rate so small that the loss falls only a tiny amount per update. Training may eventually improve, but the compute and time budget can become unreasonable.

Trade-off: A larger learning rate can make early progress faster, but it costs stability and can overshoot. A smaller one gives controlled motion, but it costs iterations. This can still fail when the inputs are badly scaled, the gradient is noisy, or the chosen loss does not represent the real decision. The signal to watch is the loss curve: steady decline suggests useful progress; persistent oscillation or growth suggests that the update settings or model deserve inspection.

There is no universal best learning rate. It depends on the model, data, parameter scale, and training setup. A value that behaves well in this one-weight example is not a reusable magic number.

What a Batch Changes—and What It Does Not

The trace used one example so that the arithmetic stayed small. Real training usually estimates an update from several examples. The batch size is how many examples contribute before parameters are updated.

Return to the four rows from lesson 002:

The objective does not change: reduce the chosen loss over training examples. What changes is the estimate of the gradient and the number of updates. One example can point in an unusual direction; averaging more examples usually makes a less noisy estimate, but requires more work before each update.

For a dataset of 1,000 examples trained for one pass:

full batch of 1,000 -> 1 update
mini-batches of 100 -> 10 updates
batch size 1 (stochastic) -> 1,000 updates

Those counts alone do not choose the right setup. Hardware, dataset size, model, memory limits, and the observed training behavior all matter. “Mini-batch is common” is a practical observation, not a mathematical guarantee.

What Gradient Descent Does Not Decide

Gradient descent adjusts parameters for the loss it receives. It does not choose the label, repair leakage, establish causation, or prove that low training loss will generalize.

That distinction matters in the support scenario. If practice_sessions is recorded incorrectly, or if the real use case needs a model before sessions can be observed, a perfectly executed update loop still learns the wrong operational problem. If the loss falls on training rows but rises on unseen rows, the issue is not “the gradient failed”; it may be an overfitting, representation, or evaluation problem.

This mechanism also has mathematical limits. The bowl-shaped toy loss has one obvious minimum. Some richer losses have many flat regions or locally low areas, and optimization behavior becomes harder to diagnose. This track uses gradient descent to make parameter updates visible; it does not attempt a formal optimization proof.

Check Your Understanding

Check 1: At a current weight, dL/dw = -6. Ignoring the exact loss, should a gradient-descent update with a positive learning rate increase or decrease the weight?

Think first, then reveal.

Answer: Increase it. The update subtracts a negative number: w_next = w_current − η × (-6). Locally, raising the weight moves toward lower loss.

Check 2: A loss curve falls for several steps, then repeatedly rises and falls by large amounts. What is one plausible explanation?

Think first, then reveal.

Answer: The learning rate may be too large, causing overshooting. Noisy small batches or an unstable setup can also create variation, so inspect the update settings and data rather than diagnosing from one step alone.

Check 3: Does a mini-batch change the model's target from “lower loss” to “higher accuracy”?

Think first, then reveal.

Answer: No. It changes how many examples estimate each gradient before an update. The training objective is still the chosen loss.

Practice: Trace a Different Step Size

Use the same toy loss:

L(w) = 9(w − 5)²
dL/dw = 18(w − 5)

Start with w = 4 and η = 0.05.

1. Calculate the gradient.
2. Calculate the next weight.
3. Calculate the old and new loss.
4. Explain why the update moves in the direction it does.
5. Name one reason this successful one-example update is not enough evidence to deploy a student-support model.

Self-check: At w = 4, the gradient is 18 × (4 − 5) = -18. The next weight is 4 − 0.05 × (-18) = 4.9. The loss falls from 9 to 0.09. The negative gradient means increasing w moves downhill locally. One example cannot establish data quality, generalization, or a useful intervention policy.

Connections and Next Step

Gradient descent turns the static regression equation from lesson 002 into a training process. It explains how a chosen loss changes weights; it does not decide whether the model has the right capacity.

The next lesson, Polynomial Features and Regularization, changes the representation and adds a constraint on fitting. That creates a more important comparison than “did training loss fall?”: which model behavior is likely to remain useful on unseen examples?

Resources

• [TUTORIAL] Google Machine Learning Crash Course: Gradient descent — Focus: the repeated loss, direction, and update loop for linear regression.
• [TUTORIAL] Google Machine Learning Crash Course: Hyperparameters — Focus: how learning rate and batch size affect update size, noise, and convergence.
• [INTERACTIVE] Google Machine Learning Crash Course: Gradient descent exercise — Focus: observe slow convergence and overshooting by changing the learning rate.

Key Takeaways

1. A gradient describes the local uphill direction of loss; gradient descent subtracts it to move parameters downhill.
2. The learning rate controls step size: too small wastes updates, while too large can overshoot and destabilize training.
3. Batch size changes the evidence used to estimate each update, trading per-step cost against gradient noise.
4. Lower training loss shows progress on a chosen objective; it does not validate the data, decision, or future performance.