Backpropagation Algorithm - Step by Step

By the end of this lesson, you will be able to...

• trace a backward pass through one affine layer;

• distinguish parameter gradients from the signal passed to an earlier layer;

• separate gradient calculation from the optimizer update.

Idea in one sentence: Backpropagation reuses the error gradient already computed downstream, so each layer can calculate its own gradients without deriving the whole network again.

Core Insight

A loss says that the inspection model is wrong. The chain rule explains how one weight influenced that loss. The next practical question is how to do this for every weight without starting from the full formula each time.

Backpropagation is a reverse walk through the operations that produced the loss. A layer receives the derivative with respect to its output. It combines that incoming signal with cached forward values and its local rule. It returns gradients for its parameters and a new incoming signal for the layer before it.

That repeated local contract is the algorithm. The numbers below are a teaching example, not the output of a trained system.

The Layer Contract

Use an affine layer with batch rows:

Z = X @ W + b
L = mean((Z - T)^2)

Here X has shape (B, D), W has shape (D, O), and Z has shape (B, O). The forward pass must cache X and any values a local derivative requires.

If the backward pass receives dZ = dL/dZ, the affine rule is:

dW = X.T @ dZ
db = sum_rows(dZ)
dX = dZ @ W.T

dW and db belong to trainable parameters. dX is not normally used to change the data; it carries the loss signal to the operation that created X.

The shapes explain why the formulas have this form. dZ has one row per example and one column per output, so it has shape (B, O). Transposing X makes its shape (D, B). Therefore X.T @ dZ has shape (D, O), exactly the shape of W. Each entry of dW answers a local question: if this input feature and this output unit meet in one weight, how did all examples in the batch vote to change it?

The bias has no input-feature axis. The same bias value is added to every row, so every row that depends on it contributes to the same derivative. Summing the batch rows reduces (B, O) to (O). Conversely, dX = dZ @ W.T has shape (B, D): the layer sends one sensitivity value back for every input feature of every example. This is a teaching model for a dense layer; a convolution, normalization layer, or attention operation has different local formulas but follows the same receive-compute-send pattern.

It is useful to keep two ideas separate. The batch aggregation happens in dW and db, because parameters are shared across examples. The incoming dZ still keeps examples separate. If a value in row two is wrong, it must not silently become evidence about row one before the layer has applied its local rule.

A Complete Two-Example Trace

Let:

X = [[1], [3]]
T = [[2], [1]]
W = [[0.5]]
b = [0]

Forward evaluation gives:

Z = [[0.5], [1.5]]
Z - T = [[-1.5], [0.5]]
L = ((-1.5)^2 + 0.5^2) / 2 = 1.25

For mean squared error, the gradient arriving at the layer output is:

dZ = 2*(Z - T)/B
   = [[-1.5], [0.5]]

Now perform the backward rules:

dW = [[1, 3]] @ [[-1.5], [0.5]]
   = [[0]]

db = -1.5 + 0.5
   = [-1]

dX = [[-1.5], [0.5]] @ [[0.5]]
   = [[-0.75], [0.25]]

The zero weight gradient is not a bug. The first example contributes -1.5; the second contributes 3*0.5 = 1.5. These contributions cancel for this shared weight on this batch. The bias gradient remains negative, so the batch still asks for a bias increase.

Here are the parameter contributions before the layer aggregates them. They make the cancellation visible rather than mysterious:

This does not mean the examples agree or that the model is correct. It means that, at this particular point, their first-order requests for this one weight point in opposite directions with equal size. A different batch can give a nonzero dW; a larger batch can reduce some random variation but can still contain real disagreement. Do not remove a gradient merely because it cancels: inspect the per-example or per-batch contributions when the result is surprising.

So far, the layer has received one downstream signal and emitted three results:

incoming dZ
  -> dW and db for its parameters
  -> dX for the preceding operation

How ReLU Fits Between Two Affine Layers

Real networks place an activation between affine layers. Suppose an affine layer produced the cached preactivation Z, and the next forward operation was:

A = ReLU(Z)

ReLU keeps positive values and replaces negative values with zero. In the backward direction, it has one local decision: an input that was not allowed through in the forward pass does not receive a gradient through this simplified derivative. If the next layer sends dA, then:

dZ = dA * 1[Z > 0]

The indicator is 1 where the cached Z was positive and 0 otherwise. For the illustrative values below:

Z  = [[-0.2], [0.4]]
dA = [[ 0.7], [-0.3]]
mask = [[0], [1]]
dZ = [[0], [-0.3]]

The negative preactivation gets no backward signal through ReLU. The positive one passes the incoming signal unchanged. This is why the forward pass must cache Z, not only the final loss: the local backward rule needs to know which branch occurred. At exactly zero, common libraries choose a convention for the derivative; the toy rule above does not resolve that boundary. Once ReLU has produced dZ, the affine layer before it uses the same dW, db, and dX contract already traced above.

The Optimizer Comes After Backpropagation

Backpropagation has now computed gradients; it has not changed parameters. With gradient descent and eta = 0.1:

W_new = W - eta*dW = 0.5
b_new = b - eta*db = 0.1

The new scores are [[0.6], [1.6]]. Checking the same batch explicitly gives errors [-1.4, 0.6], so its mean squared error is (1.96 + 0.36) / 2 = 1.16, down from 1.25. This is evidence for this chosen batch and step, not a promise of improvement on all data. A larger learning rate could raise the loss by stepping too far; another batch can point in a different direction. Optimizers such as SGD or Adam decide how gradients become updates; they do not replace the backward pass.

When a Backward Trace Is Trustworthy

Manual backward code can be almost right and still train badly. Test the smallest local operation before placing it inside a network. For an affine layer, check that the cached X has the forward shape, that dZ has the output shape, and that dW, db, and dX have the shapes of W, b, and X. A shape that broadcasts without an error is not automatically the intended shape.

Then compare one analytical derivative with a finite-difference estimate. Hold every other value fixed and perturb one scalar weight by a small epsilon:

numerical dL/dw ≈ (L(w + epsilon) - L(w - epsilon)) / (2 * epsilon)

For the first trace, dW is zero. If a small symmetric perturbation changes the loss equally in both directions, the numerical estimate should be close to zero too. This check does not prove an entire network correct, but it is strong evidence that one local formula and its reduction are wired correctly. Pick epsilon carefully: too large measures curvature rather than a local slope; too small can expose floating-point rounding. Compare with a tolerance rather than demanding identical decimal strings.

For a compact debugging routine, use this order:

1. Confirm the forward output and loss on a tiny synthetic batch.
2. Print shapes and reductions in the backward pass.
3. Compare one or two parameter entries with finite differences.
4. Make one deliberately small optimizer step and remeasure the same loss.

If step four fails while the numerical comparison succeeds, the likely fault may be update sign, learning rate, or a parameter not actually being updated—not the local derivative itself.

Test dW and db independently when possible. A mistaken row reduction can leave one correct while corrupting the other, especially when batch size is one and summing or averaging happens to look harmless. Use at least two examples for this local test so the batch dimension has something real to reveal.

Costs, Limits, and Signals

Backpropagation buys efficient reuse of downstream derivatives. The trade-off is memory and bookkeeping: save forward values and use memory, or recompute them and use more time. Every custom operation also needs a correct local backward rule. Long networks make that trade-off visible because many intermediate tensors may need to survive until the reverse walk.

A correct derivative does not prove a useful model. The loss can be mismatched, a batch can cancel gradients, and a learning rate can overshoot. Inspect shapes, loss before and after a controlled update, and gradient norms by layer. A numerical gradient check is especially useful for small custom implementations.

Common Confusions

Confusion: Backpropagation updates weights.

Better model: it computes gradients; an optimizer applies updates afterward.

Confusion: A zero gradient means training is finished.

Better model: it can be cancellation on one batch, a flat local region, or an implementation problem.

Confusion: dX is irrelevant because inputs are data.

Better model: dX is the upstream gradient that lets earlier layers continue the chain rule.

Check Your Understanding

Check: Why does db sum the rows of dZ?

Answer: One bias is shared across all examples, so each row contributes to its derivative. Summing rows preserves the output dimension: (B, O) becomes (O).

Check: Why can loss be nonzero while dW is zero?

Answer: Different examples can make equal and opposite contributions to that one parameter gradient. In the worked trace, -1.5 + 1.5 = 0 for the weight even though the loss remains 1.25.

Check: A ReLU cached Z = [-2, 3] and receives dA = [4, 4]. What dZ does it send to its preceding affine layer?

Think first, then reveal.

Answer: [0, 4]. The first preactivation was negative, so the ReLU mask blocks that local path; the second was positive, so its incoming gradient passes through.

Trace It Yourself

Use X = [[2], [4]], T = [[1], [3]], W = [[0.5]], and b = [0].

1. Compute Z and dZ.
2. Compute dW and db.
3. State which parameters gradient descent increases with eta = 0.1.

Model answer: Z = [[1], [2]], dZ = [[0], [-1]], dW = -4, and db = -1. Gradient descent increases both W and b: with eta = 0.1, they become 0.9 and 0.1. The resulting scores are [[1.9], [3.7]]; this step helps the second example but makes the first worse, a useful reminder that a batch update is a compromise. Before trusting manual code, compare the analytical dW with a symmetric numerical estimate while holding b fixed.

Resources

• [DOCUMENTATION] PyTorch: Autograd mechanics — Focus: saved forward values and local backward functions.
• [TUTORIAL] CS231n: Backpropagation, Intuitions — Focus: local gradient flow through larger graphs.
• [BOOK] Deep Learning, Chapter 6 — Focus: feedforward-network gradients.

Key Takeaways

• A layer converts its incoming gradient into parameter gradients and an upstream gradient using cached forward values.
• Backpropagation computes gradients; an optimizer separately changes parameters.
• Shared parameters aggregate evidence across the batch, so cancellation can be meaningful without making the loss zero.
• Inspect shapes, finite-difference checks, gradients, and controlled loss changes instead of assuming correct-looking code is correct.