Implementing Backpropagation from Scratch

By the end of this lesson, you will be able to...

• implement the forward, loss, backward, and update phases of a tiny dense network;

• assign each cached value and gradient to the layer that owns it;

• locate a shape, cache, or update-order bug in a manual backward pass.

Idea in one sentence: A hand-built network becomes understandable when every layer saves exactly what its local derivative needs, returns the gradient for its input, and waits to update its parameters until the whole backward pass is complete.

Core Insight

Calling backward() in a framework can make a network look like one large operation. A manual implementation exposes the smaller agreement underneath: each layer accepts a value in the forward direction, caches a local fact, and later accepts a gradient in the reverse direction.

Consider a tiny classifier that should label [1, 0] as positive and [0, 1] as negative. A reasonable first attempt is to put all the equations in one training function and update each weight as soon as its gradient appears. It can work for a one-layer example. It breaks as soon as an earlier layer needs the old weights of a later layer to compute its incoming gradient. The code may run and still use the wrong computational graph.

The stronger model is to give every operation a forward contract, a backward contract, and ownership of its cache. The numbers below are an illustrative teaching case, not a trained model or a recommended architecture.

The Small Network and Its Owners

Our network has two affine layers with a ReLU between them:

X -> Linear 1 -> Z1 -> ReLU -> A1 -> Linear 2 -> logits -> binary loss

Use batch rows. Linear 1 owns W1, b1, and the input X that it must cache. ReLU owns its preactivation Z1, because its backward rule needs to know which entries were positive. Linear 2 owns W2, b2, and A1.

For binary classification, use a loss written directly from logits. It combines the final sigmoid and binary cross-entropy in a numerically stable interface:

loss = mean(log(1 + exp(logits)) - y * logits)
dlogits = (sigmoid(logits) - y) / B

B is the number of batch rows. The first line is a loss value; the second line is the derivative that begins the backward pass. Keeping them together avoids a common mismatch: computing a mean loss but forgetting its division by B in the initial gradient.

Here is the shape ledger for this exact network. Write this before relying on any matrix multiplication.

The @ operator is matrix multiplication, not elementwise multiplication. Its inner dimensions must match. A shape error is useful evidence; broadcasting that happens to succeed is not evidence that the intended derivative was computed.

Give Each Layer a Small Contract

The following NumPy-like code deliberately does not use automatic differentiation. It is short enough to inspect, but each cache and gradient has one clear owner.

class Linear:
    def __init__(self, W, b):
        self.W = W
        self.b = b

    def forward(self, x):
        self.x = x                 # cache owned by this layer
        return x @ self.W + self.b

    def backward(self, dout):
        self.dW = self.x.T @ dout
        self.db = dout.sum(axis=0, keepdims=True)
        return dout @ self.W.T     # gradient for the preceding operation


class ReLU:
    def forward(self, z):
        self.z = z                 # cache: the branch chosen in forward
        return maximum(z, 0)

    def backward(self, dout):
        return dout * (self.z > 0)


def sigmoid(z):
    return 1 / (1 + exp(-z))


def bce_with_logits(logits, y):
    # logaddexp is a stable spelling of log(1 + exp(logits)).
    return mean(logaddexp(0, logits) - y * logits)


def bce_logits_backward(logits, y):
    return (sigmoid(logits) - y) / y.shape[0]

The Linear.backward method returns dX, but it also stores dW and db. Those have different jobs. dX continues the chain rule. dW and db wait for the optimizer step. Returning all three as one anonymous tuple is possible, but named fields make a first implementation easier to inspect.

The code uses a sum for each parameter gradient because dout already contains the division by batch size from the mean loss. A different convention is valid—sum loss and omit that division—but the forward loss and its first backward signal must agree. Do not average twice by accident.

A Complete Forward and Backward Trace

Start with two examples and deliberately symmetric parameters:

X  = [[1, 0],
      [0, 1]]
y  = [[1],
      [0]]

W1 = [[ 1, -1],
      [-1,  1]]       b1 = [[0, 0]]
W2 = [[ 1],
      [-1]]           b2 = [[0]]

The forward pass is a ledger, not merely a prediction:

Z1     = X @ W1 + b1 = [[ 1, -1], [-1,  1]]
A1     = ReLU(Z1)    = [[ 1,  0], [ 0,  1]]
logits = A1 @ W2+b2  = [[ 1], [-1]]
probs  = sigmoid(logits) ≈ [[0.7311], [0.2689]]
loss   ≈ 0.3133

The first example has a positive logit and the second a negative one, so the predictions already match their labels. The loss is still not zero because a probability of about 0.73 is not total confidence. This is an important correction: correct class decisions and low loss are related but not identical claims.

Now start backward at the loss. With two batch rows:

dlogits = (probs - y) / 2
        ≈ [[-0.1345], [ 0.1345]]

Pass that value through Linear 2 before changing W2:

dW2 = A1.T @ dlogits ≈ [[-0.1345], [ 0.1345]]
db2 = sum_rows(dlogits) = [[0]]
dA1 = dlogits @ W2.T  ≈ [[-0.1345,  0.1345],
                          [ 0.1345, -0.1345]]

ReLU now uses the cached Z1, not A1 by habit. Its mask is positive only on the diagonal in this example:

ReLU mask = [[1, 0], [0, 1]]
dZ1      ≈ [[-0.1345,  0],
             [ 0,     -0.1345]]

Finally, Linear 1 turns that signal into its parameter gradients:

dW1 = X.T @ dZ1 ≈ [[-0.1345,  0],
                     [ 0,     -0.1345]]
db1 = sum_rows(dZ1) ≈ [[-0.1345, -0.1345]]

Every zero now has an explanation. ReLU blocked the off-diagonal paths, and each input row selects one feature. If a manual implementation instead produces dense dW1 values here, the trace gives a concrete place to investigate: the activation mask, transpose, or input cache.

So far, the mechanism has three phases with different responsibilities:

forward:  create values and caches
backward: create gradients while all forward parameters remain unchanged
update:   change parameters from the stored gradients

Update Only After the Reverse Walk

With eta = 0.5, gradient descent changes both matrices and the first bias:

W2_new ≈ [[ 1.0672], [-1.0672]]
b2_new = [[0]]
W1_new ≈ [[ 1.0672, -1], [-1, 1.0672]]
b1_new ≈ [[0.0672, 0.0672]]

Running a fresh forward pass with those values gives logits of about [[1.2106], [-1.2106]] and a lower loss of about 0.2605 on this toy batch. That lower value is a local observation, not a guarantee that this learning rate will improve every batch or generalize beyond these two points.

Why insist on updating last? dA1 was calculated with the old W2. If code updates W2 before calculating dA1, the network no longer differentiates the same forward pass that produced the loss. The error can be small enough to look plausible, especially with a small learning rate, which makes it more dangerous than a crash.

A clear update loop keeps mutation in one place:

for layer in [linear1, linear2]:
    layer.W -= eta * layer.dW
    layer.b -= eta * layer.db

This is a teaching implementation. A production framework also manages parameter registration, devices, precision, and optimizer state. Those conveniences do not change the local ownership rule shown here.

The Trade-off: Visibility Versus Repeated Work

Writing these layers by hand improves visibility. A failed check can be narrowed to one cache, one reduction, one matrix product, or one update boundary. That is valuable when learning the mechanism or when testing a genuinely custom operation.

The trade-off is that the same visibility creates maintenance work. Every new operation needs its own correct backward rule, cache policy, shape tests, and handling for edge cases. A sigmoid can overflow if written carelessly; a ReLU has a boundary at zero; larger models need more saved activations and therefore more memory. A hand-built loop is not automatically more trustworthy merely because it is explicit.

Automatic differentiation is usually the practical preference once the local behavior is understood and the operation is supported by a well-tested framework. It removes much of the repeated derivative bookkeeping, but it does not remove the need to define the right loss, preserve the intended shapes, or understand which tensors are part of the graph. The manual trace earns its place because it gives the learner a reference model for those framework decisions.

Use the smallest implementation that exposes the question you need to answer. For a new layer, start with a tiny deterministic batch and a few parameters. For a normal model, let the framework calculate standard gradients and inspect only the signals relevant to the failure. The boundary appears when hand-written code grows faster than the hypothesis it is testing: at that point, a gradient check and a framework comparison give more evidence for less risk.

Where Manual Code Usually Breaks

Cache mismatch. ReLU's derivative needs the forward branch information. Caching Z1 makes the condition visible. Some activations can derive their derivative from output instead; the layer's contract should state which value it needs rather than relying on a naming convention.

Batch convention mismatch. A mean loss and an unscaled output gradient make gradients B times too large. A sum loss and an already divided gradient make them too small. Print the batch size and state exactly where the reduction occurs.

Silent shape compatibility. A transpose may produce a legal multiplication while swapping feature and output axes. Compare every gradient shape with the parameter it updates: dW1 must match W1, and db1 must match b1.

Early mutation. Updating a later layer before computing dA1 makes earlier gradients depend on new weights. Keep all updates after all backward calls.

A zero ReLU path. For X = [[1, 1]] in this example, Z1 = [[0, 0]], so the ReLU sends zero gradient to Linear 1 under the common zero-at-the-boundary convention. That is a real boundary of this toy network, not proof that the code is wrong. The next lesson will test such claims with numerical gradient checking.

Checks Before You Trust the Loop

Check: Which layer owns the cached A1, and why?

Think first, then reveal.

Answer: Linear 2 owns it because its weight gradient is A1.T @ dlogits. ReLU owns Z1 for a different reason: it needs the preactivation to build its local mask.

Check: In the trace, why is db2 zero even though the loss is positive?

Answer: The two rows contribute equal and opposite values to the shared output bias: -0.1345 + 0.1345 = 0. This cancellation says nothing by itself about dW2, dW1, or whether a different batch would update the bias.

Check: A colleague divides dlogits by the batch size and also divides every dW and db by that size. What has happened?

Answer: For a mean loss, the gradients were averaged twice. The direction is usually unchanged, but their scale is wrong by a factor of B, so learning-rate behavior becomes misleading.

Trace It Yourself

Keep W1, b1, W2, and b2 from the worked trace, but use the single example X = [[1, 1]] with y = [[1]].

1. Compute Z1, A1, the logit, and dlogits.
2. Compute dA1 and apply the ReLU mask to obtain dZ1.
3. Decide which of dW1, db1, dW2, and db2 can be nonzero.
4. Name one test you would run before declaring the zero first-layer gradient a bug.

Model answer: Z1 = [[0, 0]], A1 = [[0, 0]], and the logit is 0, so dlogits = [[-0.5]]. dA1 = [[-0.5, 0.5]], but the ReLU mask is [[0, 0]], giving dZ1 = [[0, 0]]. Therefore dW1 and db1 are zero for this row; dW2 is also zero because A1 is zero; db2 is -0.5 and can update. Inspect the cached Z1 and the layer's zero-derivative convention first. Then use the finite-difference check in the next lesson for a parameter away from the ReLU kink; a numerical check exactly at a nondifferentiable boundary is not a clean oracle.

Resources

• [DOCUMENTATION] NumPy: matmul — Focus: matrix-product shape rules behind every @ in the trace.
• [TUTORIAL] CS231n: Backpropagation, Intuitions — Focus: local gradient flow and staged computation.
• [DOCUMENTATION] PyTorch: Autograd mechanics — Focus: saved tensors and backward functions that a framework manages for you.
• [BOOK] Deep Learning, Chapter 6 — Focus: feedforward networks and their training gradients.

Key Takeaways

• A manual training step is reliable only when forward caches, backward gradients, and parameter updates have distinct owners.
• Parameter gradients are accumulated across batch rows, while dX or dA continues the chain rule to earlier layers.
• All backward calculations must use the parameters from the forward pass; mutate weights only after the reverse walk finishes.
• A small trace with shapes and known zeros turns a vague training failure into a local, testable hypothesis.