Gradient Checking & Debugging

By the end of this lesson, you will be able to...

• compare one analytical gradient with a centered finite-difference estimate;

• read a gradient-check report as evidence about a local implementation;

• choose a controlled debugging step when the two gradients disagree.

Idea in one sentence: Gradient checking perturbs one parameter while holding the rest fixed, then asks whether the observed loss change agrees with the gradient that backpropagation claims for that same point.

Core Insight

A learner has just written the tiny two-layer classifier from the previous lesson. The script runs, its loss falls from 0.31 to 0.30 after one update, and a review of the code finds no exception. Yet a missing batch division can make every update too large, and a wrong ReLU cache can block the wrong path. Both mistakes can produce numbers that look reasonable.

A tempting response is to train longer and tune the learning rate. That only mixes a question about derivatives with questions about optimization, initialization, and data. Before changing those knobs, make one parameter testable.

Gradient checking supplies an independent comparison. Backpropagation uses the chain rule. A finite difference reruns the forward loss after a tiny positive and negative change to one parameter. Agreement is evidence that the local forward definition and its local backward rule match. The values in this lesson are a deterministic teaching trace, not a claim about a production model.

The Controlled Situation

Reuse the two-example classifier from lesson 007. Its hidden activation was:

A1 = [[1, 0],
      [0, 1]]

W2 = [[ 1],
      [-1]]

The logits are [[1], [-1]], the binary labels are [[1], [0]], and the mean binary cross-entropy loss is about 0.3132617. The backward pass reported:

dW2[0, 0] = -0.1344707

We will check exactly that coordinate. The other parameters, the batch, the model mode, and the loss reduction must stay fixed. If a check changes two things at once, disagreement becomes hard to interpret.

The initial model says: “If the code runs and the loss falls after an update, the gradient must be fine.” That works sometimes, but it misses silent errors. The stronger model is narrower and more useful: compare two calculations of the sensitivity of this one loss to this one scalar parameter.

What the Centered Difference Measures

Let w mean W2[0, 0] and let h be a small perturbation. The centered estimate is:

numerical_grad = (L(w + h) - L(w - h)) / (2h)

For h = 0.0001, changing w changes only the first example's logit because its activation row is [1, 0]. Recompute the complete mean loss each time:

The positive perturbation lowers the loss, so the derivative should be negative. Substitution gives:

(0.3132482 - 0.3132751) / 0.0002 ≈ -0.13447

That agrees with dW2[0, 0] from backpropagation to the displayed precision. The centered formula matters. A one-sided difference asks only about L(w+h) and has a larger truncation error; evaluating both sides cancels the first-order curvature term. It costs two forward evaluations for each coordinate checked, which is why gradient checking is a diagnostic rather than a training algorithm.

The comparison is about a derivative at the current parameter values. It does not say the model will generalize, that the loss is appropriate, or that the learning rate is useful. It says something smaller: this coordinate's implementation agrees with the local change observed through the forward loss.

Build a Check That Restores Its State

The forward-only function must use the same loss definition as training and must not update parameters. A small coordinate checker can make that contract explicit:

def numerical_coordinate(loss_only, parameter, index, h=1e-4):
    original = parameter[index]

    parameter[index] = original + h
    loss_plus = loss_only()

    parameter[index] = original - h
    loss_minus = loss_only()

    parameter[index] = original      # restore, even when the check succeeds
    return (loss_plus - loss_minus) / (2 * h)


def relative_error(analytic, numerical):
    return abs(analytic - numerical) / max(
        1e-12, abs(analytic) + abs(numerical)
    )

For the coordinate above, a report might read:

parameter: W2[0, 0]
analytic:  -0.1344707
numerical: -0.1344707
relative error: very small

Restoring the original value is not a cosmetic detail. If w+h leaks into the next check, later losses are no longer evaluated at the intended point. In real code, use a try/finally block or a copy so an exception also restores state.

Check a few coordinates from every parameter tensor, not only a random slice of one large flattened vector. A sample that accidentally contains only weights can leave bias gradients untested. For a small educational network, checking every coordinate is affordable and easier to reason about.

A Deliberately Broken Backward Rule

Now introduce one common bug. The loss is a mean over two examples, but the output derivative is implemented without dividing by batch size:

wrong dlogits = sigmoid(logits) - y
correct dlogits = (sigmoid(logits) - y) / B

For the chosen coordinate, the broken backward pass reports -0.2689414, while the numerical estimate remains -0.1344707. The mismatch is not a rounding quirk. With the relative-error formula above:

abs(-0.2689414 - (-0.1344707)) /
(abs(-0.2689414) + abs(-0.1344707))
= 0.3333

That pattern is useful evidence. The analytical value is exactly twice the numerical one, which points toward a batch reduction mismatch. It does not prove that this is the only bug, but it narrows the next inspection to the loss reduction and the initial gradient rather than to every layer.

This is the debugging loop:

choose a deterministic tiny case
-> compare one analytical and one numerical coordinate
-> classify the mismatch pattern
-> inspect the smallest responsible operation
-> rerun the same check after the repair

Do not “fix” a mismatch by changing the learning rate. Learning rate changes the update after a gradient is computed; it cannot make a wrong analytical derivative agree with the forward loss.

Choose h and the Test Conditions Deliberately

The centered estimate is an approximation. A large h samples too much curvature, so it no longer describes the local slope. An extremely small h subtracts nearly equal floating-point loss values, and rounding can dominate the result. There is no universal best number.

For this kind of small double-precision teaching example, 1e-4 or 1e-5 is a sensible starting point, not a law. If the check is suspicious, try nearby values such as 1e-3, 1e-4, and 1e-5 and inspect the raw losses as well as the reported error. A stable region of agreement is better evidence than one magically chosen epsilon.

The check also assumes the forward computation is repeatable. Freeze random seeds; disable dropout and random augmentation; use the same batch in L(w+h) and L(w-h); and make evaluation behavior explicit for layers with training-time state. If a different random mask appears in each loss evaluation, the difference estimates randomness rather than a derivative.

ReLU adds another boundary. Its derivative changes at zero. If a perturbation moves a preactivation from negative to positive, the two forward evaluations cross a kink and the centered finite difference is not a clean comparison with one local derivative convention. Inspect preactivations, choose a coordinate away from the boundary, or use a smooth activation temporarily to isolate the rest of the network.

Read the Failure Signal Before Changing Code

Gradient checking improves isolation, not certainty about the whole learning system. It costs repeated full forward evaluations and can become expensive quickly. It is a good fit for a new custom layer, a custom loss, or a tiny implementation. It is a poor fit for every parameter on every normal training step.

The trade-off is explicit: finite differences give an independent reference for a local derivative, but their repeated forward passes are slow and sensitive to numerical and state-control choices. Use that cost to buy evidence during debugging, not to replace backpropagation during learning.

Cluster Check

Check: A two-example mean loss has an analytical bias gradient of 0.08. Your centered finite difference gives 0.04, and several other parameters are also exactly twice the numerical estimate. What should you inspect first?

Think first, then reveal.

Answer: Inspect where the mean reduction is applied. The pattern suggests the backward pass omitted division by batch size or applied a sum convention while the forward loss reports a mean. Do not start by changing the optimizer.

Check: A gradient check passes for W2[0, 0]. What can you responsibly conclude?

Answer: At the tested parameter point and deterministic setup, that coordinate's analytical gradient agrees with the implemented forward loss. You cannot conclude that every parameter, every branch, generalization, or training configuration is correct.

Check: Why can a check near a ReLU preactivation of zero be unstable even if the code is correct?

Answer: The plus and minus perturbations can take different branches of a nondifferentiable function. They are then not estimating one smooth local slope.

Debug It Yourself

Use the network from lesson 007 and deliberately replace its correct output gradient with sigmoid(logits) - y while keeping the mean loss unchanged.

1. Run analytical backpropagation and record one W2 coordinate.
2. Restore the correct forward loss and compute the centered numerical value for the same coordinate.
3. Calculate relative error and identify the error pattern.
4. Repair the batch scaling, rerun the check, then check one coordinate from W1 and one from b2.

Model answer: For W2[0, 0], the faulty analytical gradient is twice the numerical estimate, producing a relative error near 0.3333 with the symmetric denominator used here. Divide dlogits by B once for a mean loss, then the two values should agree within a tolerance appropriate to the data type and operation. Checking W1 and b2 after the repair tests that the correction did not only make one output-layer coordinate look good.

Resources

• [TUTORIAL] CS231n: Gradient checks — Focus: centered differences, relative error, kinks, precision, and deterministic checks.
• [DOCUMENTATION] PyTorch: torch.autograd.gradcheck — Focus: what an automated finite-difference checker requires and its limits.
• [DOCUMENTATION] NumPy: floating-point machine limits — Focus: why precision constrains an extremely small epsilon.
• [BOOK] Deep Learning, Chapter 4 — Focus: numerical computation and stability around learning systems.

Key Takeaways

• A centered finite difference checks whether an analytical gradient matches loss changes at one fixed parameter point.
• Freeze randomness, loss definition, batch, mode, and every parameter except the coordinate being perturbed.
• Mismatch patterns are diagnostic evidence: a shared scale error points to reduction; a local failure points to a local rule or cache.
• Gradient checking verifies derivatives, not optimizer choice, model quality, or generalization.