Chain Rule & Computational Graphs

By the end of this lesson, you will be able to...

• draw the dependency path from a parameter to a loss;

• calculate a parameter gradient by multiplying local derivatives along that path;

• explain why reverse-mode automatic differentiation reuses downstream information.

Idea in one sentence: The chain rule lets a loss tell an earlier weight how it mattered by multiplying the small changes at every operation connecting them.

Core Insight

The inspection model has made a score and the loss says that score is too low for a defective part. A reasonable question follows: which earlier weight should change? The tempting answer is “the loss knows the answer somehow.” That is not enough to debug or trust training.

An early weight does not touch the loss directly. It changes an intermediate score; that score changes a prediction; the prediction changes the loss. A computational graph records those dependencies. The chain rule turns them into a calculation: measure the local change at each edge, then combine the changes on the path.

This lesson uses one deliberately tiny scalar graph. Its numbers are a teaching model, not a trained inspection system. The next lesson turns the same idea into the reusable backpropagation procedure for many parameters and layers.

A Small Graph With One Weight

For one feature value x, one weight w, one bias b, and target t, define:

z = w*x + b
L = (z - t)^2

z is the model score and L is a squared-error loss. The graph is:

w -----\
        (*) ---> m ---\
x -----/               (+) ---> z ---> (- t) ---> e ---> square ---> L
b --------------------/

The extra node m = w*x makes the multiplication visible. During the forward pass, values move left to right. During differentiation, we ask how the final L changes if one earlier value changes. A derivative written dL/dw means exactly that local sensitivity of the final loss to the weight.

The initial model “differentiate the whole formula in one leap” works for this two-line expression. It becomes fragile once a network contains matrix products, activations, branches, and many reused values. The graph gives each operation one small derivative rule instead of requiring a new global formula for every architecture.

The Chain Rule Names the Missing Link

Suppose L depends on z, and z depends on w. The chain rule says:

dL/dw = dL/dz * dz/dw

Read it from right to left if that is easier: change w a little; that changes z by dz/dw; the changed z changes L by dL/dz. Multiply the two rates to get the total effect on L.

For our graph, the loss also has an intermediate error value:

e = z - t
L = e^2

So the full path is:

dL/dw = dL/de * de/dz * dz/dm * dm/dw

The addition step has dz/dm = 1, so it does not change the numeric value. It still belongs in the graph: an addition can have several inputs, and each input receives its own local derivative. Keeping the node visible prevents the common mistake of treating an expression as a mysterious black box.

Worked Calculation: Send One Error Backward

Use these illustrative values:

x = 2
w = 0.5
b = -0.5
t = 1

First run the graph forward:

The loss is positive because the score is below the target. Now move backward from L. The local derivatives are:

dL/de = 2*e = -1
de/dz = 1
dz/dm = 1
dm/dw = x = 2

Multiply them in path order:

dL/dw = (-1) * 1 * 1 * 2
      = -2

The negative sign is useful information. Increasing w slightly will increase z because x is positive; increasing z moves it toward the target and reduces this loss. With a teaching learning rate eta = 0.1, gradient descent makes the update:

w_new = w - eta*(dL/dw)
      = 0.5 - 0.1*(-2)
      = 0.7

Holding b fixed just for this small check, the new score is 0.7*2 - 0.5 = 0.9, and the new loss is (0.9 - 1)^2 = 0.01. This one update improves this one example. It does not prove that the model fits other parts, that squared error is the right production objective, or that the learning rate is safe for a whole dataset.

The same backward trace gives the other local sensitivities:

dL/db = dL/dz * dz/db = (-1)*1 = -1
dL/dx = dL/dz * dz/dx = (-1)*w = -0.5

The first tells training how this bias affects the loss. The second is not a parameter update here; it is the signal the preceding operation would need. This distinction—parameter gradient versus upstream gradient—is the bridge to backpropagation.

Why a Graph Is More Than a Diagram

The graph is an execution record. Its forward values tell each local derivative what number to use: dL/de needed the stored value e = -0.5; dm/dw needed x = 2. Automatic-differentiation systems record or reconstruct this information, then apply each operation's local backward rule.

Reverse mode is efficient when one scalar loss depends on many parameters. It begins with dL/dL = 1 and sends the already-computed downstream sensitivity backward. A node does not solve the whole model. It receives a derivative with respect to its output, multiplies by its local derivatives, and passes contributions to its inputs.

When a value feeds two later operations, it receives contributions from both paths. They are added. That accumulation is essential in real graphs: a parameter can influence a loss through more than one route. The scalar path above has one route, so it shows multiplication clearly before introducing that extra bookkeeping.

What the Chain Rule Does Not Promise

The chain rule computes a gradient of the loss you supplied at the current parameter values. It does not say the gradient is large enough to learn quickly, that the loss matches the real decision, or that a single gradient step improves every example.

This buys us a systematic method for credit assignment, but it costs memory or recomputation: backward rules often need forward values. The trade-off is concrete: save more activations and use more memory, or recompute some of them and use more time. In deep networks, products of many local derivatives can become very small or very large. The signal to inspect is the gradient magnitude by layer, not the assumption that a mathematically correct chain rule guarantees easy optimization.

It also has a boundary at nondifferentiable or numerically unstable operations. Libraries define practical derivative behavior for common operations, but a custom operation still needs a valid local rule. A graph cannot recover information an operation discarded without recording enough state.

Common Confusions

Confusion: The gradient says how important a parameter is in general.

Why it is tempting: it is called a sensitivity.

Better model: it is a local sensitivity of the current loss at the current values and batch. Its size can change after one update or on another example.

Confusion: Backpropagation is a separate alternative to the chain rule.

Why it is tempting: both names appear in different lessons.

Better model: backpropagation is the efficient reverse traversal that applies the chain rule repeatedly across a graph.

Confusion: A negative gradient means the parameter is bad.

Why it is tempting: negative sounds like failure.

Better model: the sign gives a local direction. With gradient descent, subtracting a negative gradient increases the parameter.

Check Your Understanding

Check: In the worked graph, if x were zero, what would dm/dw be and what would happen to dL/dw?

Think first, then reveal.

Answer: dm/dw = x = 0, so dL/dw would be zero along this path. Changing w does not change w*x when the current input is zero.

Check: Why do contributions add when one node feeds two downstream operations?

Think first, then reveal.

Answer: The final loss changes through each route that depends on the node. The total derivative includes the effect through every route, so reverse mode sums those contributions at the shared node.

Practice

Use the same graph z = w*x + b, L = (z-t)^2, with x = 3, w = 0.2, b = 0, and t = 1.

1. Calculate z, e, and L.
2. Calculate dL/dw by listing the local derivatives on the path.
3. With eta = 0.1, calculate one updated weight.
4. State one reason this calculation is not yet a full training result.

Model answer: z = 0.6, e = -0.4, and L = 0.16. The local derivatives are dL/de = -0.8, de/dz = 1, dz/dm = 1, and dm/dw = 3, so dL/dw = -2.4. The update gives w_new = 0.2 - 0.1*(-2.4) = 0.44. It only describes one example with a fixed bias and a chosen loss and learning rate; other examples can respond differently.

Resources

• [DOCUMENTATION] PyTorch: Autograd mechanics — Focus: relate saved forward values and backward functions to the graph trace.
• [DOCUMENTATION] JAX: Automatic differentiation — Focus: compare reverse-mode differentiation with the scalar path in this lesson.
• [TUTORIAL] CS231n: Backpropagation, Intuitions — Focus: extend local derivatives and gradient accumulation to larger graphs.
• [BOOK] Deep Learning, Chapter 6: Deep Feedforward Networks — Focus: connect computational graphs to neural-network training.

Key Takeaways

• A computational graph makes the intermediate dependencies between a parameter and a loss explicit.
• The chain rule multiplies local derivatives along a path; shared paths also require accumulated contributions.
• A gradient is a local training signal, not a promise of global improvement or generalization.