The Chain Rule, and Why Depth Is Hard

Strip the vocabulary away and a deep network is one function applied after another:

$f = f_L \circ f_{L-1} \circ \cdots \circ f_2 \circ f_1$

Each $f_i$ is a layer: an affine map, an activation, an attention block, a normalisation. Nothing in that list is difficult to differentiate on its own. The difficulty is entirely in the composition, because you never get to differentiate a layer in isolation. You need the derivative of the whole stack with respect to a weight buried near the bottom.

Key idea: The chain rule is the only tool that converts local knowledge (each layer knows its own derivative) into global knowledge (how the final loss responds to any parameter anywhere). Backpropagation is not a separate algorithm. It is the chain rule with a good evaluation order.

For $y = f(g(x))$ the rule is a product of two local slopes:

$\frac{dy}{dx} = f'(g(x)) \cdot g'(x)$

The reason it multiplies rather than adds is worth holding onto. A nudge of size $h$ in $x$ produces a nudge of about $g'(x)h$ in the intermediate value, and that nudge is then scaled again by $f'$ at the point it arrives. Sensitivities compose by multiplication because each stage amplifies whatever reaches it.

Concretely, with $f(u) = u^3$ and $g(x) = \sin x$:

$\frac{d}{dx}\sin^3 x = 3\sin^2 x \cdot \cos x$

That multiplicative structure is the whole story of this lesson. It is what makes backpropagation efficient, and also what makes deep networks numerically fragile.

In a real graph an input rarely reaches the output by one route. A residual block sends the same tensor down two paths; a shared embedding matrix is read at every position in the sequence.

When $y$ depends on $x$ through several intermediates $u_1, \dots, u_k$, the contributions add:

$\frac{dy}{dx} = \sum_{j=1}^{k} \frac{\partial y}{\partial u_j} \frac{\partial u_j}{\partial x}$

Multiply along a path, add across paths. That single sentence is the complete multivariable chain rule, and it is exactly what a framework implements: gradients arriving at a node from different consumers are accumulated with +=, never overwritten.

Gotcha: A hand-written backward pass that assigns instead of accumulating will be silently correct on a plain feed-forward stack and silently wrong the moment anything is reused. The bug looks like a model that trains, just worse.

Here $x$ reaches $y$ through both $u$ and $v$. The derivative $dy/dx$ is the product along the top path plus the product along the bottom one. Every branch in a computation graph is a term in that sum, and every merge is where the terms get added back together.

Once layers map vectors to vectors, each local derivative is a Jacobian and the chain rule becomes matrix multiplication. For $y = f_3(f_2(f_1(x)))$:

$J_f(x) = J_{f_3} \cdot J_{f_2} \cdot J_{f_1}$

The shapes have to interlock, and checking them catches most errors before you run anything. Take a stack that goes from 1000 inputs to 100, then to 10, then to a single scalar loss:

• $J_{f_1}$ is $100 \times 1000$
• $J_{f_2}$ is $10 \times 100$
• $J_{f_3}$ is $1 \times 10$

The product $(1 \times 10)(10 \times 100)(100 \times 1000)$ is $1 \times 1000$: one row, one entry per input parameter. That row is the gradient. The whole of backpropagation is evaluating this product without ever building the matrices in it.

Written out for a small network, the chain rule is unremarkable code. Each line is one Jacobian applied to whatever arrived from above.

import numpy as np

# forward: h = tanh(W1 x),  y = W2 h,  L = 0.5 * ||y - t||^2
z = W1 @ x
h = np.tanh(z)
y = W2 @ h
L = 0.5 * np.sum((y - t) ** 2)

# backward: start at the loss and walk back
dL_dy = y - t                      # dL/dy
dL_dW2 = np.outer(dL_dy, h)        # dL/dy times dy/dW2
dL_dh = W2.T @ dL_dy               # dy/dh is W2, so transpose it
dL_dz = dL_dh * (1 - h ** 2)       # tanh' = 1 - tanh^2, a diagonal Jacobian
dL_dW1 = np.outer(dL_dz, x)

Notice dL_dz: the Jacobian of an elementwise activation is diagonal, so applying it is a vector multiply rather than a matrix one. That single optimisation is why activations are nearly free in the backward pass.

Matrix multiplication is associative, so the value of $J_3 J_2 J_1$ does not depend on where you put the brackets. The cost does, and dramatically.

This is not a micro-optimisation. Working right to left is forward-mode differentiation; working left to right, from the loss backwards, is reverse mode, which is to say backpropagation. They compute the identical number.

The gap widens as the network gets wider and deeper, because the forward order pays for intermediates whose height is the input dimension while the reverse order never exceeds the output dimension.

Machine learning sits at the extreme end of that trade: millions of inputs, one scalar output. That asymmetry, and nothing else, is why every training framework differentiates backwards.

A product of $L$ Jacobians behaves like a number raised to the power $L$. If each factor typically shrinks the signal, the product collapses; if each amplifies, it explodes. Neither needs anything exotic to happen.

The logistic sigmoid makes the point exactly. Its derivative is $\sigma(x)(1 - \sigma(x))$, and here is that curve:

The peak is 0.25, at the single point $x = 0$. Even in the best case every sigmoid layer multiplies the gradient by at most a quarter, so ten stacked layers shrink it by at most $0.25^{10}$, roughly $10^{-6}$. Away from zero it is far worse: at $x = 6$ the factor is 0.0025.

The standard remedies all read as edits to the Jacobian product rather than to the optimiser.

• ReLU has derivative exactly 1 wherever it is active, so it contributes a factor of 1 instead of at most 0.25.
• A residual connection computes $y = x + F(x)$, whose Jacobian is $I + J_F$. The identity term leaves a route with a factor of 1 through the entire depth, so the product cannot vanish just because it is long.
• Gradient clipping rescales the product after the fact when it explodes, which treats the symptom and is honest about it.
• Normalisation layers hold activations in the range where derivatives are largest, keeping factors near 1 rather than out on the flat tails.

Key idea: "Vanishing gradient" is not a pathology of deep learning. It is what happens when you multiply many numbers smaller than one together, and the architectural answer is to keep the factors near one.