The Derivative Is a Local Linear Model

A model with a billion parameters is, mathematically, a function from those parameters to a single number: the loss. Training asks the same question at every step, about every parameter: if I nudge this one, does the loss go up or down, and by how much?

That is a derivative. Not a symbolic exercise on paper, but a number you need in order to decide where to step next. Everything in this course exists to answer that question cheaply enough that you can ask it a billion times per second.

Key idea: Calculus enters machine learning as a search strategy. You cannot see the shape of the loss surface, so you feel it locally: measure the slope where you are standing, step downhill, repeat.

Start from the statement that does the most work later:

$f(x + h) = f(x) + f'(x)\,h + O(h^2)$

Read it as a prediction. Near $x$, the function behaves like a straight line, $f'(x)$ is that line's slope, and the error you make by trusting the line shrinks like $h^2$. Halve the step and the error drops by a factor of four.

The familiar limit definition says the same thing, rearranged:

$f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$

Definition: The derivative is the coefficient of the best linear approximation to $f$ near a point. Gradients, Jacobians and backpropagation are all consequences of that one sentence carried into more dimensions.

Trust the linear model for one small step and gradient descent falls out of it:

$x_{k+1} = x_k - \eta\, f'(x_k)$

Substitute that step back into the approximation and you can see why it works:

$f(x_{k+1}) \approx f(x_k) - \eta\, f'(x_k)^2$

The correction term contains a square, so it is never positive. Any sufficiently small step in the direction opposite the derivative reduces $f$. That is the entire guarantee, and it is a local one.

"Sufficiently small" is carrying real weight. The learning rate $\eta$ decides how far out you are willing to trust a model that was only ever accurate nearby. Push it too far and you leave the region where the line described the function at all, which is what a loss curve diverging to infinity looks like from the inside.

You can get a derivative without ever differentiating anything, by treating $f$ as a black box and measuring it twice.

def forward_diff(f, x, h=1e-8):
    return (f(x + h) - f(x)) / h

def central_diff(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)

The forward difference is the limit definition with the limit left unfinished, so its error is the $O(h)$ term. The central difference is better: the two $h^2$ terms cancel by symmetry, leaving $O(h^2)$ error, at the price of a second evaluation.

This needs nothing from you. No formula for $f$, no source code, no differentiability proof. That generality is exactly why it is still used to check other methods, and why it is useless for training anything large.

The error term says $O(h)$. Shrink $h$, shrink the error. So take it all the way down.

Two error sources are pulling in opposite directions. Truncation error, the $O(h)$ term you are trying to kill, falls as $h$ falls. Rounding error rises: $f(x+h)$ and $f(x)$ agree in more and more leading digits, so their difference retains fewer and fewer significant ones, and you then amplify that damaged quantity by dividing by something tiny.

Truncation grows like $h$; rounding grows like $\varepsilon/h$, where $\varepsilon$ is machine epsilon, about $2.2 \times 10^{-16}$ in float64. Their sum is smallest near $h = \sqrt{\varepsilon}$, which is about $1.5 \times 10^{-8}$. For the central difference the truncation term is $h^2$ instead, and the optimum moves out to $\varepsilon^{1/3}$, about $6 \times 10^{-6}$.

Measure it and the theory holds exactly:

The peak is around 8 digits out of the 16 that float64 carries. Differencing costs you half your precision, and no cleverness in the formula recovers it.

A gradient needs one derivative per parameter. With finite differences, each of those needs its own perturbed evaluation of $f$.

For a model with $10^9$ parameters, the numerical row means a billion and one forward passes to take one training step. At a millisecond per pass that is eleven days for a single update. The third row is what makes deep learning arithmetically possible at all, and the last lesson of this course is about how it works.

The rectified linear unit, $\mathrm{ReLU}(x) = \max(0, x)$, has no derivative at zero: the slope approaching from the left is 0, from the right it is 1, and there is no single line that fits both. Convex analysis handles this with the subdifferential, a set rather than a number:

$\partial\,\mathrm{ReLU}(0) = [0, 1]$

Any value in that interval is a legitimate stand-in. Frameworks simply pick one; PyTorch and TensorFlow both use 0.

Gotcha: In exact arithmetic the choice is irrelevant, because a single point has measure zero and you will essentially never land on it. In floating point you land on it constantly. Bertoin, Bolte, Gerchinovitz and Pauwels (NeurIPS 2021, "Numerical influence of ReLU'(0) on backpropagation") found that changing this one value alters backpropagation outputs around half the time at 32-bit precision, and systematically at 16-bit, while the effect disappears in double precision.

Gradient descent stops where $f'(x) = 0$. That tells you the linear model is flat, and nothing more. Extend the approximation one term further to see what is actually there:

$f(x + h) = f(x) + f'(x)h + \tfrac{1}{2}f''(x)h^2 + O(h^3)$

At a stationary point the linear term vanishes and the quadratic one takes over. The sign of $f''(x)$ decides the character of the point:

• $f''(x) > 0$: the curve bends upward, a local minimum
• $f''(x) < 0$: it bends downward, a local maximum
• $f''(x) = 0$: undecided, and you need a further term

That second derivative is also the thing that says how fast the first one goes stale, which is the real limit on how big a step you can take. In many variables it becomes a matrix, and the next lesson is about what that matrix tells you.

Nobody trains with numerical derivatives, but almost everybody debugs with them. If you hand-write a backward pass, differencing gives you an independent second opinion that shares none of its bugs.

def grad_check(f, x, analytic, h=1e-5):
    numeric = (f(x + h) - f(x - h)) / (2 * h)
    denom = max(abs(numeric), abs(analytic), 1e-12)
    return abs(numeric - analytic) / denom

Compare relative error, never absolute: a discrepancy of 0.01 is nothing beside gradients of size 1000 and fatal beside gradients of size 0.001. Stanford's CS231n notes give the working thresholds still used today: below $10^{-7}$ the analytic gradient is almost certainly right, and above $10^{-2}$ it is almost certainly wrong.

In practice: Check in float64 and away from kinks. A ReLU network checked at an input that lands near zero will fail for a reason that has nothing to do with your code.