optimization

Second derivatives buy something gradients cannot: a step shaped by curvature, immune to conditioning, converging quadratically. This lesson builds Newton's method, then layers it on a log barrier to get interior point methods, the machinery that made large constrained problems solvable with a certificate rather than a hope.

Gradient descent is three lines of code and a hundred years of theory. This lesson derives why a safe step size is one over the smoothness constant, why the condition number governs everything, and why acceleration reaching order one over k squared is provably the best any first-order method can do.

Convexity is what separates optimization problems you can solve with a guarantee from ones you can only hope about. This lesson defines convex sets and functions, proves why every local minimum is global, and gives you the operations that let you recognise convexity without touching a Hessian.

One derivative becomes three objects once a function has many inputs and many outputs. This lesson builds the gradient, the Jacobian and the Hessian, shows what each one actually tells you, and explains why curvature decides how many steps an optimiser needs and why nobody ever writes the Hessian down.

A benchmark is an experiment, and most are badly designed enough to produce confident wrong answers. This lesson covers what a measurement must control to mean anything, the ways microbenchmarks lie including code the compiler deletes, how to catch regressions in continuous integration despite noisy machines, and how to recognise the point where optimising stops paying.

Slow software is slow for a short list of reasons, and each one has a signature you can recognise before you find the code. This lesson covers the six recurring bottleneck classes, waiting on I/O, chatty queries, allocation pressure, lock contention, memory access patterns and serialisation, with the symptom that identifies each and the fix that actually works.

Most optimisation effort is spent on code that was never the problem, and the reason is that intuition about where time goes is reliably wrong. This lesson covers why guessing fails, the arithmetic that caps what any optimisation can buy, the difference between latency and throughput, and how to set a target that tells you when to stop.

Round-to-nearest works on small models and falls apart on large ones, because beyond a certain scale transformers grow systematic activation outliers in a few fixed channels, a hundred times larger than everything else. This lesson shows arithmetically why one outlier destroys a tensor, then works through the three families of fix: decompose, smooth, and rotate.

Decoding is memory-bandwidth-bound, so fewer bits per weight means more tokens per second, not fewer. This lesson builds the mechanism from the arithmetic up: the affine mapping, a worked example done by hand, why group size costs fractional bits, the difference between W4A16 and W8A8, and which formats the hardware actually accelerates.

What differentiable rendering is actually used for, how it relates to NeRF and Gaussian splatting, and the four failure modes that make inverse rendering harder than the optimisation loop suggests.

Differentiating a renderer is easy until geometry moves. Why silhouettes break naive automatic differentiation, and the three families of solutions: edge sampling, reparameterization, and warped-area methods.

Where this machinery earns its place: low-rank matrix completion, synchronization, and PCA. Plus the two ideas that explain why non-convex manifold problems are often solved to global optimality anyway.

How to move along a curved space without solving differential equations: retractions as cheap approximations to geodesics, vector transport, and the Riemannian versions of gradient descent, conjugate gradients, and trust regions.

Building the machinery: the linear space of allowed directions at a point, the inner product that gives it geometry, and why the Riemannian gradient is the ambient gradient projected rather than a new derivative.

Rotations, subspaces, low-rank matrices and covariances are not vectors in flat space. Why treating those constraints as penalties or projections wastes structure, and what it means to say the search space is a manifold.

Why local training on non-identical data pulls clients apart, how averaging their updates produces a model that suits nobody, and the control-variate fix that corrects the drift.

The federated setting: why hospitals, phones, and banks cannot pool their data, what changes when the training loop crosses a network, and the FedAvg algorithm that made the idea practical.

Taking the framework beyond solvable toy models: sharp transitions in real learning, why the loss landscape of a neural network is not the fractured one theory warns about, and what the physics lens genuinely explains.

Some functions cost hours or dollars to evaluate once, give no gradient, and must be optimized in as few tries as possible. This lesson sets up that problem, shows why grid and random search waste the budget, and introduces the surrogate-model loop that Bayesian optimization is built on.

How draft-then-verify decoding gets multiple tokens per forward pass of a large model, why rejection sampling makes it provably lossless, and where the draft comes from (small models, Medusa heads, self-speculation, EAGLE trees).

Go from a raw neural network to one that actually generalizes. Covers loss functions (MSE, cross-entropy), gradient descent variants (SGD, momentum, Adam), learning-rate effects, overfitting vs underfitting, and the regularization toolkit (L2/dropout/early stopping/batch norm).

The typed AST is high-level — too high for a CPU. Learn why compilers lower to an intermediate representation first, what SSA form buys you, how classic optimizations (constant folding, dead-code elimination, CSE) transform IR, and how instruction selection and register allocation finally produce machine code.

Every constrained optimization problem has a twin. Learn how to build the Lagrangian, derive the dual problem, and use weak duality, strong duality, and the KKT conditions to certify optima — with worked examples from linear programming and SVMs.

A working CUDA kernel is the start, not the finish. How to measure occupancy, spot uncoalesced loads and warp divergence, and read the three numbers in Nsight Compute that actually matter.

Why naive matmul on a GPU is bandwidth-starved, and how tiling with __shared__ memory reduces global memory traffic by a factor of the tile size. The classic optimisation, with the kernel that demonstrates it.