optimization

Go from a raw neural network to one that actually generalizes. Covers loss functions (MSE, cross-entropy), gradient descent variants (SGD, momentum, Adam), learning-rate effects, overfitting vs underfitting, and the regularization toolkit (L2/dropout/early stopping/batch norm).

The typed AST is high-level — too high for a CPU. Learn why compilers lower to an intermediate representation first, what SSA form buys you, how classic optimizations (constant folding, dead-code elimination, CSE) transform IR, and how instruction selection and register allocation finally produce machine code.

Every constrained optimization problem has a twin. Learn how to build the Lagrangian, derive the dual problem, and use weak duality, strong duality, and the KKT conditions to certify optima — with worked examples from linear programming and SVMs.

A working CUDA kernel is the start, not the finish. How to measure occupancy, spot uncoalesced loads and warp divergence, and read the three numbers in Nsight Compute that actually matter.

Why naive matmul on a GPU is bandwidth-starved, and how tiling with __shared__ memory reduces global memory traffic by a factor of the tile size. The classic optimisation, with the kernel that demonstrates it.