numerical-methods

Two different things can make a numerical answer inaccurate: the problem may amplify any input perturbation, or the algorithm may introduce its own. They are separate quantities, one belongs to the problem and one to the code, and only the second is fixable. This lesson separates them and shows what follows.

Equality fails on floats, and both standard replacements fail too: absolute tolerance breaks at scale, relative tolerance breaks near zero. This lesson works through why each fails, what the combined form actually does, and how to choose a tolerance from the problem rather than copying a magic constant.

Every operation is correctly rounded, yet results can be wrong in the first digit. The reason is that subtracting nearly equal numbers is exact and still catastrophic: it does not create error, it promotes error that was already there. This lesson builds that mechanism and the summation techniques that recover the lost accuracy.

Floating point is not a slightly inaccurate version of real numbers. It is a finite set of values, spaced logarithmically, with exact rules. This lesson builds the IEEE 754 layout, shows why 0.1 cannot be represented, and derives the one error bound that every later result depends on.

Before you can differentiate a renderer you have to see it as mathematics: the rendering equation, why it has no closed-form solution, and how Monte Carlo path tracing turns light transport into an estimation problem.

Where this machinery earns its place: low-rank matrix completion, synchronization, and PCA. Plus the two ideas that explain why non-convex manifold problems are often solved to global optimality anyway.

How to move along a curved space without solving differential equations: retractions as cheap approximations to geodesics, vector transport, and the Riemannian versions of gradient descent, conjugate gradients, and trust regions.

Building the machinery: the linear space of allowed directions at a point, the inner product that gives it geometry, and why the Riemannian gradient is the ambient gradient projected rather than a new derivative.

Rotations, subspaces, low-rank matrices and covariances are not vectors in flat space. Why treating those constraints as penalties or projections wastes structure, and what it means to say the search space is a manifold.