Tangent Spaces, Metrics, and the Riemannian Gradient

The previous lesson said a manifold looks flat locally. The tangent space is the precise version of that statement, and it is the object every algorithm actually works in.

At a point x on the manifold, consider every smooth curve passing through x while staying on the manifold. Each curve has a velocity vector at x. The set of all such velocities is the tangent space at x, written $T_x\mathcal{M}$.

Three properties make it the right object.

It is a genuine vector space: you can add tangent vectors and scale them, so ordinary linear algebra applies inside it. It has the same dimension as the manifold, so it captures exactly the available degrees of freedom. And it depends on where you are: the tangent plane at the north pole of a sphere differs from the one at the equator, which is what curvature means concretely.

For the sphere the description is immediate. A curve staying on the sphere cannot move radially, so tangent vectors at x are exactly those orthogonal to x. The tangent space is the plane through the origin perpendicular to the position vector.

Each common manifold has a tangent space with a clean characterisation, and these are the formulas an implementation needs.

Sphere. At x, tangent vectors v satisfy $x^\top v = 0$. Projection onto the tangent space subtracts the radial part: $P_x(u) = u - (x^\top u)\,x$.

Stiefel manifold. At X with orthonormal columns, tangent vectors satisfy $X^\top V + V^\top X = 0$, meaning $X^\top V$ is skew-symmetric. This is the differentiated form of the constraint $X^\top X = I$.

Symmetric positive definite matrices. These form an open set within symmetric matrices, so the tangent space at any point is simply all symmetric matrices. The interesting geometry here lives in the metric rather than in the tangent space.

Fixed-rank matrices. Writing a rank-r matrix in factored form, tangent vectors decompose into a part changing the row space, a part changing the column space, and a part changing the core, which is what makes efficient implementations possible.

The pattern generalises. For a manifold defined by equations $h(x) = 0$, the tangent space is the null space of the derivative of h: the directions in which the constraints do not change to first order.

A tangent space is a vector space, but to do optimisation you need geometry on it: lengths of vectors and angles between them. That is supplied by choosing an inner product on each tangent space, and a smoothly varying choice is a Riemannian metric.

The metric decides what "steepest" means, which is why it is not an incidental detail. The gradient is defined as the direction of fastest increase per unit length, and changing which vectors count as short changes which direction is steepest.

Two choices dominate practice.

The embedded metric simply inherits the ambient inner product. If your manifold sits inside a matrix space, use the ordinary trace inner product restricted to tangent vectors. It is the default and it is usually right.

A problem-adapted metric deviates deliberately. The standard example is symmetric positive definite matrices, where the natural geometry is not the flat one. Under the affine-invariant metric, the boundary where a matrix loses positive definiteness sits at infinite distance, so an algorithm respecting that geometry can never step outside the set. Interpolation between covariances also behaves better, avoiding the determinant shrinkage that flat averaging produces.

Now the central definition, and the good news is how little new machinery it requires.

The Riemannian gradient of f at x is the unique tangent vector g such that, for every tangent direction v, the directional derivative of f along v equals the inner product of g with v under the metric.

That is the same definition as the Euclidean gradient, with two restrictions: the direction must be tangent, and the inner product is the Riemannian one.

For an embedded manifold with the inherited metric, this yields a formula that is almost anticlimactic:

$\operatorname{grad} f(x) = P_x\big(\nabla f(x)\big)$

Project the ordinary Euclidean gradient onto the tangent space. That is all.

The practical consequence is large. You do not need to redevelop calculus. Compute the ambient gradient however you normally would, including by automatic differentiation, then apply the manifold's projection.

The interpretation is worth keeping. The normal component of the ambient gradient points off the manifold, where you cannot go. Discarding it is not an approximation; it is removing a direction that was never available.

This is the pipeline every Riemannian first-order method follows. Only two manifold-specific operations are needed, projection and retraction, and everything else is standard calculus.

Take the dominant eigenvector problem again and derive the gradient properly.

Maximise $f(x) = x^\top A x$ over unit vectors, with A symmetric.

Ambient gradient. Ordinary calculus gives $\nabla f(x) = 2Ax$.

Tangent projection. On the sphere, project by subtracting the radial component: $P_x(u) = u - (x^\top u)x$.

Riemannian gradient. Apply the projection:

$\operatorname{grad} f(x) = 2Ax - 2(x^\top A x)\,x$

Now read what this says. The Riemannian gradient vanishes exactly when $Ax = (x^\top A x)\,x$, which states that x is an eigenvector of A with eigenvalue $x^\top A x$.

So the critical points of the Riemannian problem are precisely the eigenvectors, recovered without imposing the eigenvalue equation anywhere. The geometry produced it.

def rgrad(A, x):
    egrad = 2 * A @ x                 # ambient gradient
    return egrad - (x @ egrad) * x    # project onto the tangent space

Two lines, and the second is the only one that knows about the sphere.

First-order methods need only the gradient. Trust-region and Newton-type methods need curvature, and this is where genuine geometry enters rather than mere projection.

The Riemannian Hessian is not simply the projected Euclidean Hessian. There is an extra term, and the reason is instructive: as you move along the manifold, the tangent space itself rotates. Differentiating a vector field on a curved space has to account for the fact that the space in which the vector lives is changing.

The tool that handles this is the Riemannian connection, and the resulting Hessian has the form

$\operatorname{Hess} f(x)[v] = P_x\big(\nabla^2 f(x)[v]\big) + \text{a curvature correction}$

On the sphere the correction term involves the radial component of the ambient gradient. If you forget it, your Newton method converges slowly or not at all, and the bug is subtle because the code still runs and still improves the objective.

This is the first place where the geometry is not optional bookkeeping. For gradient methods you can get by with projection; for second-order methods you need the connection.

The practical summary: to run Riemannian optimisation on a new manifold, you must supply a small fixed list of operations.

This is exactly the interface the Manopt toolbox exposes. Boumal is a lead developer of it, and its manifold descriptions consist of projections onto tangent spaces, retractions, and helpers converting Euclidean derivatives into Riemannian ones. It was introduced in the Journal of Machine Learning Research, volume 15.

The payoff of that interface is separation of concerns: the algorithms are written once against these primitives, and adding a manifold means implementing the list rather than modifying any solver.

One gotcha for practitioners: your objective's gradient must be the ambient one, computed as if unconstrained. Feeding an already-projected gradient into a library that projects again is a common and quiet error.