Eigenvalues and Eigenvectors

Most vectors get both rotated and scaled when you multiply by a matrix. But special vectors — eigenvectors — only get scaled:

$A\mathbf{v} = \lambda\mathbf{v}, \quad \mathbf{v}\ne\mathbf{0}.$

$\lambda\in\mathbb{C}$ is the eigenvalue (the scaling factor); $\mathbf{v}$ is the corresponding eigenvector (the invariant direction).

Geometric meaning: $A$ maps the line through $\mathbf{v}$ back to itself, just stretched or flipped. If $\lambda>1$, the direction is amplified. If $0<\lambda<1$, it's compressed. If $\lambda<0$, it's flipped and scaled. If $\lambda=0$, the entire direction is killed — $\mathbf{v}$ lands in the null space.

This is the skeleton of the transformation: eigenvectors are the axes that $A$ acts on most simply. Any other vector is a superposition of eigenvectors, and the matrix just scales each component independently.

$A\mathbf{v}=\lambda\mathbf{v}$ rearranges to $(A-\lambda I)\mathbf{v}=\mathbf{0}$. For a nonzero $\mathbf{v}$ to exist, $A-\lambda I$ must be singular:

$\det(A - \lambda I) = 0.$

This is the characteristic polynomial in $\lambda$. For an $n\times n$ matrix it has degree $n$, so there are exactly $n$ eigenvalues (counted with multiplicity, possibly complex).

Example. $A=\begin{bmatrix}3&1\\0&2\end{bmatrix}$:

$\det(A-\lambda I) = (3-\lambda)(2-\lambda) - 0 = \lambda^2 - 5\lambda + 6 = (\lambda-3)(\lambda-2).$

Eigenvalues: $\lambda_1=3$, $\lambda_2=2$. Then solve $(A-3I)\mathbf{v}=\mathbf{0}$ for $\mathbf{v}_1$ and $(A-2I)\mathbf{v}=\mathbf{0}$ for $\mathbf{v}_2$:

$\mathbf{v}_1=\begin{bmatrix}1\\0\end{bmatrix},\quad \mathbf{v}_2=\begin{bmatrix}1\\-1\end{bmatrix}.$

If an $n\times n$ matrix has $n$ linearly independent eigenvectors, pack them as columns of $P$ and their eigenvalues on the diagonal of $D$:

$AP = PD \implies A = PDP^{-1}, \quad D = \begin{bmatrix}\lambda_1 & & \\ & \ddots & \\ & & \lambda_n\end{bmatrix}.$

This is diagonalization. Why it matters — matrix powers become trivial:

$A^k = (PDP^{-1})^k = PD^kP^{-1}, \quad D^k = \begin{bmatrix}\lambda_1^k & & \\ & \ddots & \\ & & \lambda_n^k\end{bmatrix}.$

Multiplying $A$ by itself $k$ times is now just raising $n$ scalars to the $k$th power. Computing $A^{100}$ for a $1000\times 1000$ matrix is otherwise nightmarish; in the eigenbasis, it's trivial. Diagonalization is the upgrade from "apply $A$ repeatedly" to "scale each component by $\lambda^k$".

For a real symmetric matrix $A=A^\top$, the spectral theorem guarantees:

1. All eigenvalues are real.
2. Eigenvectors from different eigenvalues are orthogonal.
3. $A$ is always diagonalizable: $A=Q\Lambda Q^\top$ where $Q$ has orthonormal columns.

This orthogonal diagonalization — $Q^{-1}=Q^\top$ — is far nicer than generic $PDP^{-1}$. Symmetric matrices arise everywhere: covariance matrices, graph Laplacians, Hessians in optimization, kernel matrices in SVMs. Their eigenvalues are real (no complex surprises), and their eigenvectors form a clean orthonormal basis of $\mathbb{R}^n$.

For PCA, the covariance matrix $\Sigma=(1/m)X^\top X$ is symmetric positive semi-definite: all eigenvalues $\ge 0$, and the eigenvectors are the principal components — the directions of maximum variance in the data.

Eigenstructure drives three landmark algorithms:

PCA (Principal Component Analysis). Given centered data matrix $X$, form $\Sigma=X^\top X$. Its eigenvectors are the principal components. Project data onto the top-$k$ eigenvectors to reduce dimension while preserving maximum variance. Eigenvalue $\lambda_i$ tells you the variance explained by component $i$.

PageRank. The web is a graph; $A_{ij}=1/\text{out-degree}(j)$ if page $j$ links to $i$. The rank vector $\mathbf{r}$ satisfies $A\mathbf{r}=\mathbf{r}$ — it is the eigenvector for eigenvalue 1 (the Perron eigenvector of a stochastic matrix). Google's trillion-dollar insight: importance is the dominant eigenvector.

Dynamical stability. A linear system $\mathbf{x}_{t+1}=A\mathbf{x}_t$ is stable iff every eigenvalue has $|\lambda|<1$. If any $|\lambda|>1$, the system blows up. The eigenvalue with largest magnitude — the spectral radius $\rho(A)$ — determines long-term behavior.

An eigenvalue $\lambda$ may appear more than once as a root of the characteristic polynomial.

• Algebraic multiplicity $m_a$: how many times $\lambda$ is a root of $\det(A-\lambda I)$.
• Geometric multiplicity $m_g$: the dimension of the eigenspace $\ker(A-\lambda I)$ — how many independent eigenvectors share $\lambda$.

Always $1\le m_g\le m_a$. If $m_g < m_a$ for any eigenvalue, $A$ is defective — it lacks enough eigenvectors to diagonalize.

Example: $A=\begin{bmatrix}2&1\\0&2\end{bmatrix}$ has $\lambda=2$ with $m_a=2$ but $m_g=1$ (the eigenspace is only $\operatorname{span}([1,0]^\top)$). $A$ cannot be diagonalized; its Jordan form has a $2\times 2$ Jordan block. Defective matrices are rare in practice (they require exact coincidences) but occur in control theory and differential equations.

NumPy computes eigenvalues via np.linalg.eig (general) or np.linalg.eigh (symmetric, faster and guaranteed real):

import numpy as np

# General matrix
A = np.array([[3, 1], [0, 2]], dtype=float)
vals, vecs = np.linalg.eig(A)
print("eigenvalues:", vals)        # [3., 2.]
print("eigenvectors:\n", vecs)     # columns are eigenvectors

# Verify A @ v = lambda * v
for i in range(len(vals)):
    lhs = A @ vecs[:, i]
    rhs = vals[i] * vecs[:, i]
    print(f"lambda={vals[i]:.1f}: residual = {np.max(np.abs(lhs - rhs)):.2e}")

# Diagonalization: A = P D P^{-1}
P = vecs
D = np.diag(vals)
A_reconstructed = P @ D @ np.linalg.inv(P)
print("Reconstruction error:", np.max(np.abs(A - A_reconstructed)))  # ~1e-15

# Powers via diagonalization
k = 10
Ak = P @ np.diag(vals**k) @ np.linalg.inv(P)
print("A^10 via diagonalization:\n", np.round(Ak))

For a symmetric matrix, always use eigh — it's faster ($O(n^3/3)$ vs $O(n^3)$) and returns real eigenvalues in sorted order.

Not every matrix is diagonalizable. Compare two matrices with the same characteristic polynomial:

The defective matrix $A$ produces polynomial factors when iterated — critical in delay-differential equations and resonance phenomena. The Jordan form $J=\begin{bmatrix}2&1\\0&2\end{bmatrix}$ is the canonical form for this behavior. It's not a curiosity: defective modes appear in structural mechanics (repeated natural frequencies under damping) and in digital filters.

Two elegant identities connect eigenvalues to matrix entries:

$\operatorname{tr}(A) = \sum_{i=1}^n \lambda_i \qquad \text{(trace = sum of eigenvalues)}$

$\det(A) = \prod_{i=1}^n \lambda_i \qquad \text{(determinant = product of eigenvalues)}$

These are immediate from the characteristic polynomial $(\lambda-\lambda_1)\cdots(\lambda-\lambda_n)$. Quick consequences:

• $\det(A)=0$ iff at least one $\lambda_i=0$ — confirming the singular/null-space connection.
• For a $2\times 2$ matrix: $\lambda^2 - \operatorname{tr}(A)\lambda + \det(A) = 0$ — a quick formula for eigenvalues without expanding the full determinant.
• The trace is invariant under similarity transformations: $\operatorname{tr}(PDP^{-1})=\operatorname{tr}(D)=\sum\lambda_i$. The trace is also the sum of diagonal entries — so you can read off $\sum\lambda_i$ instantly from the matrix without computing a single eigenvalue.