Vectors, Spans, and Subspaces

A vector in $\mathbb{R}^n$ lives two simultaneous lives.

Geometric life: an arrow from the origin, carrying direction and magnitude. Adding two arrows tip-to-tail gives a third; scaling stretches or flips.

Algebraic life: a column of numbers:

$\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix} \in \mathbb{R}^n.$

Both views are true at once. When computing, think algebraically. When confused, draw arrows. A neural network's weight vector may be $10^6$-dimensional — impossible to draw — but the arrow intuition still tells you whether two gradient steps "agree" (positive dot product) or "fight" (negative). Throughout this course: bold lowercase for vectors ($\mathbf{v}$), $\|\mathbf{v}\|$ for Euclidean norm, $\mathbf{u}^\top\mathbf{v}$ for the dot product.

A linear combination of $\mathbf{v}_1, \dots, \mathbf{v}_k$ is any expression

$c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \cdots + c_k\mathbf{v}_k, \quad c_i \in \mathbb{R}.$

The span is the set of all such combinations — every reachable point:

$\operatorname{span}(\mathbf{v}_1, \dots, \mathbf{v}_k) = \left\{\sum_{i=1}^k c_i \mathbf{v}_i \;\ \middle|\;\ c_i \in \mathbb{R}\right\}.$

Geometrically: one nonzero vector spans a line; two non-collinear vectors span a plane; add a third off that plane and you span all of $\mathbb{R}^3$. Add a fourth — the span doesn't grow, because you can't escape $\mathbb{R}^3$.

This is the first hint that dimension is about the smallest set of vectors you need, not the largest. The span is always a subspace: contains $\mathbf{0}$, closed under addition and scalar multiplication.

Vectors $\mathbf{v}_1, \dots, \mathbf{v}_k$ are linearly independent if the only solution to

$c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k = \mathbf{0}$

is $c_1 = \cdots = c_k = 0$. Otherwise one vector hides inside the span of the rest — pure redundancy.

Key facts:

• Two vectors are dependent iff one is a scalar multiple of the other.
• You can never fit more than $n$ independent vectors into $\mathbb{R}^n$.
• Three vectors in $\mathbb{R}^2$ are always dependent.

Worked example. Are $\mathbf{v}_1 = [1,2,3]^\top$, $\mathbf{v}_2 = [2,5,7]^\top$, $\mathbf{v}_3 = [1,3,4]^\top$ independent? Row-reduce:

$\begin{bmatrix}1&2&1\\2&5&3\\3&7&4\end{bmatrix} \xrightarrow{R_2-2R_1,\,R_3-3R_1} \begin{bmatrix}1&2&1\\0&1&1\\0&1&1\end{bmatrix} \xrightarrow{R_3-R_2} \begin{bmatrix}1&2&1\\0&1&1\\0&0&0\end{bmatrix}.$

A zero row means a free variable: dependent. Indeed, $\mathbf{v}_3 = \mathbf{v}_2 - \mathbf{v}_1$.

A basis of a subspace $V$ is a set of vectors that (1) spans $V$ and (2) is linearly independent — no redundancy, nothing missing.

The standard basis of $\mathbb{R}^3$:

$\mathbf{e}_1=\begin{bmatrix}1\\0\\0\end{bmatrix},\quad \mathbf{e}_2=\begin{bmatrix}0\\1\\0\end{bmatrix},\quad \mathbf{e}_3=\begin{bmatrix}0\\0\\1\end{bmatrix}.$

Every basis of $V$ has exactly the same size. That common size is the dimension $\dim(V)$. A line through the origin: $\dim=1$. A plane through the origin: $\dim=2$. All of $\mathbb{R}^n$: $\dim=n$.

Every vector in $V$ has a unique coordinate representation in a given basis: $\mathbf{v} = x_1\mathbf{b}_1 + x_2\mathbf{b}_2 + \cdots + x_k\mathbf{b}_k.$

Change the basis, change the numbers — but the vector itself is the same geometric object. Basis changes are the engine behind change-of-variable tricks everywhere in applied math.

The dot product of $\mathbf{u},\mathbf{v}\in\mathbb{R}^n$:

$\mathbf{u}\cdot\mathbf{v} = \mathbf{u}^\top\mathbf{v} = \sum_{i=1}^n u_i v_i.$

The Euclidean norm: $\|\mathbf{v}\| = \sqrt{\mathbf{v}^\top\mathbf{v}}$. The single most useful formula connecting algebra to geometry:

$\mathbf{u}^\top\mathbf{v} = \|\mathbf{u}\|\,\|\mathbf{v}\|\cos\theta \quad\Longrightarrow\quad \cos\theta = \frac{\mathbf{u}^\top\mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|}.$

Orthogonality ($\mathbf{u}^\top\mathbf{v}=0$) means the vectors share zero information. The Cauchy-Schwarz inequality $|\mathbf{u}^\top\mathbf{v}| \le \|\mathbf{u}\|\,\|\mathbf{v}\|$ guarantees $\cos\theta\in[-1,1]$, making the angle formula valid in any dimension.

An orthonormal basis $\{\mathbf{q}_1,\dots,\mathbf{q}_n\}$ satisfies $\mathbf{q}_i^\top\mathbf{q}_j = \delta_{ij}$ — pairwise orthogonal unit vectors. These are golden: coordinates are just dot products.

The projection of $\mathbf{b}$ onto $\mathbf{a}$:

$\operatorname{proj}_{\mathbf{a}}\mathbf{b} = \frac{\mathbf{a}^\top\mathbf{b}}{\mathbf{a}^\top\mathbf{a}}\,\mathbf{a}.$

The residual $\mathbf{b} - \operatorname{proj}_{\mathbf{a}}\mathbf{b}$ is orthogonal to $\mathbf{a}$ — this right-angle split is the engine of least squares.

Gram-Schmidt converts any basis to an orthonormal one: at each step, subtract off the projections onto vectors already processed, then normalize. Mechanically: "remove what I've already explained, then report what's new and perpendicular." GS underlies QR factorization, which is how numerical linear algebra software actually runs least-squares solvers — Gaussian elimination is slower and less stable.

A subspace of $\mathbb{R}^n$ must satisfy three conditions:

1. Contains $\mathbf{0}$.
2. Closed under addition: $\mathbf{u},\mathbf{v}\in V \Rightarrow \mathbf{u}+\mathbf{v}\in V$.
3. Closed under scalar multiplication: $\mathbf{v}\in V, c\in\mathbb{R} \Rightarrow c\mathbf{v}\in V$.

Critical trap: the set $\{\mathbf{x}: A\mathbf{x}=\mathbf{b}\}$ with $\mathbf{b}\ne\mathbf{0}$ is not a subspace — it misses the origin. It is an affine subspace (one particular solution plus the null space). This distinction is essential when studying solutions to linear systems.

Row reduction by hand is tedious. NumPy makes it a one-liner:

import numpy as np

v1 = np.array([1, 2, 3], dtype=float)
v2 = np.array([2, 5, 7], dtype=float)
v3 = np.array([1, 3, 4], dtype=float)

A = np.column_stack([v1, v2, v3])  # columns are our vectors
rank = np.linalg.matrix_rank(A)
print(f"Rank = {rank}")          # 2 → dependent! (v3 = v2 - v1)
print(f"v3 = v2 - v1: {np.allclose(v3, v2 - v1)}")  # True

# Angle between v1 and v2
cos_t = np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))
angle = np.degrees(np.arccos(np.clip(cos_t, -1, 1)))
print(f"Angle(v1, v2) = {angle:.2f}°")  # ~5.77° — nearly parallel

matrix_rank uses SVD under the hood and counts singular values above a threshold. Never use np.linalg.det to check independence for floating-point data — near-zero determinants look like zero even when they aren't. SVD-based rank is the numerically stable professional tool.

Here's the foundation this lesson built:

• Vectors are elements of $\mathbb{R}^n$ with dual geometric-algebraic identity.
• Linear combinations are the only operations linear algebra permits — scale and add.
• Span is the reachable set; subspace is its rigorous container.
• Independence means no redundancy; basis = minimal spanning set.
• Dimension = basis size, an intrinsic property of the subspace.
• Dot product encodes length, angle, and orthogonality in a single formula.

The next lesson asks: what happens when you apply a matrix to every vector in $\mathbb{R}^n$? The answer reshapes space — stretching, rotating, collapsing — and understanding it requires everything built here. Rank, column space, null space, and invertibility all trace back to span, independence, and dimension.