Neural Networks and Backpropagation

A single neuron takes a vector of inputs $\mathbf{x} \in \mathbb{R}^n$, computes a weighted sum, adds a bias, and passes the result through a nonlinearity:

$y = \sigma\!\left(\mathbf{w}^\top \mathbf{x} + b\right)$

$\mathbf{w}$ and $b$ are the learnable parameters; $\sigma$ is the activation function. Without $\sigma$, stacking neurons just produces another linear map — the whole network collapses to a single matrix multiply. The activation is what makes deep networks expressive.

Biologically inspired but only loosely: real neurons fire spikes; artificial ones output a smooth scalar. What matters is the math, not the metaphor.

The three workhorses, and when to reach for each:

ReLU dominates hidden layers: cheap to compute, sparse activations, gradient stays 1 for positive inputs (no vanishing). Its flaw — "dead neurons" (always zero) — is fixed by Leaky ReLU or GELU. Sigmoid and tanh saturate at extremes, causing near-zero gradients; avoid them in deep hidden layers.

Stack neurons in parallel to get a layer: for $m$ neurons with weight matrix $W \in \mathbb{R}^{m \times n}$ and bias $\mathbf{b} \in \mathbb{R}^m$,

$\mathbf{h} = \sigma(W\mathbf{x} + \mathbf{b})$

A forward pass threads the input through $L$ such layers in sequence: $\mathbf{h}^{(0)} = \mathbf{x}$, then $\mathbf{h}^{(l)} = \sigma^{(l)}(W^{(l)}\mathbf{h}^{(l-1)} + \mathbf{b}^{(l)})$ for $l = 1, \ldots, L$. The final layer's output is the prediction $\hat{y}$.

Every intermediate value $\mathbf{h}^{(l)}$ (called an activation) must be stored during the forward pass — backpropagation will need them to compute gradients.

Suppose every $\sigma$ were the identity. Then:

$\mathbf{h}^{(L)} = W^{(L)} \cdots W^{(2)} W^{(1)} \mathbf{x} = W_{\text{eff}}\, \mathbf{x}$

No matter how many layers you add, the network is a single linear map. It can only fit data that lies on a hyperplane — it cannot learn XOR, spirals, or image classifiers.

The universal approximation theorem guarantees that a two-layer network with enough neurons and a nonlinear $\sigma$ can approximate any continuous function on a compact domain to arbitrary precision. Depth gives efficiency: exponentially fewer neurons needed to represent the same function compared to a single-layer network.

The network is only as good as what it's trained to minimize:

• Mean squared error (regression): $\mathcal{L} = \frac{1}{N}\sum_i (y_i - \hat{y}_i)^2$
• Cross-entropy (classification): $\mathcal{L} = -\frac{1}{N}\sum_i y_i \log \hat{y}_i$

The loss collapses the full prediction into a single scalar. Training = finding weights $\{W^{(l)}, \mathbf{b}^{(l)}\}$ that make this scalar small on the training set.

The gradient of $\mathcal{L}$ with respect to every parameter tells us which direction to move each weight. Computing those gradients efficiently is backpropagation's entire job.

If $\mathcal{L}$ depends on $z$ which depends on $w$ (via $z = wh$), then:

$\frac{\partial \mathcal{L}}{\partial w} = \frac{\partial \mathcal{L}}{\partial z} \cdot \frac{\partial z}{\partial w}$

Backpropagation applies this recursively from the output layer back to the first layer. Define the upstream gradient $\delta^{(l)} = \partial \mathcal{L}/\partial \mathbf{z}^{(l)}$ (where $\mathbf{z}^{(l)} = W^{(l)}\mathbf{h}^{(l-1)} + \mathbf{b}^{(l)}$ is the pre-activation). Then:

$\delta^{(l-1)} = \left({W^{(l)}}^\top \delta^{(l)}\right) \odot \sigma'^{(l-1)}(\mathbf{z}^{(l-1)})$

$\odot$ is elementwise multiply, $\sigma'$ is the activation derivative. The weight gradient is then $\partial \mathcal{L}/\partial W^{(l)} = \delta^{(l)} {\mathbf{h}^{(l-1)}}^\top$.

A minimal two-layer network with ReLU, squared loss, and exact gradients:

import numpy as np

# Data
X = np.array([[1.0, 2.0]])   # (1, 2) input
y = np.array([[0.5]])         # scalar target

# Parameters
W1 = np.random.randn(4, 2) * 0.1
b1 = np.zeros((4, 1))
W2 = np.random.randn(1, 4) * 0.1
b2 = np.zeros((1, 1))

# Forward pass
z1 = W1 @ X.T + b1            # (4,1)
h1 = np.maximum(0, z1)        # ReLU
z2 = W2 @ h1 + b2             # (1,1)
y_hat = z2                    # linear output
loss = 0.5 * (y_hat - y.T)**2

# Backward pass
dL_dz2 = y_hat - y.T          # (1,1)
dL_dW2 = dL_dz2 @ h1.T       # (1,4)
dL_dh1 = W2.T @ dL_dz2       # (4,1)
dL_dz1 = dL_dh1 * (z1 > 0)   # ReLU grad
dL_dW1 = dL_dz1 @ X           # (4,2)

Notice: the ReLU gradient is simply the indicator z1 > 0. Every gradient flows cleanly back via the chain rule.

The chain rule multiplies Jacobians across layers. With sigmoid activations, each factor is $\sigma'(z) = \sigma(z)(1-\sigma(z)) \le 0.25$. Multiply 20 such terms and the gradient at the first layer is $\approx 0.25^{20} \approx 10^{-12}$ — effectively zero. Weights in early layers stop updating: the vanishing gradient problem.

Solutions and their mechanisms:

• ReLU: gradient is 1 for positive inputs — no shrinkage.
• Residual connections (ResNet): add a skip $\mathbf{h}^{(l)} + F(\mathbf{h}^{(l)})$ so gradients have a direct path.
• Batch normalization: keeps activations in a healthy range, stabilizing gradient magnitudes.
• LSTM gates: learned gating prevents gradients from decaying over time in sequences.

Data flows left to right; gradients flow right to left.

A single gradient-descent update bundles everything:

1. Forward pass — compute $\hat{y}$ and $\mathcal{L}$, cache intermediate activations.
2. Backward pass — propagate $\partial \mathcal{L}/\partial W^{(l)}$ and $\partial \mathcal{L}/\partial \mathbf{b}^{(l)}$ for every layer.
3. Parameter update — $W^{(l)} \leftarrow W^{(l)} - \eta \cdot \partial \mathcal{L}/\partial W^{(l)}$, where $\eta$ is the learning rate.

Modern frameworks (PyTorch, JAX) do steps 1–3 automatically via automatic differentiation: they build a computational graph during the forward pass and call .backward() to traverse it in reverse.

Backpropagation doesn't magically find good weights — it's gradient descent that does. Backprop just makes computing the gradients cheap: $O(\text{forward pass cost})$ rather than the $O(P)$ forward passes that finite differences would need.