Convolutional Neural Networks

A 224×224 RGB image has $224 \times 224 \times 3 = 150{,}528$ input values. A fully-connected (FC) hidden layer with 1024 neurons would require $150{,}528 \times 1024 \approx 154$ million parameters — for a single layer. Problems:

• Parameter explosion: you run out of memory before the network gets interesting.
• No spatial inductive bias: FC layers treat pixel (0, 0) and pixel (100, 100) as unrelated — they know nothing about locality.
• No translation invariance: a cat in the top-left corner activates completely different weights than the same cat in the bottom-right corner.

Images are not arbitrary 1D vectors. Nearby pixels are strongly correlated, and the same pattern (edge, texture) can appear anywhere. The convolution operation exploits exactly this structure.

A 2D convolution slides a small filter $K \in \mathbb{R}^{k \times k}$ over the input $I$ and computes a dot product at each position:

$(I * K)[i, j] = \sum_{m=0}^{k-1} \sum_{n=0}^{k-1} I[i+m,\, j+n] \cdot K[m, n]$

The filter detects a pattern; the output — called a feature map — encodes where that pattern appears. Each value in the feature map requires only $k^2$ multiplications, regardless of image size.

import torch
import torch.nn as nn

# 3-channel input, 16 filters of size 3x3
conv = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, padding=1)
x = torch.randn(1, 3, 224, 224)  # (batch, channels, H, W)
out = conv(x)  # shape: (1, 16, 224, 224) with padding='same'

The 16 filters learn 16 different feature detectors. The first layers typically learn Gabor-like edge detectors; deeper layers learn complex semantic patterns.

Two hyperparameters control the spatial extent of the output:

Stride ($s$): how many pixels the filter moves at each step. $s=1$ produces a dense output; $s=2$ halves spatial dimensions (common for downsampling without pooling).

Padding ($p$): zeros added around the input border. padding=1 with a $3 \times 3$ kernel preserves spatial size.

Output size formula for input size $H$, kernel $k$, stride $s$, padding $p$:

$H_{\text{out}} = \left\lfloor \frac{H + 2p - k}{s} \right\rfloor + 1$

The 7×7, stride-2 pattern in ResNet's first layer aggressively shrinks the feature map early, reducing compute for the rest of the network.

A single filter has $k^2 \times C_{\text{in}}$ parameters, shared across every spatial position. This is the key savings: a $3 \times 3$ filter applied to a $224 \times 224$ input has only $3 \times 3 \times C_{\text{in}}$ weights, not $224^2 \times C_{\text{in}}$.

Translation equivariance: if the input shifts by $(\Delta i, \Delta j)$, the feature map shifts by the same amount — the detected pattern follows the object. Formally: $(\text{shift}(I)) * K = \text{shift}(I * K)$.

This is equivariance, not invariance — the output changes, but in a predictable way. True translation invariance (same output regardless of position) comes from pooling.

Parameter sharing also provides regularization: the filter is forced to be useful everywhere, not just at one location. This is a strong inductive bias for spatially stationary patterns like textures and edges.

Pooling summarizes a local region into a single value, discarding exact position and reducing spatial size:

• Max pooling (2×2, stride 2): takes the maximum of each 2×2 block. Halves $H$ and $W$. Retains the presence of a feature, ignoring exact location — providing local translation invariance.
• Average pooling: takes the mean. Smoother but less discriminative.
• Global average pooling (GAP): averages all spatial positions into a single value per channel. Replaces a large FC layer at the network head, used in ResNet and MobileNet.

# Max pooling: 224x224 -> 112x112
pool = nn.MaxPool2d(kernel_size=2, stride=2)

# Global average pooling: (B, C, H, W) -> (B, C, 1, 1)
gap = nn.AdaptiveAvgPool2d(1)
features = gap(x).squeeze(-1).squeeze(-1)  # (B, C)

Modern architectures (EfficientNet, Vision Transformer) often drop max pooling in favor of strided convolutions, which are learned rather than fixed.

The classic pattern is: repeated Conv → BN → ReLU → Pool blocks, followed by a classifier head.

class SimpleCNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(),
            nn.MaxPool2d(2),                   # 112x112
            nn.Conv2d(32, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),
            nn.MaxPool2d(2),                   # 56x56
            nn.Conv2d(64, 128, 3, padding=1), nn.BatchNorm2d(128), nn.ReLU(),
            nn.AdaptiveAvgPool2d(1),           # 1x1
        )
        self.classifier = nn.Linear(128, num_classes)

    def forward(self, x):
        x = self.features(x).squeeze(-1).squeeze(-1)
        return self.classifier(x)

Depth increases (32 → 64 → 128 channels) while spatial dimensions shrink — a hallmark of all CNN designs. Channels encode what, spatial positions encode where.

Spatial dimensions shrink; channel depth grows toward the classifier.

As networks grew deeper (50, 100, 152 layers), a paradox emerged: deeper networks performed worse than shallower ones — not from overfitting but from optimization failure. Gradients vanished before reaching early layers.

He et al. (2015) introduced the residual connection:

$\mathbf{y} = F(\mathbf{x}, \{W_i\}) + \mathbf{x}$

where $F$ is a stack of layers (e.g., Conv → BN → ReLU → Conv → BN). The $+\mathbf{x}$ skip connection lets the gradient flow directly backward through the identity path:

$\frac{\partial \mathcal{L}}{\partial \mathbf{x}} = \frac{\partial \mathcal{L}}{\partial \mathbf{y}}\left(1 + \frac{\partial F}{\partial \mathbf{x}}\right)$

Even if $\partial F/\partial \mathbf{x} \approx 0$, the gradient is at least $\partial \mathcal{L}/\partial \mathbf{y}$ — it never vanishes completely. The network learns the residual $F(\mathbf{x})$ rather than the full transformation, which is an easier optimization problem when $F \approx 0$ is a good starting point (identity shortcut).

The evolution of CNN design encodes hard-won lessons:

Key takeaways: small 3×3 kernels stack cheaper than one large kernel (VGG). 1×1 convolutions reduce channel depth before expensive 3×3 convs (bottleneck). Residual connections are now standard in almost all deep architectures, including transformers.