Contrastive Learning

The previous lesson ended on the collapse problem: making two views of an image agree, on its own, lets the model cheat by mapping everything to a constant. Contrastive learning blocks that cheat with a second force.

The idea has two parts working against each other. For a given image, its two augmented views form a positive pair, and their embeddings are pulled together. Every other image in the batch is a negative, and its embedding is pushed apart from the anchor. Attraction alone would collapse; repulsion alone would scatter meaninglessly. Together they carve out a representation where each image sits near its own augmentations and far from everything else. This push-and-pull is the entire mechanism, and it was the first approach to make joint-embedding SSL competitive with supervised pretraining.

The objective that implements pull-together-push-apart is InfoNCE (Noise-Contrastive Estimation applied to information). For an anchor embedding, InfoNCE is essentially a classification loss: among the anchor, its one positive, and many negatives, correctly identify the positive by similarity.

Concretely, you compute the similarity of the anchor to its positive and to every negative, scale by a temperature, and apply a softmax-style loss that is minimized when the positive is by far the most similar. A lower temperature sharpens the focus on the hardest negatives. Theoretically, minimizing InfoNCE maximizes a lower bound on the mutual information between the two views, so the shared content survives while view-specific noise is discarded. In practice you do not need the theory to use it: InfoNCE says "the matching view should be the nearest, and all the others should be far," and that is enough to drive strong representations.

SimCLR (Chen and colleagues, 2020) showed how far a clean contrastive recipe could go. Its pipeline is deliberately simple: take an image, apply two strong random augmentations to make the two views, encode each, pass the result through a small projection head (a couple of layers used only during pretraining), and apply the InfoNCE loss.

SimCLR made two findings the field absorbed. First, strong augmentation matters enormously: the combination of random cropping and color distortion is what forces the model to learn semantic content rather than trivial cues. Second, it needs many negatives, and SimCLR gets them from the current batch. That means it depends on very large batch sizes, thousands of images, to supply enough negatives for the loss to bite. That appetite for huge batches, and the compute they require, is SimCLR's main practical limitation, and it directly motivated the next method.

MoCo (Momentum Contrast, He and colleagues, 2020) attacks SimCLR's batch-size problem directly. Instead of drawing negatives only from the current batch, MoCo maintains a queue: a running memory bank of embeddings from many recent batches, giving a large supply of negatives without needing a huge batch.

The catch is consistency. If the encoder changes quickly, embeddings sitting in the queue from earlier steps become stale and mismatched. MoCo's fix is a momentum encoder: a second copy of the encoder whose weights are updated as a slow moving average of the main encoder rather than by gradients. This slowly-changing encoder produces the queued embeddings, keeping them consistent over time. The result is many negatives at a modest batch size. SimCLR and MoCo thus reach the same goal by different routes, batch versus queue, and together they defined the contrastive era of SSL.

It is worth stating precisely why negatives solve the problem from the previous lesson. Collapse is the state where every embedding is the same constant vector. But InfoNCE explicitly demands that each negative be dissimilar from the anchor. If all embeddings were identical, every negative would be maximally similar to the anchor, which is exactly what the loss punishes hardest.

So the repulsion term makes collapse the worst possible solution rather than the easiest one.

# InfoNCE, in essence (one anchor)
sim_pos = dot(anchor, positive) / tau
sim_neg = [dot(anchor, n) / tau for n in negatives]
loss = -log( exp(sim_pos) / (exp(sim_pos) + sum(exp(s) for s in sim_neg)) )

The loss shrinks only when the positive is close and the negatives are far. A collapsed encoder, where positive and negatives are equally close, gives the maximum loss. Negatives are, quite literally, the force that holds the representation open.

One anchor image yields two views; their embeddings are pulled together as a positive pair. Other images, supplied either by a large batch (SimCLR) or a momentum-encoder queue (MoCo), are negatives whose embeddings are pushed away. The InfoNCE loss balances both forces, and the repulsion is what forbids collapse.

Contrastive learning works, but negatives carry real costs that shaped what came next:

The false-negative problem is especially awkward: the method blindly treats every other image as a negative, so two different dogs get repelled even though a good representation would place them near each other. These frictions raised a natural question: could you prevent collapse without any negatives at all, keeping only the pull-together half? Answering yes, in more than one way, is the subject of the final lesson.