Why Predict Representations, Not Pixels

Supervised learning needs labels, and labels are scarce and expensive. Self-supervised learning (SSL) sidesteps this by inventing a prediction task out of the unlabeled data itself: hide part of the input, predict it from the rest. No human annotation required, so the method can consume the effectively unlimited supply of raw images, video, and text.

The payoff is a representation: an encoder that maps raw input to a vector capturing what matters. A good representation transfers, you can attach a small classifier or controller and solve downstream tasks with few labels. This is the engine behind modern foundation models.

The entire design question in SSL is: what exactly should the model predict, and in what space? That single choice determines what the representation learns. Predict the wrong thing and the encoder wastes its capacity on details nobody needs.

Three broad families answer that question differently:

• Generative / reconstructive: predict the missing input itself (the pixels).
• Contrastive: make two views of the same thing agree, and different things disagree.
• Joint-embedding predictive: predict the representation of the missing part, not the part itself.

This lesson examines all three, then shows why the third, the family that includes JEPA, is a distinct and compelling answer.

The most intuitive family is generative: mask part of the input and reconstruct it in input space. The masked autoencoder (MAE) of He and colleagues is the canonical vision example, mask most patches of an image, and have a decoder rebuild the missing pixels. In language, masked and next-token prediction work the same way and are spectacularly effective.

The appeal is obvious: the objective is unambiguous, needs no augmentations, and the pixel target is exactly defined. But in vision it carries a hidden tax.

The model must predict everything, including what is fundamentally unpredictable. Consider masking a patch of grass, a tree canopy, or the texture of a carpet. The precise arrangement of blades, leaves, and fibers is essentially noise: no amount of understanding lets you recover those exact pixels. Yet a pixel-level loss penalizes every wrong value, so the network is pushed to spend capacity modeling irreducible detail.

The consequence, argued in the JEPA line of work, is that pixel-level objectives tend to produce less semantically meaningful representations: the encoder is dragged toward low-level texture statistics rather than the structure that actually matters. The information you want, there is a tree, it is in front of a house, is a tiny fraction of the bits the loss cares about. You are optimizing overwhelmingly for detail you will then throw away.

The second family, contrastive learning (SimCLR, MoCo and relatives), never reconstructs anything. Instead it takes two augmented views of the same image, a crop, a color jitter, a flip, and trains their embeddings to be close, while pushing apart embeddings of different images (the negatives). The negatives are essential: without something pushing representations apart, the model could map everything to one point.

This works well and produces strongly semantic features. But it has a structural cost: the augmentations define what the model learns to ignore.

By training a crop and its color-jittered twin to have identical embeddings, you are declaring "color and position do not matter." The representation becomes invariant to whatever you augmented. That is a hand-crafted, human-chosen prior, and it is biased for particular downstream tasks. Invariance to color is helpful for object recognition and harmful for a task where color is the signal (ripe versus unripe fruit, a warning light). Invariance to position hurts anything spatial.

So contrastive learning buys semantics by injecting a fixed set of human assumptions about what is irrelevant. Those assumptions do not transfer cleanly to every task or every domain, and choosing the augmentation recipe becomes a delicate, domain-specific art rather than something learned from the data.

The joint-embedding predictive architecture (JEPA), proposed by Yann LeCun in his 2022 position paper A Path Towards Autonomous Machine Intelligence, takes a third route that avoids both problems.

The idea in one sentence: predict the representation of the missing part from the representation of the visible part. Not the pixels, and not via augmented-view agreement. You encode the context, encode the target, and train a predictor to map the context embedding to the target's embedding.

Why this dissolves the generative tax: because the target is a learned representation rather than raw input, the model is allowed to discard unpredictable detail. Nothing forces the target embedding of a grass patch to encode each blade. If a detail is not predictable, the representation can simply not represent it, and the loss will not punish that. The system can focus its capacity on the structure that is predictable, which is precisely the semantic structure you want.

Why this dissolves the contrastive bias: the prediction task is created by masking, an operation on the data itself, not by hand-crafted transformations. So the model is not told "be invariant to color." It is told "figure out what about the hidden region is inferable from the visible region," and it learns which regularities matter from the data.

That is the bet: give the model a target it is allowed to simplify, and the simplification it chooses is semantics.

It is worth dwelling on the conceptual move, because it inverts an assumption people carry from generative modeling: that a model which cannot reproduce the input does not really "understand" it.

JEPA's premise is the reverse. Discarding information is what intelligence does. When you look out a window and predict what happens next, you do not render future pixels. You predict at an abstract level, that car will pass, that person will reach the door, while remaining completely uncommitted about the exact texture of the asphalt or the flutter of every leaf. Those details are unpredictable and irrelevant to any decision you would make.

A pixel-predicting model has no way to express "I don't know, and it doesn't matter." It must output something for every value, and it is graded on all of them. A representation-predicting model has exactly that freedom: the encoder decides what to encode, so unpredictable content can be dropped from the target rather than badly guessed.

This is why LeCun frames JEPA as the backbone for world models, systems that predict how the world evolves in order to plan. Prediction for planning must happen at a level of abstraction where prediction is actually possible. Rendering the future in full detail is both intractable and unnecessary. Predicting its representation is neither, a theme Lesson 3 develops when JEPA moves to video and robot control.

The argument would be philosophy if it did not show up in measurements. It does, most clearly in I-JEPA, the image instantiation from Assran and colleagues at CVPR 2023 (with LeCun among the authors), examined in detail in Lesson 3.

Two findings matter here.

Compute efficiency. Predicting in representation space is dramatically cheaper than predicting pixels. The I-JEPA authors report pre-training a vision transformer (ViT-H/14) on ImageNet in under 1200 GPU hours, over 2.5 times faster than a much smaller ViT-S/16 trained with iBOT, and over 10 times more efficient than a ViT-H/14 trained with MAE. The intuition is direct: no pixel decoder, and no capacity burned rendering irreducible texture.

Semantic quality without hand-crafted priors. I-JEPA learns strong semantic features without relying on pre-specified invariances from hand-crafted augmentations, exactly the bias that contrastive methods must accept, while still beating pixel-reconstruction methods on semantic evaluations.

That combination is the point. The two established families each pay a price: generative methods pay in wasted capacity on the unpredictable, contrastive methods pay in injected human bias about what to ignore. The joint-embedding predictive approach declines both bills, and the measured result is better semantics for less compute. The mechanism that makes it work, and the failure mode it must dodge, is Lesson 2.

Here is the landscape in one view.

Read the bottom row carefully, because it sets up the next lesson. Each family has a characteristic failure mode. Generative methods are safe but wasteful: the pixel target is fixed, so the objective can never be gamed. Contrastive methods need negatives precisely to stop the model from cheating.

JEPA's freedom, that the target representation is learned and can discard information, is also its danger. If the target is learned, what stops the encoder from learning to represent nothing at all? A constant target is trivially predictable and drives the loss to zero. That is representation collapse, and every joint-embedding method must defeat it. How JEPA does so, without negatives, is the mechanism at the heart of Lesson 2.

All three self-supervised families start from unlabeled data but differ in what they predict: generative methods reconstruct pixels and pay for unpredictable detail, contrastive methods match augmented views and inherit hand-crafted invariance biases, and JEPA predicts the target's learned representation, escaping both at the cost of having to prevent collapse.