Learning Without Labels: Pretext Tasks

Supervised deep learning is spectacularly effective and spectacularly hungry: it needs large datasets where every example carries a human-provided label. Labels are the expensive part. Annotating millions of images, transcribing audio, or tagging medical scans costs money, time, and expertise, and for many domains the labels simply do not exist at scale.

Meanwhile, raw unlabeled data is nearly free and effectively unlimited: the web is full of images, text, and video with no annotations attached. The obvious question is whether a model can learn useful structure from that raw data alone, without waiting for labels. Yann LeCun has called this the central challenge, describing self-supervised learning as the "dark matter of intelligence": the vast, unlabeled bulk from which understanding should mostly be built. This lesson is about how machines learn from data that comes with no answer key.

Self-supervised learning (SSL) solves the label problem with a trick: manufacture the labels from the data itself. You hide part of the data and train the model to predict it from the rest. The supervision is real, but it costs nothing, because the answer was in the data all along.

The task you invent for this purpose is called a pretext task. Its labels are free because they are derived automatically: mask a word and predict it, remove color and predict it, cut out a patch and predict what was there. Crucially, the pretext task is not the goal. Nobody actually needs a model that predicts missing patches. The goal is the representation the model builds along the way, the internal features it must develop to solve the pretext task well. Those features then transfer to the real tasks you care about. This inversion, train on a throwaway task to get reusable features, is the heart of SSL.

The first wave of SSL in vision invented clever pretext tasks by hand. A sampler:

• Rotation prediction: rotate an image by 0, 90, 180, or 270 degrees and predict the angle. To succeed, the model must recognize objects and their canonical orientation.
• Colorization: strip an image to grayscale and predict the colors. This forces understanding of what things are (grass is green, skies are blue).
• Jigsaw: shuffle image patches and predict their correct arrangement, requiring a grasp of object layout.
• Inpainting: mask a region and reconstruct it from context.

Language had the most powerful pretext task of all: predict the next word, or a masked word, from surrounding text. That single self-supervised objective, trained at scale, is what produced modern language models. The pattern is general: a well-chosen prediction problem, with labels taken for free from the data, drives a model to learn genuinely useful structure.

The payoff of SSL is what happens after pretraining. You take the model that solved the pretext task, discard the pretext-specific output layer, and keep the encoder: the part that maps an input to a feature vector. That encoder now carries general knowledge of the domain, learned without labels.

Two standard ways to use it on a real, labeled task:

• Linear probing: freeze the encoder and train only a simple linear classifier on top, using a small labeled dataset. If the frozen features already separate the classes well, the representation is good.
• Fine-tuning: continue training the whole encoder on the labeled task, starting from the self-supervised weights rather than from scratch.

Either way, the expensive labeled dataset can now be small, because the encoder learned most of what it needs from unlimited unlabeled data. This is why SSL underpins modern foundation models: pretrain once, self-supervised, on a huge unlabeled corpus, then adapt cheaply to many downstream tasks.

Reconstructing pixels has a drawback: it forces the model to waste capacity on irrelevant detail. To inpaint a patch of grass perfectly, a model must predict the exact position of every blade, information no downstream task cares about. A different strategy sidesteps this.

Instead of predicting raw data, learn an embedding that is invariant to augmentation. Take an image, produce two altered views of it (crop, recolor, blur), pass both through the encoder, and train so their embeddings are close. The logic: two views of the same scene share meaning, so their representations should agree, while the exact pixels can differ. This is the joint-embedding approach, and it moved the field away from pixel reconstruction toward comparing representations directly. It is also where a serious danger appears, one that shapes every method in the next lesson.

Here is the danger. If your entire training signal is "make the two views' embeddings close," the model discovers a devastating shortcut: map every input to the same constant vector. Then all embeddings are identical, so any two views are trivially close, the loss is zero, and the model has learned absolutely nothing. This failure is called representational collapse, or just collapse.

Collapse is not a rare edge case; it is the default outcome of a naive joint-embedding objective. The whole difficulty of modern SSL is designing the training so that the useful solution (embeddings that capture real structure) is reachable while the trivial constant solution is blocked. Every method in the next lesson, contrastive and non-contrastive alike, is fundamentally a different answer to one question: how do you make views of the same thing agree, without letting the model cheat by making everything agree?

The core loop of modern SSL: from one unlabeled image, generate two augmented views, encode both into embeddings, and train so matching views agree. A collapse-prevention mechanism sits on that objective to forbid the trivial constant solution. Afterward, the frozen or fine-tuned encoder feeds a small labeled downstream task.

You now have the frame for the whole field. Self-supervised learning invents a pretext task to get free labels; the modern version learns augmentation-invariant embeddings; and the central obstacle is collapse.

The next two lessons are the two great families of solutions. Contrastive methods prevent collapse by also pushing apart the embeddings of different images: if everything must be far from everything else, nothing can collapse to a constant. Non-contrastive methods achieve the same end without negatives, using architectural tricks or explicit regularization, and a separate masked-modeling family returns to prediction but does it cleverly. As a preview of a deeper connection: this collapse-avoidance question turns out to be exactly the training problem of energy-based models, LeCun's other central theme, which is why these ideas sit so close together in his research.