Beyond Negatives: Non-Contrastive and Masked Methods

The previous lesson made contrastive learning work but at a cost: many negatives, large batches or queues, and the false-negative problem. That raised a provocative question. Collapse happens when you only pull views together; contrastive methods stop it by also pushing negatives apart. But is pushing apart the only way to prevent collapse?

The surprising answer is no. A family of non-contrastive methods keeps just the pull-together half, no negatives at all, and still avoids the constant solution. They do it in two broad ways: distillation methods use architectural asymmetry to break the collapse dynamic, and regularization methods constrain the statistics of the embeddings directly. A third, separate family, masked modeling, sidesteps the whole augmentation game. This lesson covers all three, then connects them to the energy-based picture that unifies the field.

BYOL (Bootstrap Your Own Latent, Grill and colleagues, 2020) startled the field by matching contrastive performance with no negatives. It uses two networks: an online network and a target network. The online network sees one view and tries to predict the target network's embedding of the other view. Only the online network is trained by gradients; the target is updated as a slow moving average of the online network, and a small predictor head sits on the online side, making the two paths asymmetric.

The combination of that asymmetry, the momentum target, and a stop-gradient (no gradient flows back through the target) is what prevents collapse without any repulsion. Exactly why this works was debated at first, but empirically the asymmetric, self-distilling setup simply does not fall into the constant solution. BYOL proved that negatives are one tool for avoiding collapse, not the only one.

SimSiam (Chen and He, 2021) ran the reductive experiment: what is the minimal ingredient that stops collapse? It stripped BYOL down, removing the momentum target and the negatives, keeping two identical encoders (a Siamese network), a predictor on one side, and crucially a stop-gradient on the other.

The striking result: the stop-gradient alone prevents collapse. With it, SimSiam learns strong representations; remove it, and the model collapses immediately. This isolated the essential mechanism that BYOL had bundled with other tricks. The intuition is that the stop-gradient makes one branch a fixed target the other chases, turning the objective into an alternating optimization that has no incentive to shrink everything to a constant. SimSiam matters less as a state-of-the-art method than as a clarifying result: it showed the whole non-contrastive family hinges on breaking symmetry between the two branches, not on any repulsive force.

A different non-contrastive strategy avoids collapse not through architecture but by regularizing the embedding statistics directly. Two methods define it.

Barlow Twins (Zbontar and colleagues, 2021) computes the cross-correlation matrix between the two views' embedding dimensions and pushes it toward the identity: each feature should correlate with its own counterpart (invariance) and be decorrelated from all other features (reducing redundancy). A collapsed representation cannot satisfy this.

VICReg (Variance-Invariance-Covariance Regularization, Bardes, Ponce, and LeCun, 2022) makes the anti-collapse condition fully explicit with three terms: variance (each embedding dimension must maintain variance across the batch, directly forbidding a constant), invariance (matching views stay close), and covariance (decorrelate dimensions to spread information). VICReg is especially clarifying because its variance term is a direct, literal answer to collapse: if every dimension must vary, the constant solution is simply illegal. These methods need neither negatives nor asymmetry, only well-chosen constraints on the embeddings.

A third family returns to prediction, but in a form that scales beautifully. Masked modeling hides part of the input and trains the model to fill it in. Language led the way: BERT masks random words and predicts them from context, and that objective produced the foundational language encoders.

In vision, the Masked Autoencoder (MAE) (He and colleagues, 2022) splits an image into patches, hides a large fraction of them (often around 75%), and trains an encoder-decoder to reconstruct the missing patches from the few visible ones. Because so much is hidden, the model must learn real structure rather than local texture copying. Masked methods are not joint-embedding and do not face collapse in the same way; their challenge is that they reconstruct in input space (pixels or tokens), spending capacity on detail. Their advantages are simplicity, no need for hand-designed augmentations or negatives, and excellent scaling, which is why masked pretraining dominates in both language and, increasingly, vision.

Four routes to a useful representation. Contrastive methods push negatives apart. Distillation methods (BYOL, SimSiam) use asymmetry and a stop-gradient. Regularization methods (Barlow Twins, VICReg) constrain embedding statistics so variance is preserved and features are decorrelated. Masked methods reconstruct hidden input and dodge joint-embedding collapse entirely. Different mechanisms, one shared purpose.

These methods are not just a list; they are variations on one theme, and two ideas unify them.

First, JEPA (the Joint-Embedding Predictive Architecture, covered in its own cursus) sits naturally in this family. It predicts, like masked modeling, but predicts in embedding space rather than pixel space, combining the strengths of joint-embedding and prediction, and it uses non-contrastive, regularization-style training to avoid collapse. It is the architecture LeCun advocates as the successor to both contrastive and pixel-reconstruction methods.

Second, and deeper: every method here can be read as shaping an energy landscape, giving low energy to matching views and preventing that landscape from flattening to a constant. Contrastive methods push energy up on negatives; regularization methods constrain the landscape's shape instead. That is precisely the energy-based-model view of self-supervised learning, and it is the subject of the companion cursus on energy-based models. SSL is where the abstract idea of an energy landscape becomes concrete engineering.

In practice, the families trade off rather than dominate. Contrastive methods are conceptually simple but need negatives and careful augmentation. Distillation methods (BYOL, SimSiam) remove negatives at the cost of asymmetry tricks. Regularization methods (VICReg, Barlow Twins) make the anti-collapse condition explicit and are stable and interpretable. Masked methods scale best and need no augmentation, but reconstruct in input space. The right choice depends on the modality, the compute budget, and how much augmentation design you can afford.

The durable takeaway of this cursus is a single idea: modern self-supervised learning is the art of learning representations from unlabeled data by making related things agree, while using negatives, asymmetry, regularization, or masking to keep the representation from collapsing. That one problem, and its family of solutions, is the engine behind today's foundation models, and it is why self-supervised learning sits at the center of where machine learning is heading.