EBMs as a Unifying Lens

The first two lessons built the energy-based model as a specific technique: a scalar energy, inference by minimization, training by shaping a landscape. This lesson makes the larger claim that motivates the whole framework for LeCun. The energy-based view is not just one more method; it is a lens through which a large fraction of machine learning turns out to be the same thing.

The recurring pattern is this: define an energy that scores compatibility, arrange for low energy at the correct configurations, and read prediction, generation, or representation-learning off the resulting landscape. Once you see it, classification, generative modeling, self-supervised learning, JEPA, and diffusion all reveal themselves as different faces of energy-based learning. This lesson walks that unification, then closes with an honest account of what the lens does and does not buy you in practice.

Start with the familiar. A standard classifier outputs a score per class, usually turned into probabilities by a softmax. Read this through the energy lens: the negative of each class score is an energy $E(x, y)$ for pairing input $x$ with class $y$, and the softmax is exactly the Boltzmann formula $e^{-E}/Z$ from lesson 1, where the sum over the handful of classes makes $Z$ trivial to compute. Classification is an energy-based model where the partition function happens to be cheap.

Regression fits too: predicting a value by minimizing squared error is the same as placing a simple parabolic energy valley around the target and descending to its bottom. These everyday methods are not usually described this way because their $Z$ is easy and their landscape is simple. The energy view does not change how you train them; it reveals that they are the tractable corner of a much larger space, the corner where normalization is free.

The tightest connection, and the reason these two cursus are companions, is that self-supervised learning is energy-based learning applied to representations. The mapping is exact.

Define the energy of a pair of inputs as the distance between their embeddings: matching views should have low energy (close embeddings), non-matching should have high energy. Training the representation is then sculpting this energy landscape, and the collapse problem, everything mapping to one point, is precisely the flat-landscape failure from lesson 2. Now the two SSL families line up perfectly with the two EBM training strategies. Contrastive SSL (SimCLR, MoCo) pushes up the energy of negative pairs, exactly the contrastive push-up. Non-contrastive SSL (VICReg, Barlow Twins) constrains the embedding statistics to cap the low-energy volume, exactly the regularized strategy. The two literatures are not analogous; they are the same mathematics viewed from representation-learning and energy-modeling angles respectively.

This unification is exactly why LeCun frames his flagship architecture, JEPA (the Joint-Embedding Predictive Architecture, covered in its own cursus), in energy terms. A JEPA takes two related inputs, encodes both, and predicts one embedding from the other. Read as an EBM, its energy is the prediction error in embedding space: configurations the model can predict well have low energy; surprising ones have high energy.

Because JEPA is a joint-embedding method, it faces the collapse problem, and because it is LeCun's design, it uses the regularized (non-contrastive) strategy to avoid it, rather than contrastive negatives. JEPA is thus a concrete instance of everything in this cursus: an energy defined by predictability, inference as finding low-energy (predictable) configurations, and regularized training to keep the landscape from flattening. It also predicts in representation space rather than pixels, which sidesteps the wasted-capacity problem of pixel reconstruction, the reason LeCun argues it is a better route to world models than generative pixel prediction.

Generative modeling is energy-based at its core: a generator should assign high probability, hence low energy, to realistic samples and low probability to unrealistic ones. Producing a sample is finding or sampling from a low-energy region of the landscape.

Diffusion models, today's leading image and video generators, are the vivid case. As lesson 2 noted, training a diffusion model by denoising is a form of score matching: it learns the gradient of the log-density, which is the negative gradient of an energy. Generation then works by starting from noise and repeatedly stepping in the direction that lowers energy, descending the learned landscape toward realistic data. So the most powerful generative models in use are, mechanically, learning and then descending an energy landscape, even when they are never described with the word energy. The lens reveals a shared skeleton beneath methods that look unrelated on the surface.

One idea, many instances. A classifier is an EBM with a cheap partition function. Contrastive self-supervised learning is contrastive EBM training on embeddings; non-contrastive SSL is regularized EBM training. JEPA is an EBM whose energy is embedding-space prediction error. Diffusion models learn and descend an energy landscape via score matching. The center is always: low energy at compatible configurations.

Why insist on this lens rather than treating each method on its own terms? A few reasons follow directly from what we have built.

First, generality: one language covers prediction, generation, and representation learning, so an advance in one (say, a better anti-collapse regularizer) transfers to the others. Second, uncertainty and multiplicity: an energy landscape can have many valleys, so it represents situations with several valid answers, which a single-output function cannot, the point that recurred from robot policies to perception. Third, normalization-free reasoning: because inference only compares energies, you often avoid the intractable partition function entirely and just ask which configuration is most compatible. For LeCun, these properties make energy the right substrate for systems that perceive, predict, and plan: score the compatibility of possible world states and pick low-energy ones. It is a conceptual commitment about how to build machines that model the world, and it is the thread connecting his work on self-supervised learning, JEPA, and world models.

The lens is powerful, but intellectual unification is not the same as practical dominance, and it is worth being clear-eyed.

Explicit, general energy-based models remain hard to train: sampling with MCMC is slow and unstable, and this is a real reason they are less common in production than their conceptual reach suggests. In practice, the winning systems are usually the ones where the energy structure is implicit and the hard parts are dodged: classifiers with a cheap $Z$, contrastive and regularized self-supervised methods, and diffusion models trained by denoising. These succeed precisely because they avoid computing the partition function head-on.

So the honest summary is twofold. As a unifying theory, the energy-based view is genuinely illuminating: it shows that classification, SSL, JEPA, and diffusion share one skeleton, and it guides design, especially around avoiding collapse. As a direct recipe, training a big explicit EBM is often harder than using one of its tractable special cases. Both statements are true at once, and holding them together is the mature way to understand why this framework is central to LeCun's thinking yet not the label on most deployed systems.