Energy Landscapes: The EBM View

Most neural networks are functions: feed an input in one end, get an output the other. An energy-based model (EBM) reframes the whole setup. Instead of a function from input to output, it learns a single scalar energy that scores how well things fit together: low energy for configurations that are correct or compatible, high energy for those that are wrong.

This sounds abstract, but it is a powerful shift, and it is the framing Yann LeCun has advocated for decades, laid out in his 2006 "A Tutorial on Energy-Based Learning." Rather than committing the model to produce one answer, you give it a way to judge any proposed answer. Asking for the best answer then becomes a search for the lowest-energy one. This lesson builds that picture: what energy is, how you use it to make predictions, and why measuring compatibility, rather than computing an output directly, buys real flexibility.

Formally, an EBM defines an energy function that takes a configuration and returns a single number. In the simplest case it scores one variable, written $E(x)$: how plausible is this data point $x$? More often it scores a pair, $E(x, y)$: how compatible is output $y$ with input $x$?

The convention is that lower energy means better. A real photograph gets low $E(x)$; random noise gets high $E(x)$. A correct caption $y$ for an image $x$ gets low $E(x, y)$; a nonsense caption gets high energy. The energy function is typically a neural network, and it can be arbitrarily expressive. Note what is different from a classifier: the energy function does not output a class or a value. It outputs a score for a configuration you hand it. Getting an actual answer out of the model is a separate step, and that step is inference.

If the model only scores configurations, how do you get an answer? By searching for the configuration with the lowest energy. This is the defining move of EBMs: inference is optimization.

Suppose you have a trained $E(x, y)$ and a specific input $x$. To predict, you fix $x$ and search over possible outputs $y$ for the one that minimizes the energy: the most compatible answer. For a small discrete set of $y$ (say, class labels) you can just evaluate them all and pick the minimum. For a large or continuous $y$ you run an optimization, for example gradient descent on $y$ while holding $x$ fixed.

The power of this is flexibility. The same trained energy function answers many kinds of question depending on what you hold fixed and what you search over: fill in a missing part, rank candidates, check plausibility, or generate. One model, many query types, because inference is just "minimize energy over the unknowns."

The most useful mental image is a landscape. Picture the space of all possible configurations spread out on a plain, with the energy as height. Correct, plausible data sits in the valleys (low energy); implausible configurations rise up the hills (high energy).

Training an EBM is sculpting this landscape: carving deep valleys exactly where the real data lives and raising everything else. Inference is then rolling downhill: from wherever you start, descend to the nearest valley, the most compatible answer. This picture makes the key ideas intuitive. A well-trained model has valleys precisely at the data and high ground everywhere else. A broken model has a flat landscape, no valleys, so every answer looks equally good and the model is useless. Keeping the landscape from flattening out is, as the next lesson shows, the entire difficulty of training, and it is the same collapse problem seen in self-supervised learning.

Two operations on the same surface. Training pushes the energy down at real data points and up elsewhere, carving valleys at the data. Inference starts from an initial guess and descends the surface to the nearest valley, returning the lowest-energy, most compatible configuration. A flat landscape (no valleys) is the failure mode both later lessons work to prevent.

Energies are unnormalized scores, but you can turn them into a probability distribution. The standard recipe (the Boltzmann or Gibbs form) is:

$p(x) = \frac{e^{-E(x)}}{Z}, \qquad Z = \sum_{x'} e^{-E(x')}$

Low energy gives high probability, and the exponential makes everything positive. The catch is $Z$, the partition function: the sum (or integral) of $e^{-E}$ over every possible configuration, needed to make the probabilities add to one. For any interesting model, that sum is astronomically large or infinite, and therefore intractable to compute.

This single quantity, $Z$, is the source of nearly all the difficulty in energy-based modeling, and the reason the next lesson exists. But notice a liberating point: many uses of an EBM, ranking candidates, finding the most compatible answer, only need to compare energies, and for comparisons the constant $Z$ cancels out. You often need the landscape's shape, not its absolute normalization.

A concrete payoff of the energy view is how naturally it handles problems with more than one correct answer. A standard regression network outputs a single value, so when several answers are valid it averages them, often landing on something wrong (the multimodality problem seen in the robot-learning cursus, where averaging "left" and "right" drives straight into an obstacle).

An EBM has no such trouble. Multiple valid answers are simply multiple valleys in the energy landscape, each a distinct low-energy region. Inference descends into one of them, returning a coherent answer rather than an incoherent average of all of them. This is why LeCun stresses energy-based models for perception and prediction: the real world is full of situations with many plausible continuations, and a model that scores compatibility can represent all of them at once, where a model forced to emit a single output cannot. The remaining question is purely practical: how do you train the energy function so those valleys land in the right places?

You now have the core of the energy-based view. An EBM learns a scalar energy that scores compatibility; inference means searching for the lowest-energy configuration; the landscape picture makes training (sculpt valleys) and inference (roll downhill) intuitive; energies convert to probabilities through the intractable partition function $Z$; and multiple valid answers are just multiple valleys.

The next lesson tackles the hard part head-on: how to train an EBM when $Z$ cannot be computed. The answer splits into two strategies, pushing energy up on wrong configurations (contrastive) versus constraining the landscape so it cannot flatten (regularized), and those two strategies are exactly the contrastive and non-contrastive families from the self-supervised-learning cursus. The final lesson then shows how this one framework unifies classification, generation, self-supervised learning, and JEPA under a single idea, which is why energy-based models sit at the center of LeCun's program.