Training Energy-Based Models

The previous lesson left us with an energy landscape to sculpt. Training an energy-based model has a strikingly simple goal when stated geometrically: push the energy down at the real data, and push it up everywhere else. Deepen the valleys where training examples live; raise the ground elsewhere.

The first half is easy: given real data, lower its energy, straightforward gradient descent on $E(x)$ for observed $x$. The hard, essential half is the second: pushing energy up on all the configurations that are not data. You cannot do this everywhere; there are far too many configurations. If you only ever push data energy down and never push anything up, the model discovers the same cheat as self-supervised collapse: make the energy low everywhere, a flat landscape that trivially assigns low energy to the data too, and has learned nothing. All of EBM training is different answers to one question: how do you raise the energy of non-data without visiting all of it?

The textbook way to fit a probability model is maximum likelihood: adjust parameters to raise the probability of the training data. For an EBM, $p(x) = e^{-E(x)}/Z$, and taking the gradient of the log-likelihood splits into two terms: one that lowers energy on the data and one that involves the partition function $Z$, the normalizer summing $e^{-E}$ over all configurations.

That second term is exactly the push-up force, and it requires $Z$ (or samples from the model's own distribution) to evaluate. As the last lesson stressed, $Z$ is intractable. So you cannot simply run maximum likelihood on an EBM; the gradient you need contains a quantity you cannot compute. Every training method in this lesson is a way to approximate or sidestep that push-up term. They fall into two broad philosophies, and the split mirrors self-supervised learning exactly.

The most direct approach approximates the push-up term with samples: rather than raise energy everywhere, generate a few plausible non-data configurations ("negative samples" or "hallucinations") and push their energy up specifically.

Contrastive divergence (associated with Geoffrey Hinton) does this using a short run of Markov Chain Monte Carlo (MCMC). Starting from a data point, it takes a few stochastic steps guided by the current energy to produce a sample from roughly where the model thinks data should be, then pushes that sample's energy up while pushing the real data's energy down.

# Contrastive divergence, one step (schematic)
x_data = sample_real_data()
x_model = mcmc_steps(start=x_data, energy=E, k=few)   # a "hallucination"
loss = E(x_data) - E(x_model)    # down on data, up on model sample

The short MCMC run is a biased but cheap stand-in for true sampling. Contrastive divergence trained the classic energy models (Boltzmann machines) and remains the archetype of the sample-based push-up.

Running MCMC is expensive and finicky. Noise-contrastive estimation (NCE), introduced by Michael Gutmann and Aapo Hyvarinen, avoids it with a clever reframing: instead of sampling from the model, contrast the real data against a fixed, known noise distribution.

NCE trains a classifier to distinguish real data from noise samples. To tell them apart, the model must assign low energy to real data and high energy to noise, and the mathematics shows that succeeding at this classification recovers the correct energy function up to a constant, without ever computing $Z$. The appeal is that generating noise is trivial and the training becomes an ordinary supervised classification problem. The limitation is that the noise distribution must be reasonably close to the data for the task to be informative; if telling data from noise is too easy, the model learns little. NCE is the conceptual bridge to contrastive self-supervised learning: InfoNCE, from the SSL cursus, is precisely this idea applied to representation learning.

A third approach sidesteps the partition function with a mathematical trick. Score matching, introduced by Aapo Hyvarinen, notices that the troublesome $Z$ is a constant, and constants vanish when you take a gradient with respect to $x$.

The score is that gradient of the log-density, which equals the negative gradient of the energy, and it does not depend on $Z$ at all. Score matching trains the model so its score matches the data's score, learning the shape of the energy landscape (which way is downhill everywhere) without ever normalizing it. In practice this is often done by teaching the model to denoise: add noise to data and predict the clean version, which turns out to estimate the score. This connection is important, because it links EBMs directly to modern diffusion and score-based generative models, which are, under the hood, learning an energy landscape's gradient by denoising. Score matching is the reason the energy-based view quietly underlies some of today's most powerful generators.

Every method so far is contrastive: it explicitly pushes energy up somewhere, on MCMC samples, on noise, or implicitly via the score. LeCun highlights a fundamentally different philosophy: regularized methods that never push energy up sample-by-sample, and instead limit the volume of low-energy space so the landscape cannot flatten.

The intuition: collapse means low energy everywhere. If you add a constraint that only a small region is allowed to have low energy, then driving the data's energy down automatically forces everything else up, because the low-energy budget is finite. You get valleys at the data without ever sampling a single negative. This is exactly what the regularization-based self-supervised methods do: VICReg's variance term and Barlow Twins' decorrelation, from the SSL cursus, are ways of capping the low-energy volume so the representation cannot collapse. Regularized EBMs trade the cost and instability of sampling for the challenge of designing a good constraint.

Every EBM training method serves the push-down-push-up goal, but splits into two philosophies. Contrastive methods explicitly raise energy on selected non-data points: MCMC samples (contrastive divergence), noise samples (NCE), or the score via denoising (score matching). Regularized methods instead constrain the landscape so its low-energy region has limited volume, forcing non-data energy up implicitly. Both prevent the flat-landscape collapse.

Step back and the structure is clean. Training an EBM means sculpting valleys at the data while keeping the landscape from going flat. Because the partition function blocks direct maximum likelihood, you approximate the push-up force one of two ways:

This is the same contrastive-versus-regularized split that organizes self-supervised learning, and that is no coincidence: SSL is energy-based learning applied to representations. Contrastive SSL pushes negatives' energy up; non-contrastive SSL regularizes the landscape. The next lesson makes this unification explicit, showing how classification, generation, self-supervised learning, and JEPA all become special cases of one energy-based idea, which is precisely why LeCun treats the energy framework as the common language for machine learning.