From I-JEPA to V-JEPA: World Models and Planning

Lessons 1 and 2 gave the idea and the machinery. This lesson is the payoff: what happens when a JEPA is actually built, scaled, and used.

The line runs in three steps, and each adds a genuinely new capability:

• I-JEPA (images, CVPR 2023): does the architecture learn good representations at all? It answers yes, efficiently and without hand-crafted augmentations.
• V-JEPA (video): extend masking through time, so the model must predict how things move, not just what is spatially adjacent. Temporal prediction is where physical understanding can appear.
• V-JEPA 2 (arXiv:2506.09985): scale to over a million hours of video, then add action conditioning, turning a passive predictor into a world model that can be used to plan.

The arc matters conceptually. A representation learner tells you what is there. A world model tells you what happens next if I do this, and that is the ingredient planning requires. LeCun's argument from Lesson 1, that prediction for planning must happen in an abstract space where prediction is possible, is exactly what this progression tests. The endpoint is a robot performing tasks in an unfamiliar lab it was never trained in.

I-JEPA (Assran, Duval, Misra, Bojanowski, Vincent, Rabbat, LeCun, and Ballas) instantiates the architecture on images. Its headline design is not the encoder, it is the masking strategy, because in a JEPA the mask is the task. Choose a bad mask and you get a bad representation, no matter how good the network.

I-JEPA's recipe, called multi-block masking, has two deliberate properties:

• Predict several large target blocks (typically four) from a single context block, rather than scattered individual patches.
• Make the blocks sizeable and semantic in scale, and remove the target regions from the context so the model cannot peek.

The reasoning is precise. If targets were small, scattered patches, the model could solve the task with local texture continuation: the patch next to grass is more grass. That is exactly the low-level shortcut Lesson 1 warned about, and it teaches nothing semantic. By forcing prediction of a large block from a distant context block, local extrapolation becomes useless. To predict the representation of a big masked region, the model must infer what object is there and what it implies, which is to say it must build semantic understanding.

This is the general lesson of the family: in a JEPA, the difficulty and the shape of the prediction task are the entire curriculum. The mask decides whether the model learns texture or meaning.

I-JEPA's results, cited in Lesson 1, are worth restating as a verdict on the whole approach, because they closed three open questions at once.

• The representations are semantic. I-JEPA learns strongly semantic features without any hand-crafted augmentations, no crops-and-jitter recipe declaring what to ignore. This was the contrastive family's structural tax (Lesson 1), and I-JEPA showed you can decline it and still get semantics.
• It is far cheaper. Pre-training a ViT-H/14 on ImageNet took under 1200 GPU hours: over 2.5 times faster than a smaller ViT-S/16 trained with iBOT, and over 10 times more efficient than a ViT-H/14 trained with MAE. Deleting the pixel decoder and the obligation to render texture is worth an order of magnitude.
• Collapse is survivable in practice. The anti-collapse machinery of Lesson 2, stop-gradient, predictor, EMA target, holds up at scale on a real workload, which is what turns a theoretical hazard into an engineering parameter.

Taken together, these establish the core claim empirically: predicting representations is not a compromise, it is better on both axes at once, more semantic and cheaper than predicting pixels, while avoiding the augmentation bias contrastive methods require. With the principle validated on static images, the interesting question becomes what happens when you add time.

Extending to video changes the problem qualitatively, not just quantitatively. V-JEPA masks regions across space and time, so the model must predict the representation of a region that is missing over a span of frames.

Why this is a bigger deal than it sounds: to predict a masked region in the future, spatial context alone is insufficient. You must know how things move, persist, and interact. Predicting that a rolling ball's representation appears further right, that an occluded object still exists behind the occluder, that a falling cup continues downward, requires internalizing regularities that are, in effect, intuitive physics. Video contains that structure for free, no labels needed.

This is also where the Lesson 1 argument becomes undeniable rather than merely elegant. Pixel-predicting the future is close to hopeless: the future is ambiguous, and a pixel loss forced to commit produces blur, the average of every possible future. Predicting representations of the future lets the model say the physically meaningful part (the cup falls and lands here) while remaining silent about the unknowable part (the exact splash pattern, the precise glint). Abstraction is not a nicety in video; it is the only level at which prediction is well-posed.

V-JEPA 2 pushed the scale accordingly: self-supervised training on over a million hours of internet video, yielding state-of-the-art visual understanding and prediction. But understanding is still passive. Planning needs one more ingredient.

A model trained on passive video learns what tends to happen. An agent needs to know what happens if I do this. That gap is closed by action conditioning, and it is the step that makes V-JEPA 2 a world model rather than a video predictor.

The second stage of V-JEPA 2 trains an action-conditioned world model: a 300-million-parameter transformer using block-causal attention that autoregressively predicts the representation of the next video frame, conditioned on an action and the previous states. In symbols, it learns a function of the form:

z_{t+1}  =  world_model( z_t , a_t )

  z_t : representation of the current state (from the V-JEPA encoder)
  a_t : the action taken (e.g. robot end-effector command)
  z_{t+1} : predicted representation of the resulting next state

Read that line carefully, because it is the entire lesson in one equation: given where I am and what I do, predict where I end up, in representation space. Everything from Lessons 1 and 2 is doing work here. The state is an embedding, not pixels, so the model predicts consequences at the level where they are predictable, and the encoder that produces z was trained by the collapse-defended machinery of Lesson 2.

Strikingly, the adaptation used only about 62 hours of unlabeled robot video. The physics came from the million hours of internet video; the small robot dataset only had to teach how actions connect to that already-learned dynamics.

With a world model in hand, control works differently from mainstream robot learning, and the difference is the point.

The common approach learns a policy: a function from observation to action, trained on demonstrations or reward for a task. It is fast but brittle, it knows this task in these conditions, and a new object or a new lab is out of distribution.

V-JEPA 2 instead does not learn a fixed policy. It plans with model-predictive control (MPC): at each step, imagine candidate action sequences by rolling them forward through the world model in representation space, score how close each predicted end state is to the goal (specified as a visual subgoal, an image of what success looks like), execute the first action of the best sequence, observe, and re-plan. The imagination happens entirely in embedding space, so it is cheap: no frames are rendered, only vectors predicted.

The generalization payoff is the headline result: zero-shot robot control in new labs, with reported success rates of roughly 65 to 80 percent for pick-and-place of new objects in new, unseen environments. Nothing was trained for that object or that room. The model knows physics from video and knows how actions move the world; the planner composes them on the spot.

This is the concrete cash-out of LeCun's thesis from Lesson 1: learn how the world works once, at the right level of abstraction, then plan any task inside it, rather than training a separate policy per task.

The progression, and what each step contributed:

And the honest limits, which matter as much as the results:

• Goals must be specified. The reported planning uses visual subgoals, an image of the desired state. Turning an abstract instruction into such goals is not solved by the world model itself.
• The horizon is short. Autoregressive rollouts in representation space accumulate error, so plans degrade over long horizons. This is precisely the gap H-JEPA (Lesson 2) proposes to close with hierarchy, coarser levels predicting further, and it is a proposal, not a finished result.
• The benchmarks are young. V-JEPA 2 shipped alongside new benchmarks for physical reasoning from video, an admission that the field lacked good ways to measure this. Evaluation is still maturing.
• The evidence is narrow. Impressive zero-shot pick-and-place is not general manipulation, and the success rates, while striking, are not near-perfect.

So the fair verdict is a validated mechanism with a real but early demonstration. The claim proven is specific: predicting representations rather than pixels yields semantic, efficient, and controllable models, and it works well enough to plan a real robot's actions in a room it has never seen.

The V-JEPA encoder turns observations into state representations; the action-conditioned world model predicts the next representation given an action, so a planner can imagine candidate action sequences in embedding space, pick the one whose predicted state best matches a visual subgoal, execute one action, and re-plan.