86. From Images to Video Worlds: V-JEPA 2 and Planning

The lesson

If predicting representations of images works, the next question is time: can the same trick learn how the world CHANGES? "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning" by Adrien Bardes, Quentin Garrido, Christian Russell, Jean Ponce, Yann LeCun, Nicolas Ballas, Mahmoud Assran and colleagues (arXiv:2506.09985, 2025, in the TPEE library) answers yes - and turns a representation-learning paper into a robotics result.

Stage one: pre-train an action-free JEPA on a video-and-image dataset of more than 1 MILLION hours of internet video. No labels, no annotations - the model predicts representations of future frames from past ones. The result is a general-purpose motion model: 77.3 top-1 accuracy on Something-Something v2 (motion understanding) and state-of-the-art human-action anticipation with 39.7 recall-at-5 on Epic-Kitchens-100.

Stage two: align the video model with a large language model. At only 8 billion parameters the combined system reaches state-of-the-art video question answering at its scale - 84.0 on PerceptionTest and 76.9 on TempCompass. Language gives the vision model a question-answering interface without retraining the vision backbone.

Stage three is the headline: V-JEPA 2-AC, an action-conditioned variant post-trained on less than 62 HOURS of unlabeled robot videos from the Droid dataset. Deployed on real Franka robot arms in two different laboratories, it picks up and places objects zero-shot - planned with image goals, with no data collected in those environments, no task-specific training and no reward function. A world model learned mostly by watching.

The conceptual lineage is visible in the TPEE library. Ha & Schmidhuber's "World Models" (NeurIPS 2018) let an agent train inside its own learned dream; Dreamer (Hafner et al., 2020, also in the library) imagined behaviors in latent space. V-JEPA 2 is the same idea at foundation-model scale - and MC-JEPA (2023) is the bridge from still images to motion, learning flow and content jointly. The whole lineage, from 1517 (Llull) to 2025, is browsable on tpee.app/papers.

For prompt engineering, video world models sharpen the target-selection lesson of lesson 85. When you ask a model to anticipate - "what breaks next?", "what will the user do after this error?" - you are asking for prediction inside a mental model of the world, not for a token stream. Good anticipation prompts name the state variables that matter (which objects, which constraints, which deadlines) instead of requesting a full video of the future.

Planning with image goals also reframes specification. V-JEPA 2-AC was given a picture of the desired outcome and planned the rest - the robot equivalent of TPEE's Examples section. When you can SHOW the target state instead of describing every step, prompts get shorter and more robust: specify the goal, let the world model fill in the dynamics. And as always: TPEE never connects to these systems; any V-JEPA model runs in its own external application, and the papers are one click away on tpee.app/papers.

Check yourself