TPEE Toptronic® Prompt Engineering Editor

85. JEPA: Predicting in Representation Space

Understand the Joint-Embedding Predictive Architecture (JEPA): why predicting representations instead of pixels or tokens changes what a model learns, and what that means for how you brief a model.

By Jacques Botte, founder of Toptronic®. Last updated 19 September 2026.

The lesson

For a decade, most AI systems learned by GENERATING: predicting the next pixel, the next sound sample, the next token. In 2023 Meta's I-JEPA paper - "Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture" by Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun and Nicolas Ballas (CVPR 2023, arXiv:2301.08243) - demonstrated a different route. JEPA stands for Joint-Embedding Predictive Architecture, and its key move is to predict in REPRESENTATION space, not output space.

The setup is simple to state. Take one image, sample a large context block, and predict the representations (embeddings) of several target blocks taken from elsewhere in the same image. Crucially the prediction happens in the latent space of an encoder, not on raw pixels - the model is never asked to reproduce texture, lighting or other unpredictable detail. It only has to capture what is SEMANTICALLY predictable.

This is why I-JEPA needs no hand-crafted data augmentations. Generative pixel-prediction methods needed crops, color jitter and other tricks to stop the model from cheating with low-level shortcuts. Lifting the target out of pixel space removes the cheat, and the masking strategy does the rest: sample target blocks large enough to be semantic, and use a spatially distributed context block that is informative enough.

The results made the case for scale: a ViT-Huge/14 trained on ImageNet with 16 A100 GPUs in under 72 hours produced representations strong across linear classification, object counting and depth prediction - with far less task-specific engineering. The idea was not new in spirit: LeCun's "A Tutorial on Energy-Based Learning" (2006, in the TPEE library) laid the energy-based foundation, and Barlow Twins (2021) and VICReg (2022) - both in the library - explored non-contrastive joint-embedding objectives.

MC-JEPA (Bardes, Ponce, LeCun and colleagues, 2023, arXiv:2301.08250, in the library) pushed the same architecture further: one network learning motion (optical flow), content and appearance features jointly. The family kept growing inside the TPEE library - Audio-JEPA for sound, A-JEPA for speech, JEPA-x for physical dynamics, JEDI distilling JEPA models to the edge - all browsable on tpee.app/papers.

What does a visual-representation paper have to do with prompts? The design principle transfers directly. A model briefed to "output everything you know" wastes its capacity on unpredictable, low-value detail - the pixel noise of prompt engineering. A model briefed with sharp targets (which exact aspects of the answer matter: TPEE's Format, Audience and Tone sections) and an informative context block (TPEE's Context and Data sections) behaves like a JEPA: it concentrates capacity on the predictable, meaningful structure of the task.

The offline boundary, always: TPEE never trains, downloads or calls any of these models. The papers live in the TPEE library of scientific papers - browse the full date timeline with links to every paper cited here on tpee.app/papers - and the lessons in this course are static, offline text. Quick reference: JEPA = Joint-Embedding Predictive Architecture; I-JEPA = image version (CVPR 2023); predict representations, not pixels; targets large and semantic, context informative; sharp targets plus rich context in a prompt is the same recipe.

Check yourself

Question 1: What does I-JEPA predict, and where?
  1. It predicts the next pixel of an image, in pixel space
  2. It predicts the representations of target blocks from a context block, in representation space — correct
  3. It predicts class labels for every target block
  4. It predicts text captions describing the image

Answer: It predicts the representations of target blocks from a context block, in representation space

I-JEPA (Assran et al., CVPR 2023) is non-generative: from one context block it predicts the EMBEDDINGS (representations) of target blocks in the same image - never raw pixels or labels.

Question 2: Why does I-JEPA need no hand-crafted data augmentations?
  1. Because it trains on fully labelled ImageNet
  2. Because it uses reinforcement learning instead
  3. Because predicting in representation space avoids the pixel-level shortcuts that augmentations were invented to prevent — correct
  4. Because it uses a much larger model than previous methods

Answer: Because predicting in representation space avoids the pixel-level shortcuts that augmentations were invented to prevent

Generative pixel-prediction lets a model cheat with low-level cues; predicting in representation space is what let I-JEPA learn semantic features without augmentation engineering (arXiv:2301.08243).

Question 3: What is the prompt-engineering lesson from I-JEPA's masking strategy?
  1. Hide your prompt from the model
  2. Give a spatially distributed, informative context block and target the larger, semantic regions - quality of WHAT you ask the model to predict beats quantity — correct
  3. Always send images instead of text
  4. Randomly delete half your prompt before sending it

Answer: Give a spatially distributed, informative context block and target the larger, semantic regions - quality of WHAT you ask the model to predict beats quantity

I-JEPA's key design choice is large-scale semantic targets plus an informative context block - the same principle as a rich Context section with sharp evaluation targets in TPEE.

← Previous lesson · All 91 lessons · Next lesson →

The full course — 91 lessons and 273 quiz questions — ships inside the app. Get TPEE to study it offline.