Vision-Language-Action Models

The previous lessons built policies from task-specific networks trained on demonstrations. They work, but each starts from scratch, knowing nothing about the world beyond its own demos. A robot asked to "pick up the extinct animal" from a table of toys has no way to know a plastic dinosaur qualifies. That knowledge lives in internet-scale text and images, not in robot data.

A vision-language-action model (VLA) closes that gap. The recipe is deceptively simple: take a vision-language model (a VLM, already pretrained on huge amounts of image-and-text data, as covered in the multimodal-AI content) and give it a way to output robot actions, then train it on robot demonstrations. The robot policy inherits the VLM's understanding of objects, language, and relationships, and only has to learn how that understanding maps to motion. This is the through-line of the whole field right now, and the rest of the lesson unpacks how it is actually done.

A quick anchor, since the VLA rests on it. A vision-language model takes images and text together and produces text. It pairs a visual encoder, which turns an image into a sequence of tokens, with a large language model that processes those visual tokens alongside word tokens. Trained on enormous image-text corpora, a VLM can describe a scene, answer questions about a photo, or reason about what it sees in words.

Crucially, that training gives the VLM broad semantic and commonsense knowledge: what objects are, how they relate, what instructions mean. It knows a dinosaur is an extinct animal and that a mug is for drinking. The bet behind VLAs is that this knowledge is exactly what a general-purpose robot lacks and cannot easily get from scarce robot demonstrations. The engineering question is how to add a physical output, actions, to a model whose native output is text.

The model that made the VLA idea concrete was RT-2 from Google DeepMind (2023). Its trick for adding actions was elegant: represent actions as text tokens. A robot action, the change in gripper position and orientation and its open/close state, is discretized into numbers, and those numbers are written as tokens the language model already emits. To the model, producing an action is just producing another short string.

Because actions are tokens, RT-2 can be trained on robot trajectories in the very same way it was trained on text, and it is co-fine-tuned: trained jointly on robot demonstrations and ordinary internet vision-language tasks like visual question answering. That joint training is what lets web knowledge flow into control. RT-2 built directly on RT-1's data and architecture, but where RT-1 struggled outside its training distribution, RT-2 could act on objects, categories, and instructions it had never physically practiced, because it had read and seen them on the web.

The payoff of putting a VLM underneath the policy is generalization the robot never explicitly practiced. Because the model already grounds language and images in web-scale meaning, a VLA can respond sensibly to novel objects, novel phrasings, and simple reasoning, on tasks no demonstration covered.

Concretely, RT-2 could follow instructions that required connecting a request to knowledge outside its robot data: choosing the object that is an extinct animal, or the one that could be a makeshift hammer, or moving something toward a named logo. None of these associations came from robot demonstrations; they came from the VLM's pretraining. This is the qualitative shift VLAs introduced. Earlier policies generalized within the neighborhood of what they were shown. A VLA can generalize using a vast body of knowledge acquired entirely outside the robot, which is what a genuinely general-purpose robot needs. The remaining question was cost and openness, since RT-2 was large and closed.

OpenVLA (2024) brought the recipe into the open. It is a 7-billion-parameter VLA trained on about 970,000 robot episodes from the Open X-Embodiment dataset of the previous lesson. Its backbone is a Llama 2 language model paired with a visual encoder that fuses features from two pretrained vision models, DINOv2 and SigLIP, so it sees both fine geometry and semantic content.

Two results made it influential. First, despite having roughly 7x fewer parameters, OpenVLA reported outperforming the much larger closed RT-2-X (55 billion parameters) by about 16.5 percentage points in absolute task success across a suite of tasks and multiple robot embodiments, evidence that a well-built 7B backbone had enough capacity for real language-conditioned generalization. Second, being fully open, weights and code, it turned VLAs from a few labs' capability into something the wider community could fine-tune and build on. OpenVLA is a good mental model of the whole class: a mid-sized VLM, a fused visual encoder, and action outputs, trained on pooled cross-embodiment data.

Representing actions as discrete tokens, RT-2's approach, is simple but coarse: real robot motion is continuous and fast, and token-by-token generation can be a clumsy, slow fit for smooth dexterous control. pi0 (written with the Greek letter pi and a zero), from Physical Intelligence in 2024, addresses this.

pi0 keeps a pretrained VLM (about 3 billion parameters) as its knowledge base but attaches a dedicated action expert that outputs continuous actions using flow matching, a generative technique related to diffusion that produces smooth, high-frequency action chunks in one efficient generation rather than many discrete token steps. This hybrid, a VLM for understanding plus a flow-matching head for fluent continuous control, targets exactly the real-time dexterity that token-based VLAs find awkward. It is representative of the current direction: keep the VLM for semantics, but pair it with an action representation, whether flow matching or diffusion, suited to fast, continuous, contact-rich motion. The mechanisms of the previous lesson reappear here, now bolted onto a web-pretrained brain.

Every model here shares one skeleton. Camera images and a language instruction enter a vision-language model pretrained on web data, which supplies semantic understanding. An action head converts that understanding into robot actions, as discrete tokens (RT-2, OpenVLA) or continuous chunks via flow matching or diffusion (pi0). Trained on pooled cross-embodiment demonstrations, the same architecture drives many different robots.

VLAs are the frontier, not a finished product, and the open problems are as important to know as the wins:

• Data scarcity. Robot demonstrations remain tiny next to web data. The VLM is pretrained on the internet, but the action-learning still leans on expensive teleoperation, and this is the field's central bottleneck.
• Inference speed. A multi-billion-parameter model must run fast enough for real-time control. This is a live engineering constraint and part of why continuous-action designs like pi0 exist.
• Reliability and safety. Generalization is impressive but uneven; success rates on novel tasks are often far from the near-perfect reliability physical deployment demands.
• Evaluation is hard. Real-robot results are costly to reproduce and hard to compare across labs, hardware, and setups, which muddies claims of progress.

The honest summary: VLAs unify the pieces of this cursus, imitation learning, chunked and generative action outputs, pooled cross-embodiment data, under a web-pretrained VLM, and that combination generalizes in ways older policies could not. Whether it scales into dependable general-purpose robots is exactly the question the field is now working on, and it is why terms like robot foundation model appear across recent research.