Action Chunking and Diffusion Policies

The previous lesson left behavior cloning with two wounds: errors compound quadratically with task length, and DAgger's fix demands an expert in the loop. This lesson takes a different route. Instead of collecting more data at the states the policy visits, it changes what the policy predicts and how it represents actions, so that a policy trained by plain behavior cloning becomes far more robust.

Two advances, both from 2023, carry most of the weight. The first, action chunking, attacks compounding error by having the policy commit to a short burst of future actions at once. The second, Diffusion Policy, changes how actions are generated so the model can represent the many equally-valid ways to perform a task. They are complementary, and together they are why imitation learning went from fragile demos to reliable manipulation. We take them in turn.

A single-step policy predicts one action, executes it, observes, and predicts again. Action chunking instead predicts a whole sequence of the next K actions from one observation, then executes that chunk before replanning.

Why this helps compounding error is direct. In the previous lesson, error accumulated at every control step across the entire episode. If the policy instead commits to K steps at a time, errors can only compound within a chunk, not across the whole trajectory; the robot re-grounds itself in a fresh observation only once per chunk rather than fighting the feedback loop at every tick. There is a tradeoff: too long a chunk and the robot stops reacting to changes in the world; too short and you lose the benefit. Chosen well, chunking turns the quadratic compounding of naive behavior cloning into something far more forgiving.

The technique was crystallized as ACT, Action Chunking with Transformers, introduced by Tony Zhao and colleagues at Stanford in 2023. ACT predicts a chunk of roughly K = 100 future actions, about 2 seconds of motion at a 50 Hz control rate, from the current observation. Architecturally it pairs a Transformer with a conditional variational autoencoder (a CVAE), which lets it capture some variation in how the demonstrated action sequence could go rather than collapsing to a single averaged guess.

ACT was introduced alongside ALOHA, a low-cost bimanual (two-arm) teleoperation setup, and demonstrated tasks long considered out of reach for imitation learning on cheap hardware: slotting a battery, opening a ziploc bag, even threading a zip tie or a needle. The headline lesson was that a fairly simple recipe, chunked predictions on good bimanual demonstrations, could achieve delicate, contact-rich manipulation. ACT remains a strong, widely-used baseline.

There is a subtler flaw in naive behavior cloning that chunking alone does not fix. Many tasks have multiple equally-valid actions at a given moment. To avoid an obstacle you could go left or right; both appear in the demonstrations. A regression loss, trained to minimize average error, will predict the average of left and right, which is straight ahead, into the obstacle.

This is the multimodality problem: when the correct answer is a distribution with several peaks, a model that outputs one number lands in the empty valley between them. It shows up constantly in manipulation, where there are many good grasps, many valid paths, many orderings. What is needed is a policy that can represent a whole distribution over actions and sample one coherent choice from it, rather than averaging incompatible options into an incoherent one. That requirement is exactly what the next idea, borrowed from image generation, provides.

Diffusion Policy, introduced by Cheng Chi and colleagues at Columbia in 2023, borrows the denoising diffusion mechanism from image generators (the same DDPM idea covered in the generative-AI track) and points it at actions instead of pixels.

Rather than regressing to one action, the policy starts from a random noise action sequence and, conditioned on the observation, iteratively denoises it into a clean, coherent action trajectory over several steps. Because diffusion models are built to represent complex, multi-peaked distributions, this naturally solves multimodality: asked to avoid the obstacle, the model samples a coherent go-left trajectory or a coherent go-right trajectory, never the averaged straight-ahead crash. Diffusion Policy also composes cleanly with action chunking, denoising a chunk of future actions at once. The cost is compute: generating an action now takes several denoising passes instead of one forward pass, which matters for real-time control. But its robustness on hard, multimodal manipulation made it one of the most influential robot-learning methods of its year.

Compare the pipelines. Naive behavior cloning maps an observation to a single action and averages away multimodality. ACT maps an observation to a chunk of K actions through a transformer with a CVAE, bounding compounding error. Diffusion Policy conditions on the observation and denoises a noisy action chunk into a coherent trajectory, capturing multiple valid modes. Each row fixes a specific weakness of the row above.

The two dominant chunking policies make different tradeoffs, and practitioners choose by task:

Neither is strictly better. ACT wins on simplicity and speed and fits smooth bimanual tasks; Diffusion Policy wins where the task genuinely has many valid solutions the model must not average. Hybrids exist that put a diffusion action head on ACT's chunked structure, keeping the smoothness while gaining multimodality. The deeper point is that both share a template: a neural network takes camera images plus robot state and outputs a chunk of actions. Change that network from a task-specific model into a large, pretrained one that also understands language, and you arrive at the subject of the final lesson.

One constraint has hovered over both lessons: demonstrations are scarce and expensive. A policy trained on one task with one robot learns only that. The response that reshaped the field was to pool demonstrations across many robots and labs into shared datasets.

Google's RT-1 (December 2022) was an early large collection: about 130,000 demonstrations gathered by a fleet of 13 robots over 17 months. The bigger step was Open X-Embodiment (October 2023), a community effort that combined data spanning 22 different robot types across 34 institutions into one dataset, showing that a single policy could be trained across many robot bodies at once, so-called cross-embodiment learning. This pooled data is the bridge to the next lesson: once you have internet-scale robot demonstrations and a way to fold in internet-scale images and text, you can train one large model to control many robots and follow language instructions. That model is the vision-language-action model.