From Control to Imitation Learning

The robotics track in this catalog builds control from models: you describe the system's dynamics, then compute the commands that drive it to a goal (PID, state-space control, trajectory tracking). That approach is powerful and, where it applies, hard to beat. It dominates when you can write down the dynamics and the task is well-structured: a robot arm following a known path, a drone holding a setpoint.

It runs out when the task is not cleanly modelable. Consider folding a shirt, threading a needle, or picking a novel object from a cluttered bin using camera images. The dynamics of cloth, the contact forces, the sheer variety of objects, none of these reduce to a tidy equation you can control against. For a growing class of contact-rich, vision-driven, high-variety tasks, the modern answer is to stop hand-designing the controller and instead learn a policy from data. This lesson is where that shift begins.

The object we want to learn is a policy: a function that maps what the robot observes to what it should do. Formally, a policy takes an observation (typically one or more camera images plus the robot's own joint positions) and outputs an action (the next command, for example target joint positions or end-effector motion, and the gripper open/close).

The robot runs this policy in a loop, many times per second: observe, compute an action, execute it, observe again. If the policy is good, that loop produces the whole behavior, reaching, grasping, placing, without anyone specifying the trajectory in advance. The learning question is then concrete: how do we obtain a good policy? One family of answers, reinforcement learning, has its own cursus and learns by trial and error against a reward. This cursus follows the other dominant path: imitation learning, where the robot learns from examples of the task done correctly.

Behavior cloning is imitation learning in its most direct form, and it is exactly supervised learning applied to control. You collect a dataset of demonstrations, each a stream of (observation, action) pairs recorded while an expert performs the task. Then you train a neural network to predict the expert's action from the observation, using an ordinary regression loss.

# Behavior cloning, in essence
for obs, expert_action in demonstrations:
    predicted = policy(obs)          # image + joint state -> action
    loss = mse(predicted, expert_action)
    loss.backward();  optimizer.step()

That is the whole training recipe. Conceptually it is appealing: no reward function to design, no dynamics model to derive, no trial-and-error on real hardware. Give the network enough good demonstrations and it learns to reproduce the mapping from what-I-see to what-to-do. Behavior cloning is the foundation the rest of this cursus builds on. But used naively it has a deep, well-understood failure mode, and understanding it explains almost every design choice that follows.

Behavior cloning trains on the states an expert visits. But once the robot drives itself, it visits the states its own policy leads to, and those are not quite the same. This mismatch is distribution shift: the training distribution (expert states) differs from the test distribution (policy states), and supervised learning gives no guarantee about behavior on states it never trained on.

Why it bites so hard in control is the feedback loop. A tiny prediction error nudges the robot slightly off the demonstrated path, into a state a little less familiar. There the policy is a little less certain, so it errs a little more, pushing it further off, into states even less like the training data. Errors do not just add up; they feed on themselves. An experienced human demonstrator never showed the robot how to recover from being off-track, because the human never went off-track. The robot, left alone, drifts into exactly that unlabeled territory.

The self-reinforcing drift has a name and a scaling law: compounding error. A classic analysis shows that if a behavior-cloned policy makes an error with probability of order epsilon at each step, the expected number of mistakes over a task of horizon T grows not as $T\epsilon$ but as $O(T^2 \epsilon)$.

That quadratic is the whole story. It means the penalty for small per-step errors explodes as tasks get longer: double the length of the task and you roughly quadruple the accumulated error. A per-step accuracy that looks excellent in a supervised-learning report can still yield a policy that reliably fails a long-horizon manipulation task, because the errors compound. This single fact, that naive behavior cloning degrades quadratically with task length, is why the field did not stop at supervised learning, and why the techniques in the next lesson exist.

The mechanism is a loop. The policy acts, a small error moves the robot slightly off the expert's distribution, the new state is less familiar, so the next error is larger, moving it further off still. Each pass around the loop enlarges the gap. The two fixes shown break the loop in different places: DAgger adds expert labels in the drifted states, and action chunking (next lesson) shortens how far errors can accumulate before a replan.

The first principled fix targets distribution shift directly. DAgger (Dataset Aggregation), introduced by Ross and colleagues in 2011, works iteratively. Train a policy by behavior cloning, then let it drive the robot and record the states it actually visits, including the off-track ones. Have the expert label the correct action for those states. Add this new data to the dataset and retrain. Repeat.

Over rounds, the dataset comes to include exactly the drifted, off-distribution states the policy encounters, so the policy learns how to recover from its own mistakes. DAgger is theoretically sound and closes the training-test gap. Its cost is practical: it requires an expert available to label the robot's visited states round after round, often watching and correcting in real time. That human-in-the-loop expense motivated a different question, less about collecting more data at the states the policy visits, and more about redesigning what the policy predicts so errors cannot compound as fast. That is the next lesson.

Every method here runs on demonstration data, so how you collect it matters as much as the algorithm. The dominant source is teleoperation: a human directly operates the robot, often through a matched pair of leader and follower arms, while the system records synchronized camera images and joint actions at each timestep. The influential ALOHA setup popularized low-cost bimanual teleoperation for exactly this purpose.

The economics are the catch. Every demonstration is human time on real hardware; there is no equivalent of scraping the internet. This scarcity shapes the whole field: it is why methods are judged partly on data efficiency, and why, as the next lessons show, the community pooled demonstrations across many labs and robot types into shared datasets. Keep this in mind as a running constraint: robot learning is gated not just by clever architectures but by how much good demonstration data anyone can afford to collect.