Imitation Learning: Learning Behavior From Demonstrations

Waymo trained a driving policy on roughly 26 million examples — about 60 days of continuous driving — and it still could not recover from a mistake. The 2019 ChauffeurNet paper puts the verdict in its abstract: “30 million examples are still not enough”. The introduction says why the authors had expected otherwise: “Despite recent results that point to error accumulation no longer being an issue with good representations and high capacity models (Laskey et al. [10]), our first finding was that not even 30 million examples were sufficient for pure imitation learning.”

What the data lacked was not volume but coverage. Waymo's own account of the system states it directly: “Expert driving demonstrations obtained from real-world driving typically contain only examples of driving in good situations, because for obvious reasons, we don't want our expert drivers to get into near-collisions or climb curbs just to show a neural network how to recover in these cases.”

The closed-loop test made that visible. Across 20 lane-departure scenarios the pure-imitation baseline M0 “is not able to recover in any of the situations”. Two other models, M3 and M4, were trained on synthesised trajectory perturbations, with environment losses and a reduced weight on the imitation loss. They “handle all deviations well”. Sixty days of good driving did not teach recovery. Twenty fabricated departures did.

Imitation learning uses examples of behavior produced by a teacher, an expert, a historical policy, or a planner. From those trajectories the learner tries to reproduce or infer a policy of its own.

Behavioral cloning is the simplest method. It treats observation–action pairs as supervised examples. Sequential deployment then creates an extra difficulty, because the learned policy changes the observations it later encounters.

The method is 37 years old, and its central defect was named in the paper that introduced it. ALVINN was a 3-layer back-propagation network trained for 40 epochs on 1,200 simulated road snapshots. In 1988 it drove Carnegie Mellon's NAVLAB van at half a metre per second, along a 400 metre path through a wooded part of the campus, under sunny fall conditions. Discussing what it would take to train on the fly with real images rather than simulated ones, Dean Pomerleau wrote that “the network must not solely be shown examples of accurate driving, but also how to recover (i.e. return to the road center) once a mistake has been made.” Everything since — NVIDIA's synthesised off-centre views, ChauffeurNet's perturbations, interactive correction — is an answer to that one sentence.

One imperfect action can move the learner away from the demonstration distribution, where subsequent predictions have weaker support.

The compounding has been quantified. Ross and colleagues stated the worst case in 2011, in one sentence: “a classifier that makes a mistake with probability ε under the distribution of states/observations encountered by the expert can make as many as T²ε mistakes in expectation over T-steps under the distribution of states the classifier itself induces”. Errors grow with the square of the horizon, not with its length.

Their Super Tux Kart experiment shows why more demonstration does not repair it. Under the plain supervised approach “performance does not improve as more data is collected … because most of the training laps are all very similar and do not help the learner to learn how to recover from mistakes it makes”. Their DAgger policy instead collects fresh expert labels in the states the learner actually reaches. It “never falls off the track after 15 iterations of training”.

The theory and the fleet agree. A quadratic bound is what 26 million examples run into.

Human information can enter sequential learning in several forms. The teacher shows a complete or partial trajectory. The teacher takes over, or supplies the better action, in the difficult states. Or the teacher evaluates outcomes and alternatives after the fact.

Each form buys something and costs something. Demonstration is efficient when good behavior is easy to show, but it omits failure recovery and copies hidden habits. Correction targets learner-specific failures, but it burdens the teacher and can endanger them. Reward or preference allows behavior beyond exact copying, but it requires objective design, and the feedback can arrive delayed or ambiguous.

One deployed system stacked two of the three and published the price of each. For InstructGPT, OpenAI hired about 40 contractors, supervised-fine-tuned GPT-3 on written demonstrations, trained a 6B reward model on human rankings, and ran PPO against that reward. The 2022 paper gives the three sizes: “The SFT dataset contains about 13k training prompts (from the API and labeler-written), the RM dataset has 33k training prompts (from the API and labeler-written), and the PPO dataset has 31k training prompts (only from the API).”

The demonstration set is the smallest of the three. It also carried the most. Labelers preferred the resulting 1.3B-parameter model's outputs to those of the 175B GPT-3, a model with over 100 times more parameters.

The teachers were not an oracle either. In the same study the training labelers agree with each other 72.6 ± 1.5% of the time. That number is a ceiling on what any of the three signals can be said to encode.

The trajectory dataset should be reviewed as a record of behavior and opportunity. The cases above give the audit its questions.

Define expertise: state who produced the behavior and what evidence supports competence. Robomimic's six teleoperators were graded better, okay and worse before a single policy was fitted.

Capture context: log observations, available actions, constraints, and intervention authority. The asthma rule was legible only once someone knew where those patients had been admitted.

Find coverage gaps: search for rare states, recoveries, boundary cases, and changing conditions. ChauffeurNet's 60 days of driving contained none of the twenty lane departures it was later tested on.

Preserve disagreement: record alternate expert actions and reasons where available, rather than averaging them into a single target.

Test closed-loop behavior: evaluate full trajectories rather than isolated action accuracy. The T²ε bound is about trajectories, not about per-step error.

Add safe fallback: define takeover, constraints, and stop conditions before deployment.

Demonstrations can provide a safe and competent starting policy. Reinforcement, preferences, planning, or targeted corrections then improve specific behaviors.

AlphaGo is that arc with numbers on it. A 13-layer supervised-learning policy network was trained on 30 million positions from the KGS Go Server. It reached 57.0% accuracy at predicting expert moves on a held-out test set, and 55.7% from raw board position and move history alone, against a prior state of the art of 44.4%. A self-play reinforcement-learning policy network was then initialised from it, and won more than 80% of games head-to-head against it.

The stronger policy is not the one that shipped inside the search. The paper, in Nature in 2016, notes the reversal: “It is worth noting that the SL policy network pσ performed better in AlphaGo than the stronger RL policy network pρ, presumably because humans select a diverse beam of promising moves, whereas RL optimizes for the single best move.”

This combination should preserve constraints and independent evaluation. Additional optimization can erase desirable teacher behavior if the new reward is poorly aligned. Here the erased property was breadth. The refined policy was better at winning and worse at proposing candidates, and the search needed candidates.

A sequential policy should be judged on trajectories and recovery, not only on agreement with the demonstrator. The first of these measures is not an informal one. California defines the event a fleet must count, in regulation: “For the purposes of this section, "disengagement" means a deactivation of the autonomous mode when a failure of the autonomous technology is detected or when the safe operation of the vehicle requires that the autonomous vehicle test driver disengage the autonomous mode and take immediate manual control of the vehicle, or in the case of driverless vehicles, when the safety of the vehicle, the occupants of the vehicle, or the public requires that the autonomous technology be deactivated.”

Every permitted manufacturer must file an annual report by 1 January, covering 1 December to 30 November. Under that rule the California DMV reported on 2 February 2024 that test vehicles drove a record 9,068,861 miles in autonomous mode on California public roads in the 1 December 2022 – 30 November 2023 period: 5,801,069 miles with a safety driver and 3,267,792 miles of driverless testing. A rate needs a denominator. That is one a regulator publishes.