Offline Reinforcement Learning and Imitation Learning

Online reinforcement learning can try an action, observe the consequence, and collect new data. Offline RL receives a fixed dataset of logged transitions and must learn without further interaction. That is attractive in healthcare, robotics, operations, and recommendation, where exploration may be costly or unsafe. The fixed dataset creates a central problem. A learned policy may choose actions rarely or never taken by the behavior policy that generated the logs. Value estimates for those actions rely on extrapolation. Small errors compound through Bellman backups, and unsupported actions start to look unrealistically valuable.

Imitation learning avoids some of this by copying observed behavior. But it may inherit suboptimal decisions, and it can drift into unfamiliar states when small mistakes accumulate. Offline RL tries to improve beyond the behavior while staying inside the dataset's support.

Two standard algorithms broke on a fixed batch in 2019, and Fujimoto and colleagues named the cause in the first line of their abstract: “In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting.” Two named algorithms, one named cause. The critic is asked about actions the log never tried, and it answers anyway.

That sentence is often quoted as if it were a law of offline learning. It is not. A year later a team at Google Brain reached the opposite result. They assembled the DQN Replay Dataset: 50 million (observation, action, reward, next observation) tuples per game across 60 Atari 2600 games, split into 5 replay datasets per game, 300 datasets in all. Offline QR-DQN and offline REM, trained with no environment interaction at all, outperformed the fully trained online DQN agent. The difference between the two results is not the algorithm family. It is the log. Extrapolation error is what a narrow dataset does to a critic, not a fixed property of learning without exploration.

One unsupported choice can move the agent into states that are also absent from the dataset. That is the chain behind extrapolation error. The critic is questioned about an action the log never tried. It answers from whatever the network extends into that gap. The policy is then pulled toward the states where that answer applies.

The methods differ in how far they try to move beyond recorded behavior.

The first column is not a toy. On 13 October 2023 a collaboration of robotics labs released a pooled behavior-cloning corpus and described it plainly: “The Open X-Embodiment Dataset contains 1M+ real robot trajectories spanning 22 robot embodiments, from single robot arms to bi-manual robots and quadrupeds. The dataset was constructed by pooling 60 existing robot datasets from 34 robotic research labs around the world and converting them into a consistent data format for easy download and usage.” Across 21 institutions the corpus covers 527 skills and 160,266 tasks. RT-1-X, the policy trained on it, had a 50% higher success rate than the original state-of-the-art methods each institution had contributed.

The honest half of the result sits in the same paper. In the large-dataset domains RT-1-X did not outperform the single-dataset RT-1 baseline. Pooling bought improvement where a lab's own log was thin and bought nothing where it was already thick. That is the support argument the other two columns make as well, with reward optimization attached. The question is never how much data there is. It is whether the data covers the decisions the policy wants to make.

The dataset is a record of decisions. It is not a neutral sample of the environment. Most healthcare offline-RL work runs on a single log, MIMIC-III, and the 2016 paper describing it is exact about what it holds: “MIMIC-III contains data associated with 53,423 distinct hospital admissions for adult patients (aged 16 years or above) admitted to critical care units between 2001 and 2012.” Those admissions cover 38,597 distinct adult patients. A further 7,870 neonates were admitted between 2001 and 2008. All of it comes from one hospital, Beth Israel Deaconess Medical Center in Boston. A policy learned from it has been shown one hospital's practice inside the window that sentence names. That is the object the checklist audits.

No single offline score should authorize a high-impact policy change. Step 2 is the one that gets treated as a formality, and it is the one with a measured warning attached. An independent group at Nanjing University and Polixir built NeoRL, a benchmark with controlled dataset sizes and low-exploration logging, and evaluated BCQ, CQL, PLAS and MOPO on it. The algorithms were often no better than the deterministic version of the behavior policy, and offline policy evaluation gave little help in choosing between them. If the offline estimator cannot rank the candidates, the policy already in production is the only ranking available. Step 4 is then a source of evidence, not a verdict.