Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism

AI summary

Overview

Research area: Offline reinforcement learning, model-based RL, and Bayesian RL. Technical level: Advanced — the paper assumes familiarity with MDPs, epistemic POMDPs, ensembles, and recurrent actor-critic training. Scope: The paper argues that explicit conservatism is not necessary for offline RL, and instead presents a Bayesian-principled, model-based algorithm (Neubay) that relies on adaptive long-horizon rollouts to control value overestimation.

What This Paper Is About

Most offline RL algorithms inject pessimism — penalizing actions outside the dataset or restricting rollouts to short horizons — to avoid trusting unreliable value estimates. This paper asks whether that pessimism is actually necessary, and instead treats the unknown environment as a random variable, maintaining a posterior over possible world models and training a history-dependent agent to maximize expected return. The central practical discovery is that once the pessimism penalty is removed, long-horizon rollouts become essential, because they shift value estimation away from error-prone bootstrapping and toward model-generated rewards, thereby absorbing the overestimation risk that conservatism used to handle.

Key Contributions

1. A Bayesian alternative to conservatism, motivated by data quality. The authors revisit the epistemic POMDP formulation of offline RL and show, both in a two-armed bandit and theoretically, that Bayesian agents provably outperform conservative ones on low-quality or narrowly covered datasets, while remaining competitive on high-quality data.

2. The insight that long-horizon rollouts curb value overestimation. They decompose the H-step TD target to show that imagined rewards are low-bias while the bootstrapped term is discounted by γ^H, so longer rollouts reduce reliance on the most overestimation-prone part of the update. This is a distinct role for long horizons, separate from data augmentation or exploration.

3. The Neubay algorithm. A practical model-based offline RL method combining uncertainty-quantile-based adaptive rollout truncation, larger model ensembles (N=100), layer normalization inside the world model (with a linear compounding-error bound), and a recurrent actor-critic with linear recurrent units for long-term memory.

4. Broad empirical validation. Evaluation across 33 datasets from D4RL and NeoRL — including locomotion, Adroit manipulation, and AntMaze navigation — with rollout horizons routinely spanning 64–512 steps, achieving new state-of-the-art on 7 datasets.

Main Findings

• Conservatism is provably limiting on skewed data. In a two-armed bandit where only arm 0 is observed, any conservatism coefficient large enough to suppress the high-uncertainty unseen arm forces the agent to always select arm 0, regardless of what the test environment actually rewards. The Bayesian agent briefly explores the unseen arm and commits if it is better.

• The performance gap is characterized by coverage ratios. The authors define Bayesian- and robust-coverage ratios; the robust ratio can grow arbitrarily large in skewed datasets, while posterior averaging bounds the Bayesian ratio — explaining the advantage.

• Long horizons lower estimated Q-values on offline data. Empirically, as the rollout truncation threshold ζ increases from 0.9 to 1.0, performance improves while the estimated Q on the offline dataset drops, and rollout horizons grow substantially.

• Short-horizon variants fail without conservatism. Once the uncertainty penalty is removed, short rollouts cause severe overestimation due to excessive bootstrapping. This reproduces a phenomenon that conservatism normally masks.

• Neubay is competitive but not dominant on average. On D4RL locomotion, Neubay reaches an average normalized score of 80.1 versus 83.6 for the strongest baseline (VIPO), but wins decisively on several individual datasets and is the only Bayesian-inspired method near the top of the table.

• Dataset characterization predicts when Neubay is preferable. It performs best on low-quality datasets and on medium-quality datasets with moderate coverage — precisely the regime where test-time adaptation beyond the data matters most.

• Layer normalization yields a linear compounding-error bound. Because a normalized feature vector has fixed ℓ2 norm √k, the H-step state deviation is bounded by H√k‖W‖, giving a controllable worst-case error growth.

Methodology in Plain English

The researchers approach offline RL as a Bayesian inference problem rather than a robustness problem. They train an ensemble of neural world models on the offline dataset; the disagreement among ensemble members serves as a measure of epistemic uncertainty about the true environment.

Training proceeds in a loop. They sample a starting state and its preceding history from anywhere in the offline dataset, draw one ensemble member and fix it for the duration of the rollout, then simulate an imagined trajectory with the current recurrent policy. Truncation is decided adaptively: a rollout stops as soon as the ensemble disagreement at the current state–action pair exceeds a quantile threshold computed from in-dataset uncertainties, which lets reliable regions of the state space be visited for hundreds of steps while cutting off unreliable ones. The policy and critic are trained on a mix of real and imagined transitions, weighted by a mixing ratio, using recurrent networks with special long-memory units and a deliberately small learning rate on the recurrent encoders to keep history representations stable.

Two architectural tweaks matter most: using a much larger ensemble than is customary (100 members rather than 5) to improve posterior fidelity over long rollouts, and inserting layer normalization inside the world model as a delta predictor to keep compounding error growth linear rather than explosive.

Why This Matters

The paper challenges a foundational assumption in offline RL — that pessimism is unavoidable — and offers a competing principle grounded in Bayesian decision theory. If validated further, it reframes long-horizon rollouts from a liability (compounding error) into a tool (overestimation control), which inverts standard model-based RL practice.

Real-world applications:

• Robotics and industrial control, where learning from a fixed set of demonstrations or logs is the only safe option and optimal behaviors may be rare in the data.
• Recommendation and personalization systems, where logged interactions cover only a narrow slice of the action space and adaptation to new users requires reasoning about unobserved options.
• Healthcare treatment planning, where retrospective patient records are skewed by clinical policy and off-policy evaluation must generalize cautiously but usefully.
• Autonomous driving and safety-critical control, where pre-collected driving logs cannot cover every rare scenario and where pure pessimism would prevent any behavioral improvement.

Industry relevance is direct for teams that own large offline datasets but cannot run live experiments: the approach offers a path to competitive performance without the hyperparameter-heavy uncertainty penalties that conservative model-based methods typically require.

Future Directions

• Better world models. The authors note that future advances in world modeling would directly raise Neubay's ceiling, since its performance is bounded by ensemble fidelity over long rollouts.
• Richer theoretical characterization. The robustness-versus-Bayesianism trade-off is currently illustrated on bandits and bounded via coverage ratios; generalizing this to full MDPs and predicting per-dataset which principle wins remains open.
• Sparse-reward and high-dimensional tasks. AntMaze and Adroit remain difficult for model-based methods generally, raising the question of how much of Neubay's advantage survives in those regimes.
• Extending the Bayesian principle beyond control. The paper hints at language reasoning as a target domain, suggesting that the same posterior-averaging logic could apply to offline sequence-decision problems.

Target Audience

Offline RL and model-based RL researchers who want a rigorous alternative to the conservatism paradigm; practitioners working with large static datasets where exploration during training is impossible; and Bayesian RL theorists interested in empirical instantiations of epistemic POMDPs at scale. Readers should already be comfortable with Bellman backups, ensemble uncertainty, POMDPs, and recurrent policy optimization.

Authors’ abstract

Popular offline reinforcement learning (RL) methods rely on explicit conservatism, penalizing out-of-dataset actions or restricting rollout horizons. We question the universality of this principle and revisit a complementary Bayesian perspective for test-time adaptation. By modeling a posterior over world models and training a history-dependent agent to maximize expected return, the Bayesian approach directly addresses epistemic uncertainty without explicit conservatism. We first illustrate in a bandit setting that Bayesianism excels on low-quality datasets where conservatism fails. Scaling to realistic tasks, we find that long-horizon rollouts are essential to control value overestimation once conservatism is removed. We introduce design choices that enable learning from long-horizon rollouts while mitigating compounding model errors, yielding our algorithm, NEUBAY, grounded in the neutral Bayesian principle. On D4RL and NeoRL benchmarks, NEUBAY is competitive with leading conservative algorithms, achieving new state-of-the-art on 7 datasets with rollout horizons of several hundred steps. Finally, we characterize datasets by quality and coverage to identify when NEUBAY is preferable to conservative methods.