Sample-efficient and Scalable Exploration in Continuous-Time RL

AI summary

Overview

• Research area: Reinforcement learning (RL) and optimal control, specifically continuous-time model-based RL with probabilistic dynamics models.
• Technical level: Advanced. The paper combines nonlinear ODE modeling, Gaussian process / Bayesian uncertainty theory, regret analysis, and deep RL experiments.
• Scope: The paper introduces COMBRL, a continuous-time optimistic model-based RL algorithm that explores by adding model epistemic uncertainty as an intrinsic reward, with sublinear-regret and sample-complexity guarantees and evaluations on Gym and DeepMind Control benchmarks.

What This Paper Is About

Most RL algorithms assume time is discretized, even though real control systems (robots, biological and physical processes) are described by continuous-time ordinary differential equations (ODEs). The authors study continuous-time model-based RL, where an agent must learn an unknown ODE from measurements taken at chosen time points, and ask how to explore efficiently when the agent must also decide when to observe and control the system. Their goal is an algorithm that balances solving a specific task with reducing model uncertainty, and that scales better than prior continuous-time methods.

Key Contributions

1. COMBRL algorithm: A continuous-time optimistic model-based RL algorithm that selects policies by maximizing a weighted sum of extrinsic reward and model epistemic uncertainty, controlled by a single scalar λ_n. It supports both reward-driven and unsupervised (no extrinsic reward) learning, unlike prior continuous-time methods that are purely exploitative or rely on costly co-optimization.
2. Theoretical guarantees: A sublinear regret bound in the reward-driven case, R_N ≤ O(√(I_N³(f*, S) · N)), and a sample complexity bound in the unsupervised setting, showing epistemic uncertainty shrinks at rate O(√(I_N³(f*, S) / N)). Both depend explicitly on the measurement selection strategy (MSS).
3. Scalability: By selecting any model from the confidence set (in practice the mean) instead of jointly optimizing over plausible dynamics with the reparametrization trick of Curi et al. (2020), COMBRL avoids the input-dimensionality increase from d_u to d_u + d_x that limits prior methods such as OCORL.
4. Empirical evaluation: Demonstrations on Gym and DeepMind Control tasks showing better scaling, greater sample efficiency, and better generalization to unseen downstream tasks than baselines, including in a time-adaptive control & sensing (TaCoS) setting.

Main Findings

• Sublinear regret in the reward-driven setting: Theorem 1 states that with probability at least 1 − δ, regret is bounded by O(√(I_N³(f*, S) · N)), where I_N(f*, S) is the model complexity — the maximum over policies of the summed integral of squared predictive uncertainty along trajectories. Since I_N is sublinear for common kernels and MSSs (for example, poly log(N) for the RBF kernel with an equidistant MSS), the policy converges to the optimal policy.
• Sample complexity in the unsupervised setting: Theorem 2 shows that for λ_n → ∞, the maximum integral of predictive uncertainty is O(√(I_N³(f*, S) / N)). The authors state they are the first to show this for continuous-time RL.
• Measurement schedule matters: In continuous time, uncertainty accumulates along whole trajectories, so regret depends on the policy and the measurement selection strategy. Recovering the full ODE asymptotically requires increasingly dense observations, for example |S_n| = n for an equidistant MSS, for which Treven et al. (2023) bound I_N(f*, S) = O(γ_N + log N) with γ_N the maximum information gain.
• Three exploration regimes defined by λ_n: Greedy (λ_n = 0, pure exploitation), balanced (0 < λ_n < ∞, task-directed exploration), and unsupervised (λ_n → ∞, pure uncertainty reduction equivalent to active learning / uncertainty sampling). Practical strategies include static (fixed hyperparameter), scheduled (for example λ_n = λ_0 · (1 − n/N)), and auto-tuned via the procedure of Sukhija et al. (2025a).
• Better asymptotic returns with GPs: With Gaussian process dynamics, iCEM planning, and the pendulum swing-up and mountaincar tasks, COMBRL achieves higher asymptotic returns than PETS and the mean planner while matching or exceeding OCORL at roughly 3× lower computational cost. Results are averaged over 5 random seeds with standard error bands; MPC on the true dynamics is reported as an estimate of the optimal policy's performance.
• Intrinsic reward helps most in sparse or underactuated tasks: Using probabilistic ensembles and SAC, with λ_n auto-tuned, COMBRL reaches higher asymptotic returns across Gym and DMC environments, with the largest gains in MountainCar and CartPole and consistent improvements in higher-dimensional domains such as HalfCheetah, Hopper, and Humanoid. Results are averaged over 10 random seeds with standard error.
• Unsupervised exploration generalizes better to unseen tasks: When the reward is ignored (λ_n → ∞), the unsupervised variant performs best on downstream tasks across all seven evaluated Gym domains, while standard COMBRL is generally best on the primary task, especially with sparse rewards.
• Trade-off in λ_n: Ablations on HalfCheetah, Hopper, Reacher, and Pusher show that growing nonzero λ_n improves downstream generalization while maintaining primary-task performance, whereas very large λ_n overly favors exploration and degrades the primary task — suggesting an intermediate value balances both.
• Time-adaptive control works: In the TaCoS framework, where interactions incur transition costs and the agent optimizes both actions and their durations, COMBRL-TaCoS achieves competitive or superior returns with fewer interactions than its fixed-control-rate variant and matches or exceeds OTaCoS, Mean-TaCoS, and PETS-TaCoS. Final returns are averaged over 10 random seeds with standard error.

Methodology in Plain English

The agent interacts with an unknown continuous-time system in episodes. In each episode it picks a policy, rolls it out for a horizon T, and takes measurements at selected time points defined by a measurement selection strategy; because measuring the state derivative directly may be infeasible, derivatives are estimated by finite differences or filtering and are subject to noise. The collected data are used to fit an uncertainty-aware dynamics model that returns both a mean prediction and an epistemic uncertainty estimate. The next policy is chosen by maximizing an objective that averages the extrinsic reward and the model's predicted uncertainty, scaled by λ_n; for Gaussian processes this yields formal optimism, and it works with any statistical model that is well-calibrated (Gaussian processes, Bayesian neural networks, ensembles, or other epistemic-uncertainty estimators). Because the model represents the ODE directly, the approach is agnostic to the solver, discretization scheme, planner, and measurement strategy. The authors also prove regret and sample-complexity results under Lipschitz continuity of dynamics, reward, and policies, sub-Gaussian noise, well-calibration of the model, and an RKHS prior on the dynamics. In experiments they use Gaussian processes and probabilistic ensembles as the dynamics model, with iCEM for real-time trajectory optimization and SAC for closed-loop control, and additionally the TaCoS framework for time-adaptive experiments.

Why This Matters

• Research impact: The paper extends the optimism-in-the-face-of-uncertainty principle to continuous time using a simple intrinsic-reward mechanism rather than intractable co-optimization, and it links regret explicitly to the measurement schedule — something discrete-time analyses do not capture. The authors describe COMBRL as agnostic to the statistical model, trajectory planner, and measurement and control strategy.
• Real-world applications (as motivated by the paper):
  • Robotics, where continuous-time dynamics and variable control frequencies match physical sensing and actuation.
  • Biological processes and other systems naturally modeled by ODEs.
  • Medicine and energy applications involving continuous physical dynamics.
  • System identification and unsupervised exploration, where the goal is an accurate global model for zero-shot reuse on new objectives rather than one task reward.
• Industry relevance: Enabling agents to choose when to sense and control, rather than acting on a fixed clock, matters for hardware with irregular or costly measurements; the unsupervised variant produces models that transfer to new objectives without retraining, and COMBRL's reduced compute (roughly 3× lower than OCORL in the GP experiments) lowers the cost of deployment-scale experiments.

Future Directions

• Extending the guarantees beyond well-calibrated GP models to Bayesian neural networks and ensembles, where the authors note re-calibration techniques can be used but the theoretical treatment differs.
• Principled selection of λ_n: the paper treats λ_n primarily as a tuned hyperparameter with scheduling and reports one auto-tuning approach; how to set it for a given task, horizon, and measurement budget remains open.
• Scaling to higher-dimensional and image-based control: the authors note that prior optimistic-dynamics methods are limited by increased input dimensionality, for example control from pixels, and position their method as more scalable, implying this frontier remains active.
• Closing the gap between theory and practice on measurement schedules: since the bounds depend on the MSS, understanding which adaptive schedules yield the best regret and sample complexity is a natural next step, alongside the time-adaptive results shown here.

Target Audience

Researchers and practitioners in continuous-time reinforcement learning, model-based RL, Bayesian optimization, and optimal control who are interested in exploration with uncertainty-aware models and regret guarantees. It also suits engineers applying system identification, active learning, or sample-efficient control to physical systems, and readers comfortable with ODEs, Gaussian processes, and RL notation. Portions of the analysis and the deeper appendices (proofs, implementation details, and additional results) are advanced, while the core idea — rewarding the agent for visiting uncertain regions — is accessible.

Authors’ abstract

Reinforcement learning algorithms are typically designed for discrete-time dynamics, even though the underlying real-world control systems are often continuous in time. In this paper, we study the problem of continuous-time reinforcement learning, where the unknown system dynamics are represented using nonlinear ordinary differential equations (ODEs). We leverage probabilistic models, such as Gaussian processes and Bayesian neural networks, to learn an uncertainty-aware model of the underlying ODE. Our algorithm, COMBRL, greedily maximizes a weighted sum of the extrinsic reward and model epistemic uncertainty. This yields a scalable and sample-efficient approach to continuous-time model-based RL. We show that COMBRL achieves sublinear regret in the reward-driven setting, and in the unsupervised RL setting (i.e., without extrinsic rewards), we provide a sample complexity bound. In our experiments, we evaluate COMBRL in both standard and unsupervised RL settings and demonstrate that it scales better, is more sample-efficient than prior methods, and outperforms baselines across several deep RL tasks.