Online Optimization for Offline Safe Reinforcement Learning

AI summary

Overview

• Research area: Offline safe reinforcement learning (OSRL), constrained Markov decision processes (CMDPs), minimax/duality-based optimization, and online learning (no-regret algorithms, multi-armed bandits).
• Technical level: Advanced. The paper combines a minimax formulation with convergence guarantees (Theorems 1 and 2) and a large empirical study on a standardized safe-RL benchmark.
• Scope: The paper proposes O3SRL, a framework that turns offline safe RL into a minimax problem solved by repeatedly calling an offline RL algorithm on cost-adjusted rewards while a bandit algorithm adapts the Lagrange multiplier, and evaluates a practical approximation on the DSRL Bullet benchmark.

What This Paper Is About

Offline safe reinforcement learning asks an agent to learn, from a fixed logged dataset alone, a policy that maximizes reward while keeping its cumulative cost below a budget. Existing methods struggle because Lagrangian relaxation approaches are unstable or learn near-zero-reward policies, while the strongest method for tight budgets (FISOR) trades away reward. The authors reframe the problem as a minimax objective, prove that combining an offline RL oracle with a no-regret algorithm converges to an approximate minimax equilibrium, and then build a practical, OPE-free version that can wrap around any offline RL algorithm.

Key Contributions

1. A minimax formulation of OSRL with convergence guarantees. The constrained problem is written as a min-max Lagrangian game over policy distributions and the Lagrange multiplier, and the authors prove that iterating an offline RL oracle with a no-regret update yields an ε-approximate equilibrium with ε = ε_offline-RL(n) + R_T(Λ)/T (Theorem 1).
2. A practical algorithm that avoids off-policy evaluation and expensive oracle calls. The continuous Lagrange variable is discretized into K arms and optimized with the EXP3 multi-armed bandit algorithm (Algorithm 2), yielding an error of order ε_offline-RL(n) + sqrt(K/T) + 1/K (Theorem 2), with K = Θ(T^(1/3)) giving 1/sqrt(n) + 1/T^(1/3).
3. A plug-and-play implementation. The final approximate algorithm performs only M truncated gradient updates per round, initializing from the previous round, returns the last-iterate policy instead of storing an averaged policy distribution, and works with any offline RL algorithm (TD3+BC is used for the main results).
4. Empirical validation on the DSRL Bullet benchmark. Eight continuous control tasks across Run and Circle objectives are used to test arm count, tight budgets, higher budgets, and compatibility with different offline RL algorithms.

Main Findings

• Two arms are already highly effective. The simplest configuration (K = 2) satisfies the cost constraint on every reported task and keeps average cost at or near zero, though it "tends to underperform slightly in terms of rewards." On BallRun the K = 2 configuration reaches reward 0.20 with cost 0.00; on CarRun, 0.96 with cost 0.01.
• Five arms give the best trade-off. With K = 5, O3SRL either matches or exceeds the highest feasible reward across all tasks while keeping costs below the threshold: BallRun 0.25 reward / 0.00 cost, CarRun 0.96 / 0.02, AntRun 0.33 / 0.14, BallCircle 0.62 / 0.06, CarCircle 0.66 / 0.11, AntCircle 0.48 / 0.00. This is why K = 5 was adopted for all main results.
• Returns diminish beyond five arms. Going from K = 2 to K = 5 brings clear gains, but increasing K to 10 "yields only marginal improvements." With K = 10 the reported rewards are 0.27 (BallRun), 0.96 (CarRun), 0.29 (AntRun), 0.64 (BallCircle), 0.67 (CarCircle), 0.49 (AntCircle).
• Strong cost control under a stringent budget. Main results are reported at a cost threshold of κ = 5. Under the DSRL protocol a policy is safe when normalized cost C_π/κ ≤ 1. O3SRL reports raw costs of 0.00 (BallRun), 0.02 (CarRun), 0.68 (DroneRun), 0.14 (AntRun), 0.06 (BallCircle), 0.11 (CarCircle) and 0.23 (DroneCircle), all well inside the budget, while several baselines exceed κ = 5 (for example, BEAR-Lag reports costs of 18.60 on BallRun and 30.17 on CarRun, and CCAC reports 24.57 on CarRun while also reporting the highest reward there, 1.82 ± 0.84).
• Competitive rewards against eight baselines. Compared with BC-Safe, BEAR-Lag, CPQ, COptiDICE, CDT, CCAC, CAPS and FISOR, O3SRL delivers rewards such as 0.25 ± 0.03 (BallRun), 0.96 ± 0.01 (CarRun), 0.62 ± 0.01 (BallCircle) and 0.66 ± 0.03 (CarCircle), placing it among the safe agents with the highest reward on several tasks.
• Robust across offline RL algorithms and higher cost limits. The introduction states that O3SRL performs effectively with different offline RL algorithms and with higher cost limits; the corresponding per-setting numbers are not reproduced in the portion of the paper available here.
• Safety-cost trade-off is reported qualitatively for capstone baselines. The authors contrast their approach with prior work by noting that Lagrangian constrained policy optimization methods can oscillate, diverge, or learn overly conservative near-zero-reward policies, and that FISOR produces low reward when targeting tight constraints.

Methodology in Plain English

The starting point is a standard trick: a constrained problem can be turned into an unconstrained one by subtracting λ times the cost violation from the reward, where λ is the price the algorithm is willing to pay for safety. Instead of fixing λ, the authors treat it as a second player in a game. One player chooses the policy distribution to maximize the Lagrangian; the other chooses λ to minimize it. If both players use good strategies, the pair converges to a saddle point that is the constrained optimum.

The general recipe alternates two steps. In step one, the offline safe dataset (state, action, reward, cost, next state) is rewritten into an ordinary offline RL dataset by replacing the reward with r − λ(c − (1 − γ)κ), and an offline RL oracle is called on it to produce a new policy distribution. In step two, a no-regret algorithm inspects that distribution and updates λ. Averaging the policies and the λ values across rounds gives the final answer, and Theorem 1 says this averaged pair is an ε-approximate equilibrium.

Two things break in practice. Running offline RL to convergence every round is expensive, and optimizing λ over a continuous range would require off-policy evaluation at every iteration — a procedure the authors call unstable and prone to compounding error. The practical fix discretizes λ into K values (Λ = {λ^(1), …, λ^(K)} with λ^(1) = 0 and a spacing of C/K) and treats each value as an arm of a multi-armed bandit, running EXP3, which updates a probability distribution over arms without needing value estimates. This removes the OPE requirement entirely. The final algorithm goes further: it runs only M gradient updates from the previous round's policy rather than training to convergence, and it returns the last-iterate policy rather than storing every policy it produced, which would be prohibitively memory intensive.

For the experiments, the authors use the DSRL Bullet benchmark with eight continuous control tasks (four "Run", four "Circle", spanning Ball, Car, Ant and Drone agents), κ = 5 for the main results, Λ = [0, C = 5], K = 5 arms, T = 100,000 iterations, and M = 10 gradient updates between arm-probability updates, with TD3+BC as the base offline RL algorithm. Every result is averaged over three random seeds with each policy evaluated across twenty episodes, and normalized reward and normalized cost follow the DSRL protocol.

Why This Matters

The work targets a gap the authors describe as both important and severely under-studied: offline safe RL with small cost thresholds, where an agent must be extremely frugal with its safety budget while still being useful. By removing off-policy evaluation and the need to train an offline RL algorithm to convergence inside every iteration, it makes a theoretically grounded constrained-RL method practical with an ordinary offline RL algorithm plugged in.

Real-world applications named or implied in the paper:

• Healthcare, where exploratory or unsafe actions are not acceptable and data is fixed.
• Smart grid / energy system control, cited as a safety-critical domain with constrained decision-making.
• Industrial process control, listed alongside energy systems as an application with tight safety budgets.
• Autonomous driving and robotics, named as domains where offline RL succeeds but where cost constraints must also hold.

Industry relevance: the practical algorithm is plug-and-play over existing offline RL implementations, requires no environment interaction, and reports near-zero cost on the DSRL benchmark under a stringent κ = 5 budget — properties that matter for deployment pipelines where violating a constraint is far costlier than missing some reward.

Future Directions

• Closing the theoretical gap with continuous λ. The paper notes that a no-regret algorithm such as Follow-The-Regularized-Leader over continuous λ would give a rate of 1/sqrt(n) + 1/sqrt(T), whereas the discretized approach with K = Θ(T^(1/3)) gives 1/sqrt(n) + 1/T^(1/3). Recovering the faster rate without reintroducing unstable OPE is an open problem.
• Better multi-round schedulers. The current design discretizes λ into K arms and truncates offline RL to M updates per round; the paper reports diminishing returns past K = 5, leaving room for smarter allocation of rounds, arms, and gradient steps.
• Reproducing the generality claims beyond the main configuration. The introduction claims effectiveness with different offline RL algorithms beyond TD3+BC and at higher cost limits, and CAPS is instantiated with IQL in the evaluation; systematic study across base algorithms and budget levels is a natural extension.
• Extending beyond the DSRL Bullet suite. Results here come from eight continuous control tasks in the Run and Circle families, so applicability to other safety-critical domains such as healthcare, smart grid and industrial process control remains to be demonstrated.

Target Audience

Researchers and practitioners working on offline reinforcement learning, safe/constrained RL, and constrained MDPs who want a method that is both theoretically grounded and compatible with existing offline RL codebases. It will also interest readers of online learning and bandit literature, since the core mechanism is a no-regret update over a discretized dual variable. Readers need comfort with Lagrangian duality, minimax equilibria, and offline RL evaluation protocols to get the most out of it, though the practical algorithm itself can be applied fairly directly.

Authors’ abstract

We study the problem of Offline Safe Reinforcement Learning (OSRL), where the goal is to learn a reward-maximizing policy from fixed data under a cumulative cost constraint. We propose a novel OSRL approach that frames the problem as a minimax objective and solves it by combining offline RL with online optimization algorithms. We prove the approximate optimality of this approach when integrated with an approximate offline RL oracle and no-regret online optimization. We also present a practical approximation that can be combined with any offline RL algorithm, eliminating the need for offline policy evaluation. Empirical results on the DSRL benchmark demonstrate that our method reliably enforces safety constraints under stringent cost budgets, while achieving high rewards. The code is available at https://github.com/yassineCh/O3SRL.