CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions

AI summary

Overview

Research area: Robot learning / safe reinforcement learning — specifically combining Control Barrier Functions (CBFs) with policy-gradient RL for high-dimensional robot control (humanoid locomotion).

Technical level: Advanced. The paper contains formal definitions, lemmas and proofs (laypunov-style comparison arguments, forward invariance), plus a full RL training pipeline.

Scope: The paper proposes and empirically validates CBF-RL, a training-time framework that combines CBF-based action filtering with a barrier-inspired reward so that a single policy learns safe behavior and can be deployed without any runtime safety filter.

What This Paper Is About

Reinforcement learning policies optimize task performance but can produce unsafe actions, and a single bad action can damage a robot or its surroundings. Existing fixes either run a CBF safety filter online (guaranteeing safety at runtime but leaving the policy unaware of the constraint, so behavior stays conservative and the filter must stay in the loop) or reshape the reward with safety penalties (no direct enforcement, sensitive to penalty weights). CBF-RL aims to get both benefits at once: filter the policy's actions during training and reward it for avoiding filter interventions, so the learned policy internalizes safety and no filter is needed at deployment.

Key Contributions

1. Conceptual: A dual CBF-RL training framework that applies both CBF-based active filtering and a barrier-inspired reward during training, with the resulting policy deployable without a runtime filter.
2. Theoretical: A continuous-to-discrete relationship analysis (Lemma 1, Theorem 1) showing that under stated conditions continuous-time CBFs provide forward-invariance conditions for the discrete-time RL environment, plus a closed-form solution of the single-constraint CBF-QP for lightweight integration.
3. Practical: Empirical demonstration across a 2D single-integrator navigation ablation and simulated and hardware humanoid experiments that dual-trained policies internalize safety and reduce unsafe actions at deployment.
4. Released code: A sample implementation of the single-integrator CBF-RL reward core is linked at https://github.com/lzyang2000/cbf-rl-navigation-demo.

Main Findings

• Closed-form filtering is valid on discrete rollouts: Theorem 1 bounds the barrier value along Euler-discretized dynamics as h(q_k) ≥ (1 − Δt α)^k h(q_0) − μ(Δt)/(Δt α), and since lim_{Δt→0} μ(Δt)/Δt = 0, the standard discrete-time CBF bound h(q_k) ≥ (1 − Δt α)^k h(q_0) is recovered as Δt shrinks. The paper states this means continuous-time CBF tools can be directly applied to discrete RL environments for small enough Δt (typically Δt ≤ 0.01 s).
• Dual training converges faster and stays safe during training: In the 2D navigation ablation, Dual and Filter-only approaches "achieve rapid convergence while remaining safe throughout training," whereas Reward-only and Nominal do not.
• Only dual training transfers to deployment without a filter: Over 1000 random test environments, Dual reached 99.0% success with no domain randomization (DR) and 99.0% with DR (−0%). Dual without a runtime filter reached 92.7% (no DR) and 91.7% (DR, −1%).
• Filter-only behavior collapses without the runtime filter: Filter-only reached 98.8% (no DR) and 96.7% (DR, −2.1%) with an active filter, but only 38.7% (no DR) and 36.8% (DR, −1.9%) without one.
• Reward-only is weaker and less robust: Reward-only reached 91.9% (no DR) and 87.6% (DR, −4.3%) — the largest degradation under domain randomization among the CBF-based variants.
• Nominal RL is far less safe: Nominal reached 51.4% (no DR) and 55.0% (DR, +3.6%).
• Robustness under injected dynamics noise: Training with DR used dynamics q_{k+1} = q_k + (v_k + d)Δt, where d is standard-normal scaled to 20% of maximum velocity. The paper states the dual-trained policy suffers least from this disturbance; the Dual (w/o rt. filt.) degradation was 1% versus 4.3% for Reward-only.
• Humanoid hardware validation: With the dual-trained policy, a Unitree G1 avoids obstacles and climbs stairs in real-world settings without a runtime safety filter, under command sequences that cause failure with nominal policies. Specific quantitative humanoid results (e.g., success rates for obstacle avoidance or stair climbing) are not reported in the available paper content.

Methodology in Plain English

The user defines a barrier function h over a low-dimensional ("reduced-order") representation of the robot — for the 2D navigation example this is the agent's (x, y) position; for the humanoid, planar base velocity relative to the nearest obstacle — along with its gradient. At each training step, the RL policy proposes an action. A safety filter checks whether that action satisfies the CBF condition ∇h(q)^T v ≥ −α h(q); if it does, the action passes through unchanged, and if not, the action is minimally projected back onto the safe set. Because the constraint is a single linear inequality, the projection has a closed-form expression instead of requiring an optimization solver, which matters when training thousands of environments in parallel.

The same calculation produces a reward term: a penalty on the amount of CBF violation (min(a^T v_policy − b, 0)) plus a smooth exponential term, exp(−||v_policy − v_safe||²/σ²) − 1, that pushes the policy to propose actions close to the safe ones. This term is bounded below by −1, so it cannot destabilize the task reward. The total reward is r_nominal + r_cbf, and the filtered action — not the raw policy action — is executed in the environment. Because the policy sees both what it proposed and what the filter did, it receives corrective supervision during training rather than in deployment.

The authors first study this on a 2D single-integrator navigation task with randomized starts, obstacles and goals, running ablations across all combinations of the two components (4 trained variants, 12 tested configurations), 1500 training steps and 4096 parallel environments, with and without domain randomization. They then apply the same recipe in IsaacLab to a Unitree G1 humanoid for planar obstacle avoidance and blind stair climbing, using 4096 environments, Δt = 0.005 s, episodes of at most 20,000 steps on an NVIDIA RTX 4090 GPU, an MLP policy with 3 hidden layers of [512, 256, 128] neurons at 50 Hz outputting joint position setpoints for the 12-DoF lower body, and zero-shot sim-to-real transfer. For blind stair climbing they use an asymmetric actor-critic method and give the critic a 1 m × 1.5 m height scan at 0.1 m resolution with history length 1.

Why This Matters

• Research impact: The paper shows filtering and reward shaping are complementary rather than competing, and provides a continuous-to-discrete safety argument that legitimizes using cheap closed-form continuous-time CBF filters inside discrete RL simulators. This is a step away from the standard "always keep a safety filter in the loop" deployment pattern.
• Real-world applications:
  • Legged and humanoid robots operating near people, where a runtime QP filter may be too slow or unreliable under noisy state estimation.
  • Navigation and mobile manipulation in cluttered or unstructured environments with obstacle-avoidance constraints.
  • Stair climbing and terrain traversal for humanoids, where unsafe actions risk falls and hardware damage.
  • Any compute-constrained embedded deployment where a policy must run without solving an optimization program every control step.
• Industry relevance: The humanoid robotics sector is scaling toward real deployments where safety guarantees and compute budgets are both binding; a method that removes the runtime filter while retaining safety behavior directly addresses deployment cost and reliability. The authors acknowledge support from the Technology Innovation Institute (TII), BP p.l.c., and The Dow Chemical Company (project #227027AW).

Future Directions

• Extend the theory beyond the single-integrator CBF setting: Lemma 1 and Theorem 1 are stated for the continuous-time single integrator q̇ = v with Euler discretization; extending the bound to the more general control-affine form and to control interfaces ψ(x, v) is an open direction the paper gestures at.
• Uncertainty handling without robust CBF machinery: The paper relies on domain randomization rather than disturbance observers, Gaussian Process models, or robustified CBFs; how far DR alone scales to severe state-estimation error is unresolved.
• Barrier design: The approach assumes analytic barrier functions and does not attempt barrier discovery, leaving open how to obtain good reduced-order coordinates and barriers for new tasks and new robots.
• Fuller quantitative reporting on hardware: The available content demonstrates obstacle avoidance and stair climbing on the Unitree G1 but does not report success-rate or convergence numbers for the humanoid tasks, so the magnitude of the safety benefit at full scale remains to be quantified.

Target Audience

Graduate students and researchers in safe control, robot learning, and legged/humanoid locomotion; robotics engineers evaluating whether they can drop a runtime safety filter from a deployed policy; and practitioners who already use CBF-QP filters or barrier-based reward shaping and want a lightweight way to combine them with standard PPO-style training.

Authors’ abstract

Reinforcement learning (RL), while powerful and expressive, can often prioritize performance at the expense of safety. Yet safety violations can lead to catastrophic outcomes in real-world deployments. Control Barrier Functions (CBFs) offer a principled method to enforce dynamic safety -- traditionally deployed online via safety filters. While the result is safe behavior, the fact that the RL policy does not have knowledge of the CBF can lead to conservative behaviors. This paper proposes CBF-RL, a framework for generating safe behaviors with RL by enforcing CBFs in training. CBF-RL has two key attributes: (1) minimally modifying a nominal RL policy to encode safety constraints via a CBF term, (2) and safety filtering of the policy rollouts in training. Theoretically, we prove that continuous-time safety filters can be deployed via closed-form expressions on discrete-time roll-outs. Practically, we demonstrate that CBF-RL internalizes the safety constraints in the learned policy -- both enforcing safer actions and biasing towards safer rewards -- enabling safe deployment without the need for an online safety filter. We validate our framework through ablation studies on navigation tasks and on the Unitree G1 humanoid robot, where CBF-RL enables safer exploration, faster convergence, and robust performance under uncertainty, enabling the humanoid robot to avoid obstacles and climb stairs safely in real-world settings without a runtime safety filter.