Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed

AI summary

Overview

Research area: Safe reinforcement learning (RL), out-of-distribution (OOD) safety generalization, and clinical decision support for diabetes management.

Technical level: Advanced. The paper assumes familiarity with constrained Markov decision processes, safe RL baselines, neural ODEs, and clinical glucose metrics.

Scope: The paper benchmarks eight safe RL algorithms under physiological distribution shift in a unified diabetes simulator, documents a "safety generalization gap," and evaluates a test-time predictive shield built on a new Basis-Adaptive Neural ODE (BA-NODE) dynamics model.

What This Paper Is About

Safe RL algorithms enforce safety constraints during training on a fixed environment, but real deployment conditions differ from training. The authors ask whether training-time safety guarantees actually transfer to unseen patients, using diabetes management as a safety-critical testbed where glucose-insulin dynamics vary substantially across individuals. Their goal is not a new safe RL algorithm but a rigorous framework — a unified clinical simulator, an OOD safety benchmark, and a runtime shielding mechanism — for studying and mitigating safety failures under distribution shift.

Key Contributions

1. Unified clinical simulator. A simulator supporting Type 1 and Type 2 diabetes with pump and non-pump therapies, modeling therapeutic decision support rather than direct control. It captures latent patient variability and partial adherence, and builds on the UVA–Padova family of mechanistic glucose-insulin models with a hybrid T2D formulation combining Hovorka secretion dynamics and Dalla Man transport models.

2. OOD safety benchmark. A reusable testbed that evaluates eight safe RL algorithms under physiological distribution shift (parameter generalization to unseen patients; horizon generalization from 1-day training episodes to 7 simulated days), isolating the gap between training-time constraint satisfaction and test-time safety.

3. Test-time predictive shielding. An algorithm-agnostic runtime safety wrapper that intercepts policy actions, forecasts their glucose trajectories, and masks actions predicted to cause hypo- or hyperglycemia, with a probabilistic safety bound formalized as one-sided (ε, α)-reliability.

4. Basis-Adaptive Neural ODE (BA-NODE). A personalized dynamics predictor combining ITransformer variate-aware history encoding, a latent neural ODE ensemble integrated with an RK4 solver, and function-space conditioning that adapts to individual patients through context-derived mixing weights.

Main Findings

• A safety generalization gap exists across all eight safe RL baselines. Policies achieving high Time-in-Range (TIR) and low Risk Index on the training patient degrade substantially on unseen patients. CPO drops from 87.28 ± 2.22 TIR and 4.09 ± 0.65 Risk Index in-distribution to 76.73 ± 3.98 and 5.72 ± 1.02 out-of-distribution (ΔTIR −10.55, ΔRisk +1.62). CUP drops from 89.36 ± 3.12 to 77.70 ± 4.90 TIR (ΔTIR −11.66), and FOCOPS from 88.08 ± 0.44 to 75.42 ± 0.38 (ΔTIR −12.66). Despite all algorithms being trained toward Risk Index < 5 and TIR > 80%, deployment under shift produces TIR reductions of more than 10%.

• PCPO is the exception in the unshielded benchmark, in the opposite direction. PCPO records 36.74 ± 1.84 TIR and 20.34 ± 0.33 Risk Index on the training patient and 38.75 ± 0.52 TIR and 19.97 ± 0.33 Risk Index on unseen patients (ΔTIR +2.01, ΔRisk −0.37), reflecting poor base performance rather than genuine safety generalization.

• BA-NODE predicts glucose more accurately than baselines over a 24-step (120-minute) horizon. BA-NODE achieves MAE 2.82 ± 0.01, FDE 3.63 ± 0.03, and RMSE 4.42 ± 0.02, compared with NODE (MAE 3.18 ± 0.60, FDE 4.11 ± 1.23, RMSE 5.10 ± 1.45) and ITransformer (MAE 4.11 ± 0.10, FDE 5.39 ± 0.16, RMSE 7.25 ± 0.29). BA-NODE also shows the lowest variance across metrics.

• Predictive shielding improves safety across algorithms and patient populations. Across eight safe RL algorithms, three diabetes types, and three age groups, shielding achieves TIR gains of 13–14% for strong baselines such as PPO-Lag and CPO while reducing Risk Index and glucose variability. Across 72 experimental settings, shielding consistently recovers safety margins.

• Type 1 diabetes results. Shielding produces an average ΔTIR of +6.08% across algorithms, and CV is reduced for all agents (FOCOPS achieves ΔCV −4.26%). PPO-Lag improves TIR by +8.05, and RCPO by +6.90. CUP and PCPO show slight TIR reductions (−0.22 and −0.14) but still improve CV (−0.72 and −3.58).

• Type 2 diabetes results. Benefits are larger: PPO-Lag improves TIR by +14.15% and CPO by +13.54%. CPO sees the largest CV improvement (−6.68%), followed by PPO-Lag (−5.50%).

• Type 2 diabetes without pump results. Gains are less consistent. PPO-Lag improves TIR by +4.46% but increases CV by +4.39%, a pattern shared with RCPO and TRPO-Lag. CRPO improves TIR by +11.80% while reducing CV by −7.28%.

• Predictive shielding generally outperforms a static rule-based shield. The rule-based shield, which mimics Low Glucose Suspend systems using fixed glucose and insulin-on-board thresholds, often degrades performance relative to the unshielded policy, producing feedback loops of aggressive bolus correction, hypoglycemia overshoot, and compensatory meal recommendations.

• Failure mode: weak base policies. When the underlying controller performs poorly — for example, PCPO with TIR below 40% and Risk Index around 20 — the rule-based shield can outperform the predictive shield, because the predictive shield can only reweight the base policy's action distribution rather than override it entirely.

Methodology in Plain English

The authors build a simulator of glucose-insulin dynamics covering three clinical settings: Type 1 diabetes with pump therapy, Type 2 diabetes with pump therapy, and Type 2 diabetes without pump therapy. Rather than letting an agent directly control basal insulin, the simulator models a recommender: the agent proposes discrete bolus insulin and meal interventions, and a patient acceptance model with safety gates decides what is actually executed. Basal insulin is fixed at patient initialization and calibrated using weight-based dosing. The observation space is 14-dimensional and includes continuous glucose monitor readings, insulin-on-board, and meal history; patient physiological parameters affecting sensitivity, absorption, and clearance remain unobserved by the agent. Reward and cost are computed from predicted glucose trajectories to capture delayed effects such as insulin stacking and rebound hyperglycemia, with asymmetric penalties that punish hypoglycemia more strongly than hyperglycemia.

To test generalization, the authors train one policy per condition and age group (Child, Adolescent, Adult), giving 9 base policies per algorithm, each trained on a single representative patient (Child#01, Adolescent#01, Adult#01). They then evaluate zero-shot on 9 unseen patients (Patients #02–#10) per diabetes type and extend episodes from 1 training day to 7 simulated days.

For the shield, BA-NODE learns a continuous-time latent state governed by a neural ODE. It encodes history with an ITransformer that treats each physiological variate (glucose, insulin, carbohydrates) as a token, initializes a latent state from that encoding, rolls out K candidate vector fields through an RK4 solver, and combines them via a learned projection. Patient-specific mixing weights over these basis trajectories are computed at inference by regularized least squares on a small set of context windows, and future glucose is predicted by accumulating the weighted increments from the last observed value.

At runtime, the shield intercepts the policy's top-k bolus candidates paired with all discrete meal levels, uses BA-NODE to forecast glucose over a finite horizon, and applies finite logit bonuses and penalties to the policy logits before a softmax. Three conditions apply: critical rescue when blood glucose is below 60 mg/dL (force rescue carbohydrates such as 15g and block insulin), predictive safety when glucose is at or above 80 mg/dL and candidate actions forecast violations, and minimal intervention otherwise. The shield is disabled in the transition zone between 60 and 80 mg/dL to avoid over-reacting to noise. Clinical failure limits are 70 mg/dL for hypoglycemia and 180 mg/dL for hyperglycemia, and the shield thresholds are set more conservatively to absorb prediction error.

Why This Matters

Impact on research. The paper reframes safety in safe RL as a generalization problem rather than a training-optimization problem. It shows that satisfying constraints in expectation on a fixed training distribution does not imply test-time safety, and it provides an open benchmark and simulator (GlucoSim and GlucoAlg) so that others can study OOD safety generalization in safety-critical control. The distinction it draws — medical shifts are latent and structural (unobservable metabolic traits) rather than observable parameters as in robotics — highlights a setting where retraining by trial and error is ethically prohibited.

Real-world applications:

• Automated insulin delivery and bolus recommender systems for people with Type 1 diabetes on pump therapy.
• Decision support for Type 2 diabetes patients on pump therapy, where insulin resistance and residual secretion complicate dosing.
• Bolus-only correction support for Type 2 diabetes patients without pumps, who rely on residual endogenous secretion.
• General safety certification of learned controllers in other latent-shift domains, such as other medical dosing or care settings, where a runtime wrapper must enforce safety without retraining.

Industry relevance. Diabetes device and digital health companies — one author is affiliated with Dexcom — have a direct interest in runtime safety layers that can wrap an existing controller without retraining it. The predictive shield is algorithm-agnostic and adds a wrapper around pre-trained policies, which maps onto regulatory and product needs for verifiable, deployable safety checks, and the reported reductions in glucose variability alongside TIR gains address clinical outcomes beyond threshold compliance alone.

Future Directions

• Improving shields for weak base controllers. The predictive shield is bounded by the probability mass of the base policy's action distribution, so it underperforms a static rule-based shield when the controller is already poor (PCPO, CUP). Extending the shield to override rather than reweight could address this.
• Closing the gap for non-pump T2D. Shielding produced inconsistent results in the T2D-without-pump cohort, with PPO-Lag, RCPO, and TRPO-Lag showing TIR gains but worse CV. Making the mechanism reliably improve both safety and stability in this regime remains open.
• Tightening the formal guarantee. Theorem 5.2 conditions hypoglycemia safety on a predictor being (ε, α)-reliable; the paper states that the shield thresholds are set more conservatively than clinical failure limits to absorb prediction error, leaving open how to verify or guarantee that reliability assumption under shift.
• Broader physiological and behavioral coverage. The authors explicitly name latent patient variability and partial adherence as modeled sources of deployment mismatch; extending the benchmark to other sources of shift and other safety-critical medical control settings is a natural next step.

Target Audience

Researchers and practitioners in safe reinforcement learning and constrained MDPs who care about out-of-distribution safety rather than only reward robustness; medical control and clinical decision support researchers, particularly those working on diabetes technology; and engineers at device or digital health companies who need to add a safety layer around a pre-trained controller. The paper is also relevant to those working on runtime verification, shielding, and neural ODE-based forecasting, since it combines those threads in a single benchmark.

Authors’ abstract

Safe Reinforcement Learning (RL) algorithms are typically evaluated under fixed training conditions. We investigate whether training-time safety guarantees transfer to deployment under distribution shift, using diabetes management as a safety-critical testbed. We benchmark safe RL algorithms on a unified clinical simulator and reveal a safety generalization gap: policies satisfying constraints during training frequently violate safety requirements on unseen patients. We demonstrate that test-time shielding, which filters unsafe actions using learned dynamics models, effectively restores safety across algorithms and patient populations. Across eight safe RL algorithms, three diabetes types, and three age groups, shielding achieves Time-in-Range gains of 13--14\% for strong baselines such as PPO-Lag and CPO while reducing clinical risk index and glucose variability. Our simulator and benchmark provide a platform for studying safety under distribution shift in safety-critical control domains. Code is available at https://github.com/safe-autonomy-lab/GlucoSim and https://github.com/safe-autonomy-lab/GlucoAlg.