DexDICE: Reconciling Safety and Performance via Dual-Expert Offline Imitation Learning

AI summary

Overview

Research area: Offline imitation learning and safe sequential decision-making, with a focus on robotics and autonomous navigation.

Technical level: Advanced. The paper develops a constrained convex optimization formulation over stationary state-action distributions (DICE), a Lagrangian dual derivation, and a high-probability safety theorem. The high-level idea is accessible, but the method section assumes familiarity with f-divergences, Bellman flow constraints, and density-ratio estimation.

Scope (one sentence): The paper introduces a new problem setting — offline imitation learning from two separate experts, one safe but suboptimal and one performant but potentially unsafe — and proposes DexDICE, a DICE-based algorithm that enforces safety through a support constraint rather than a divergence penalty, with a theoretical zero-unsafe-visitation guarantee and evaluations on 38 benchmark tasks plus a real mobile robot.

What This Paper Is About

Imitation learning normally assumes you have demonstrations from a single expert. In safety-critical settings that assumption breaks down: demonstrations that are simultaneously safe and high-performing are expensive or impossible to collect, and specifying an explicit cost function to encode safety is as hard as designing a reward. This paper proposes to instead collect two separate sets of demonstrations — one from a safety-focused expert (safe, but possibly suboptimal) and one from a performance-focused expert (high return, but possibly unsafe) — and learn a policy that behaves like the performance expert wherever that is safe, without ever seeing a reward or cost label.

Key Contributions

1. A new problem formulation: dual-expert offline imitation learning. The learner receives safe demonstrations (𝒟_S), performant demonstrations (𝒟_P), and an unlabeled dataset (𝒟_U) that provides transition coverage. The goal is to match the performant expert's state-action distribution subject to a safe support constraint — the policy may only visit state-action pairs that the safe expert demonstrates — with no explicit reward or cost signals anywhere.

2. The DexDICE algorithm. DexDICE replaces the conventional KL-divergence regularization toward the safe distribution (used by prior DICE-based offline IL methods such as DemoDICE) with an explicit support constraint. The authors state this is the first DICE-based offline IL method to do so. The key claimed benefit is that safety and suboptimality are decoupled: the constraint restricts where the policy may act, not how often, so the policy is no longer pulled toward the safe expert's poor behaviors.

3. A safety guarantee (Theorem 3). For the ideal support-constrained policy, the paper proves that unsafe-region visitation is exactly zero with probability at least 1 − δ, provided the user-chosen false-negative rate κ exceeds a boundary mass κ₀ plus a finite-sample slack of order |𝒟_S|^(−1/2). The guarantee does not require a perfect discriminator — once the discriminator error falls below half the safe margin, unsafe visitation becomes exactly zero.

4. Empirical validation on a re-curated DSRL benchmark and a real robot. Because no public benchmark exists for this new setting, the authors re-curate DSRL (38 tasks across three environment families) by relabeling trajectories along performant/non-performant and safe/unsafe axes. They also deploy the method on a physical mobile robot trained directly from human teleoperation demonstrations.

Main Findings

• Best safety-performance trade-off on the re-curated DSRL benchmark (38 tasks). DexDICE is safe on 17/21 Safety Gymnasium tasks, 7/8 Bullet Safety Gym tasks, and 7/9 MetaDrive tasks, and is the best safe method on 9/21, 5/8, and 5/9 tasks respectively. Averaged normalized return/cost: Safety Gymnasium 0.37/0.71, Bullet Safety Gym 0.59/0.63, MetaDrive 0.50/0.69 (safe means average normalized cost ≤ 1).

• Comparators frequently trade one objective for the other. Plain BC reaches higher average return on Safety Gymnasium (0.46) and MetaDrive (0.76) but violates the constraint (cost 1.99 and 1.68, safe on only 3/21 and 1/9 tasks). DemoDICE-S is safe on 18/21 Safety Gymnasium tasks with cost 0.68 but averages return 0.27 and is the best safe method on only 4/21. JFBC averages return 0.70 on MetaDrive but is safe on 0/9 tasks.

• Single fixed hyperparameter, no per-task tuning. DexDICE uses κ = 0.01 for all tasks, with N_𝒟_S = N_𝒟_P = 10 demonstrations, averaged over 5 seeds and 50 rollouts at the last checkpoint.

• Data efficiency and robustness to data quality. The paper reports that DexDICE is data-efficient with respect to the sizes of 𝒟_S and 𝒟_P, and robust to variations in demonstration quality (the proportion of safe-performant demonstrations), evaluated on controlled datasets built by mixing four demonstration types on Safety Gymnasium.

• Real-world mobile-robot navigation from teleoperation. Trained directly on human teleoperation demonstrations for a physical mobile robot navigation task, the method "substantially outperforms baselines and remains robust to data scarcity and quality degradation." Specific quantitative robot results are not reported in the provided excerpt.

• Acknowledged theory-practice gap. Theorem 3 concerns the ideal solution. The extracted policy is a Gaussian with full support over the action space, so it cannot be hard-supported on the estimated safe set. A deployment-time projection of each proposed action onto the estimated support substantially reduces the measured mismatch but cannot eliminate it — a reached state need not admit any in-support action, and a finite-budget search may fail to find one. The residual mismatch is reported separately (Appendix G.11) and is explicitly not covered by the exact guarantee.

• Support estimation through rank consistency. Rather than inverting the positive-unlabeled mixture (which requires the notoriously hard-to-identify class prior η), the authors exploit the fact that the safe-vs-unlabeled ratio and the safe-vs-unsafe ratio induce the same ordering over state-action pairs, so thresholding the former suffices.

Methodology in Plain English

The setup. You have three piles of data: demonstrations from a cautious expert, demonstrations from an aggressive expert, and a large unlabeled pile that gives broad coverage of what the world looks like. You are not allowed to interact with the environment during training, and you are never given a reward or cost number.

The optimization target. Instead of directly learning a policy, DexDICE borrows the DICE idea: learn the long-run distribution of state-action pairs the policy would visit, and compare distributions rather than policies. The ideal answer is the distribution closest to the aggressive expert's, subject to never putting any probability mass outside the region the cautious expert demonstrated.

The trick that makes it solvable. "Never leave the safe region" is written as a single equality: the sum over all state-action pairs of (an indicator that the pair is safe) times the distribution equals 1. That indicator is the only place safety enters. Combined with the standard Bellman flow constraints (which ensure the learned distribution corresponds to some actual policy) and a nonnegativity constraint, this becomes a constrained convex problem. Introducing Lagrange multipliers — λ for the support constraint and a per-state potential ν(s) for the flow constraints — and a change of variables to a density ratio w = d/d_U gives an objective whose inner maximization has a closed form: w* = exp(A − 1), where A contains the temporal-difference term, a log-ratio term log r_P toward the performant expert, and the term λ·1[safe].

Estimating "safe." Nobody hands you the safe region; it has to be inferred. Two classifiers are trained: one distinguishes 𝒟_S from 𝒟_U (giving r̂_S ≈ d_S/d_U), another distinguishes 𝒟_P from 𝒟_U (giving r̂_P ≈ d_P/d_U). The safe indicator is then r̂_S ≥ τ_κ, where τ_κ is the κ-quantile of r̂_S on the safe demonstrations. This guarantees that fewer than a fraction κ of genuinely safe pairs are wrongly rejected. The authors use a safe-vs-unlabeled discriminator rather than a safe-vs-unsafe one because the unsafe distribution is never directly observed, and they report (Appendix G.7) that this is empirically more robust than estimators using only safe demonstrations.

From distributions back to a policy. Once the optimal weights are found, a policy is extracted by weighted behavioral cloning, following prior DICE work, and at deployment each proposed action is projected onto the estimated safe support.

How it was evaluated. Since the setting is new, the authors adapt ten baselines to matched information access: BC, filtered BC, jointly filtered BC, two DemoDICE variants, DWBC-S, and the offline safe methods FISOR-IL, CDT, and TraC fed binary pseudo-labels instead of ground-truth reward and cost. Return and cost are normalized to [0, 1]; a method counts as safe when its average normalized cost is at most 1.

Why This Matters

Research impact. The paper reframes safety in imitation learning as a support question rather than a divergence question. That shift matters because divergence regularization toward safe data necessarily inherits the safe expert's suboptimality as a bias, whereas a support constraint only restricts the feasible region. The formal zero-unsafe-visitation result, together with the explicit separation between the guarantee for the ideal solution and the behavior of the practically extracted policy, gives the community a cleaner target to build on and a clearly marked gap to close. The paper also supplies a benchmark construction (re-curated DSRL) and a set of matched-information baselines for a setting that previously had none.

Real-world applications:

• Autonomous driving — the paper explicitly notes that driving data naturally contain high-speed but risky maneuvers alongside safe, conservative driving, which is exactly the dual-expert structure.
• Mobile robot navigation — demonstrated directly here, with a physical robot trained from human teleoperation.
• Healthcare and clinical decision support — the paper cites settings where some demonstrations are efficiency-oriented while others explicitly prioritize safety.
• Robotic manipulation and general safety-critical robotics — where hardware wear, limited system access, and costly resets make offline learning the only practical option.

Industry relevance. The method needs no cost-function engineering, which is often the single biggest blocker to deploying constrained RL in practice, and it uses only N_𝒟_S = N_𝒟_P = 10 demonstrations per expert on the benchmark with a single fixed κ = 0.01 across all 38 tasks. For teams that already log separate "careful" and "fast" operating data — a very common situation in vehicle fleets, warehouse robots, and industrial automation — that is an appealingly low bar to entry. The Hyundai Motor Company affiliation among the authors is consistent with this applied motivation.

Future Directions

1. Closing the theory-practice gap. Theorem 3 covers the ideal support-constrained distribution, but the deployed policy is a full-support Gaussian with a projection step whose residual mismatch is explicitly outside the guarantee. Deriving a guarantee that accounts for the weighted-BC extraction and the projection budget is the most direct open problem the paper names.

2. Robustness of the support estimate. The safety argument rests on a bounded discriminator error ε; how that error degrades in high-dimensional state-action spaces, and how sensitive the whole pipeline is to the choice of κ, are natural follow-ups. The paper reports using one fixed κ = 0.01 everywhere, but the theorem's condition κ > κ₀ + O(|𝒟_S|^(−1/2)) suggests κ is a meaningful knob worth characterizing per environment.

3. Weak overlap between the safe and performant supports. The authors note that when supp(d_S) ∩ supp(d_P) is small, the optimum may be far from d_P in divergence — the method still returns the closest safe approximation, but the approximation may be poor. Detecting and handling low-overlap regimes, or actively requesting targeted demonstrations to expand the overlap, is an obvious extension.

4. Extending beyond a two-expert partition. Dual-expert is a clean abstraction, but real data often mixes many operating modes. Generalizing to multiple graded safety levels, or to settings where safety labels are noisy or only partially available, would broaden applicability.

Target Audience

Researchers and practitioners in offline reinforcement learning, imitation learning, and safe control who are working on safety-critical sequential decision-making with limited or expensive data. It is most useful to readers already comfortable with DICE-style stationary-distribution correction or constrained MDPs, since the technical core builds directly on that machinery. Roboticists and applied engineers who have logged separate conservative and aggressive operating data — and who cannot specify a cost function — will find the problem framing and the low demonstration requirement (10 safe and 10 performant demonstrations in the benchmarks) immediately relevant. Readers purely interested in the high-level idea can get most of the value from the introduction, Figure 1, and the experimental tables.

Authors’ abstract

Ensuring high returns while providing strong safety guarantees in offline imitation learning (IL) is a fundamental challenge: existing approaches either rely on explicitly specified cost functions, which are difficult to design for complex real-world safety constraints, or require expert demonstrations that are both safe and high-performing, which are often too costly to collect. We introduce dual-expert imitation learning, which leverages two complementary data sources: (i) safety-compliant demonstrations that faithfully follow safety guidelines but may be suboptimal, and (ii) performance-oriented demonstrations that achieve high returns but may violate safety guidelines. To address this problem, we propose DexDICE (Dual-expert offline imitation learning via stationary DIstribution Correction Estimation). Instead of conventional distribution constraints, DexDICE combines DICE-style offline learning with safe support constraints, enabling agents to exploit high-return behaviors while remaining within safety-compliant regions. We prove that the ideal solution satisfying this constraint exactly has zero unsafe-region visitation with high probability. We evaluate DexDICE on a re-curated DSRL benchmark and validate it on a real-world mobile-robot navigation task trained directly from human teleoperation demonstrations, where it substantially outperforms baselines and remains robust to data scarcity and quality degradation.