Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts

AI summary

Overview

Research area: Offline reinforcement learning (RL), specifically cross-domain offline RL with robustness to changing environment dynamics (also touching on robust RL and Wasserstein-distributionally-robust optimization).

Technical level: Advanced. The paper combines a distributionally robust Bellman operator, contraction proofs, Wasserstein dual reformulation, and deep offline RL experiments on MuJoCo/D4RL benchmarks.

Scope: The paper proposes the Robust Cross-Domain Bellman (RCB) operator, proves it delivers both train-time and test-time robustness to dynamics shifts, and turns it into a practical algorithm called DROCO, which is evaluated on 16 target-domain datasets and three types of dynamics perturbations.

What This Paper Is About

Cross-domain offline RL tries to help a data-poor target domain by borrowing a large dataset from a related source domain whose physics differ. Existing methods only handle the source–target mismatch at training time, and they ignore that the environment itself may change when the policy is actually deployed. This paper studies both kinds of robustness at once — "dual robustness" — and asks whether a single algorithmic change can make a policy reliable both during training against source–target mismatch and during deployment against fresh dynamics perturbations.

Key Contributions

1. Empirically demonstrates a gap. The authors show that a cross-domain offline RL policy (IGDF) trained on full data loses 40.9% of its performance under medium kinematic perturbations and 72.4% under hard ones relative to a clean environment, and that this fragility worsens when only 10% of the target dataset is used.
2. Introduces the Robust Cross-Domain Bellman (RCB) operator. The operator applies standard in-sample Bellman backup on target-domain transitions and an in-sample robust Bellman backup on source-domain transitions, using a Wasserstein uncertainty set. It is proven to be a γ-contraction, and a "practical RCB operator" that replaces the intractable dynamics uncertainty set with a state uncertainty set is also proven to be a γ-contraction.
3. Proves dual robustness theoretically. Proposition 4.4 bounds the learned value on source data between Qμ̂(s,a) − 2γεK_Q/(1−γ) and Qμ̂(s,a), preventing overestimation from out-of-distribution dynamics; Proposition 4.5 shows that under perturbations within a Wasserstein radius c, the policy's perturbed value exceeds the worst-case learned value, giving test-time robustness. The two robustness notions trade off through the single parameter ε.
4. Produces the practical DROCO algorithm. Two fixes — a dynamic value penalty (scaled by β) computed from an ensemble dynamics model, and a Huber loss in place of the ℓ2 Bellman loss — address the value overestimation/underestimation that the RCB operator can otherwise induce. Code is released at https://github.com/zq2r/DROCO.git.

Main Findings

• DROCO leads on train-time kinematic shifts. Across 16 target datasets it scores a total normalized return of 1105.2, versus 969.8 for the second-best method OTDF — a 14.0% margin. It is best on 9 of the 16 tasks. Other baselines' totals: IGDF 964.3, IQL* 925.4, DARA 923.0, CQL* 789.9, BOSA 774.5.
• Reduced target data worsens fragility. Shrinking the target dataset to 10% of hopper-expert-v2 (with full source data) increased performance degradation at every shift level, which the authors attribute to a larger discrepancy between true and dataset-observed dynamics causing overfitting to dataset dynamics.
• DROCO holds up better under kinematic perturbations. With halfcheetah-kinematic-expert as the source dataset, easy-level kinematic shifts caused only a 19.3% drop for DROCO (67.4 to 54.4), while both IGDF and OTDF lost over 50%.
• Morphology perturbations are harder for DROCO. Under easy-level morphology shifts DROCO dropped 42.1%, which the authors attribute to morphology shifts never appearing in the source data. It still beat the baselines, which dropped 62.4% (OTDF) and 78.9% (IGDF).
• Stable against adversarial min-Q perturbations. At the largest perturbation scale tested (0.2), DROCO's performance fell 37.9%, compared with 73.6% for OTDF and 84.0% for IGDF.
• Robustness comes at a cost on some tasks. The authors explain DROCO's suboptimal results on the remaining datasets as the price of its performance-vs-robustness trade-off, noting it stays competitive with the baselines there.
• Value-penalty connection to prior work. With IQL-style policy optimization, the paper's penalty term reduces to a form resembling the value-discrepancy term in VGDF, but with an infimum over ensemble predictions instead of an expectation, and with added penalties aimed at test-time rather than only train-time shifts.

Methodology in Plain English

The authors start from a standard robust RL idea: when you do a Bellman update, plug in a worst-case transition from a small ball of possible transitions around the observed one, so the value estimate cannot be fooled by optimistic dynamics. They apply this worst-case treatment only to source-domain transitions, and use ordinary in-sample updates for target-domain transitions. The central technical step is a dual reformulation: the intractable "worst-case transition" becomes a tractable "worst-case nearby state," so the perturbation ball is over states rather than full dynamics.

That state-ball version is the practical RCB operator, but it still needs a way to pick the ball. DROCO trains an ensemble of dynamics models on the target dataset by maximum likelihood and uses the set of their next-state predictions as a stand-in sample from the uncertainty set — each ensemble member supplies one candidate next state, and the minimum Q over them is taken. Because this ensemble set will not fully cover the true target dynamics, the authors prove (Proposition 4.6) that only bounded overestimation can occur under a small total-variation prediction error, and they add two safeguards anyway: a tunable penalty β that scales the gap between the observed next-state value and the worst-case ensemble value (β = 1.0 recovers the plain operator, β > 1.0 suppresses overestimation, β < 1.0 reduces underestimation), and a Huber loss on source-domain Bellman errors so large errors bend toward ℓ1 behavior. Target-domain updates keep the ordinary squared loss. Policy optimization then proceeds with a standard offline RL learner such as IQL.

Experiments use four MuJoCo source tasks (halfcheetah-v2, hopper-v2, walker2d-v2, ant-v3), four D4RL data qualities per task (medium, medium-replay, medium-expert, expert) giving 16 target datasets, and 32 source datasets from combining 4 tasks × 2 shift types × 4 data qualities. Each run uses 1M training steps and 5 random seeds, with main evaluation in the clean target environment and separate evaluations under kinematic, morphology, and min-Q perturbations at easy, medium, and hard levels.

Why This Matters

Research impact. The paper reframes cross-domain offline RL as a problem with two distinct robustness requirements and shows a single operator can serve both, with a provable trade-off knob (ε). It also connects to the distributionally robust RL literature, which the authors note has largely been developed for single-domain settings.

Real-world applications.

• Robotic manipulation policies trained on a mix of real-robot data and imperfect simulator data, where the hardware degrades over time after deployment.
• Industrial control or process optimization where the physical system drifts from the conditions captured in logged data.
• Autonomous driving or navigation policies deployed in weather, terrain, or traffic conditions that differ from the logged target-domain trajectories.
• Any setting where collecting target-domain data is expensive, so a small target set must be augmented by a larger but physically different source set.

Industry relevance. The method relies on data that is usually already available (an ensemble dynamics model and offline datasets) and only adds a penalty term and a loss change, so it is a drop-in modification to existing offline RL pipelines. The reported 14.0% total-score improvement over the strongest cross-domain baseline and the reduced degradation under perturbations are the kind of margins that matter when deployment risk is the main blocker to shipping learned controllers.

Future Directions

• The paper reports that DROCO still suffers 42.1% degradation under easy morphology shifts that never appeared in the source data, raising the question of how to cover perturbation types the source dataset never exhibits.
• The trade-off between train-time and test-time robustness is controlled by ε, but the paper leaves the question of how to select ε in practice — and the relationship between ε, the ensemble size N, and the β penalty — as an open design choice.
• The parameter sensitivity study sweeps β over {0.1, 0.5, 1.0, 1.2}; the results for the Huber transition threshold δ and the rest of that analysis are not reported in the content provided.
• Extending the dual-robustness analysis beyond the tabular contraction guarantees and the Lipschitz-Q assumption to function-approximation settings with formal guarantees would strengthen the theory.

Target Audience

Researchers and graduate students working on offline RL, robust RL, or transfer across simulators and real systems; practitioners who already use offline RL methods such as IQL or CQL and need policies that survive environment drift after deployment; and readers interested in distributionally robust optimization applied to sequential decision making.

Authors’ abstract

Single-domain offline reinforcement learning (RL) often suffers from limited data coverage, while cross-domain offline RL handles this issue by leveraging additional data from other domains with dynamics shifts. However, existing studies primarily focus on train-time robustness (handling dynamics shifts from training data), neglecting the test-time robustness against dynamics perturbations when deployed in practical scenarios. In this paper, we investigate dual (both train-time and test-time) robustness against dynamics shifts in cross-domain offline RL. We first empirically show that the policy trained with cross-domain offline RL exhibits fragility under dynamics perturbations during evaluation, particularly when target domain data is limited. To address this, we introduce a novel robust cross-domain Bellman (RCB) operator, which enhances test-time robustness against dynamics perturbations while staying conservative to the out-of-distribution dynamics transitions, thus guaranteeing the train-time robustness. To further counteract potential value overestimation or underestimation caused by the RCB operator, we introduce two techniques, the dynamic value penalty and the Huber loss, into our framework, resulting in the practical \textbf{D}ual-\textbf{RO}bust \textbf{C}ross-domain \textbf{O}ffline RL (DROCO) algorithm. Extensive empirical results across various dynamics shift scenarios show that DROCO outperforms strong baselines and exhibits enhanced robustness to dynamics perturbations.