Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets

AI summary

Overview

Research area: Robotics and reinforcement learning, specifically at the intersection of offline reinforcement learning (offline RL) and cross-embodiment learning for robot foundation models.

Technical level: Advanced. The paper assumes familiarity with offline RL algorithms (IQL, TD3+BC), advantage-weighted regression, gradient-conflict analysis, transformer-style architectures (URMA), and graph-distance metrics (Fused Gromov–Wasserstein).

Scope in one sentence: The paper empirically studies what happens when offline RL is used to pre-train a single policy on pooled heterogeneous locomotion data from 16 robot platforms, identifies inter-robot gradient conflict as the main failure mode, and proposes a morphology-based grouping strategy to fix it.

What This Paper Is About

Robot foundation models are usually pre-trained by imitation learning, which requires expensive, high-quality teleoperated demonstrations for every robot. This paper asks whether offline RL — which can also learn from suboptimal data — can be combined with cross-embodiment learning, so that one policy is trained on pooled trajectories from many different robot morphologies. The authors build a new benchmark to measure when this combination helps, when it hurts, and how to fix the harm.

Key Contributions

1. A new benchmark. The authors introduce and analyze a benchmark that combines offline RL with cross-embodiment learning across up to 16 distinct robot platforms (nine quadrupeds, six bipeds, one hexapod), built in MuJoCo with six 1M-step dataset variants per robot.
2. Demonstration that offline RL beats behavior cloning when data is suboptimal. In the cross-embodiment setting, offline RL (IQL) outperforms BC on datasets dominated by suboptimal trajectories, and cross-embodiment pre-training accelerates downstream fine-tuning on unseen robots.
3. Diagnosis of gradient conflict as the cause of negative transfer. The authors show positive transfer among morphologically similar robots, but that increasing suboptimal-data ratios and increasing robot diversity amplify inter-robot gradient conflicts, causing performance degradation for some embodiments.
4. Embodiment Grouping (EG). A simple, static grouping strategy that encodes each robot as a morphology graph, computes pairwise Fused Gromov–Wasserstein (FGW) distances, clusters robots by that distance, and performs sequential group-wise actor updates, reporting gains up to 39.8% improvement under high suboptimal-data conditions.

Main Findings

• Offline RL matches BC on expert data, exceeds it on suboptimal data. In Table 1, BC and IQL are nearly equal on Expert Forward (63.31 ± 0.10 vs. 63.39 ± 0.05) and Expert Backward (67.17 ± 0.01 vs. 67.10 ± 0.01), but IQL leads on Expert Replay Forward (54.61 vs. 49.71), Expert Replay Backward (51.86 vs. 42.87), and 70% Suboptimal Forward (36.62 vs. 30.52). The overall mean favors IQL, 52.05 vs. 49.17. BC leads on 70% Suboptimal Backward (41.42 vs. 38.69).

• Cross-embodiment pre-training accelerates fine-tuning. In a leave-one-out experiment, the model pre-trained on cross-embodiment data converged markedly faster than a model trained without cross-embodiment pre-training, illustrated for the quadruped Badger and the bipeds Unitree G1 and Cassie.

• Positive transfer is concentrated among quadrupeds. On the 70% Suboptimal Replay Forward dataset, Unitree A1 (14.55 single vs. 27.38 cross-embodiment), Unitree Go1 (14.46 vs. 40.05), Unitree Go2 (13.76 vs. 52.39), and Badger (15.98 vs. 40.53) improved substantially, likely because quadrupeds contribute the largest share of data.

• Negative transfer hits low-resource morphologies. Unitree H1 fell from 54.47 (single) to 6.00 (cross-embodiment) and Unitree G1 from 78.93 to 0.86 on the 70% Suboptimal dataset. Overall on that dataset, the cross-embodiment mean (37.57) fell below the single-robot mean (39.42).

• More suboptimal data and more robot types both increase gradient conflict. The fraction of negative pairwise actor-gradient cosine similarities grows as the suboptimal-data proportion increases (Expert Forward vs. 30% and 70% Suboptimal Replay Forward), and also as the robot set expands from the similar group (Unitree A1, Go1, Go2) to all nine quadrupeds to all 16 robots.

• Transfer gain correlates with gradient alignment. Across robots with absolute transfer-gain magnitude greater than 10, the correlation between transfer gain and average gradient cosine similarity is r = 0.815.

• Morphology distance predicts gradient alignment. Using FGW distance between robot morphology graphs, the Pearson correlation between embodiment similarity and mean gradient cosine similarity is r = 0.63 (p = 1.26 × 10⁻¹⁴).

• Embodiment Grouping delivers the best average performance. In Table 3 over six datasets and eight methods, IQL + EG (ours) achieves the best mean of 57.29, ahead of IQL + SEL (55.07), IQL + PCGrad (53.48), BC + EG (53.42), IQL (52.05), BC (49.17), TD3+BC + EG (51.39), and TD3+BC (46.43). On the 70% Suboptimal datasets, average improvement over the IQL cross-embodiment baseline is 7.15% for PCGrad, 18.33% for SEL, and 33.99% for EG.

• EG transfers across backbones. On the 70% Suboptimal datasets, EG applied to TD3+BC improves performance by +19.5% on average (raising the overall mean from 46.43 to 51.39), and BC+EG improves BC by +26.3% on the same splits.

• Coarse heuristics fail where morphology-distance grouping succeeds. In the grouping ablation on 70% Suboptimal Forward, IQL baseline scored 37.57 ± 0.78, random grouping 38.73 ± 2.03 (+3.08%), an intuitive biped/quadruped/hexapod/torso-less-biped split 34.45 ± 1.97 (−8.31%), and EG 51.98 ± 1.70 (+38.34%).

• Few groups are enough. Sweeping the group count M over {1, 2, 4, 7, 10, 13}, performance peaked at small-to-moderate M (best score at M = 7 on 70% Forward), while wall-clock training time grew substantially with M. The paper suggests M = 2 to 4 already yields strong gains.

• Gains are not just extra updates. Under a compute-normalized comparison that matches total optimizer steps (multiplying the IQL baseline's K by M) and reduces batch size by 1/M to keep processed samples constant, normalized IQL scored 44.20 ± 2.22 and IQL + EG scored 51.98 ± 1.70, a ΔR of +7.78 on 70% Suboptimal Forward.

Methodology in Plain English

The authors built a simulation benchmark in MuJoCo using walking tasks, covering 16 robots (nine quadrupeds, six bipeds, one hexapod). Each robot's controller was trained with PPO, and the interaction records were saved as datasets of 1M steps each in six flavors: pure Expert data, Expert Replay data (all interaction steps up to expert-level performance, roughly 500M steps, uniformly subsampled to 1M), and 70% Suboptimal Replay data (700k steps from early suboptimal training mixed with 300k expert-like steps). Each of the three quality levels comes in a Forward variant (walk at 1 m/s) and a Backward variant (walk at −1 m/s).

To train one policy across robots with different numbers of joints and feet, they used the URMA architecture, which splits each observation into a shared general part and robot-specific sets of joint and foot observations, aggregates these with descriptor-conditioned attention, and conditions the policy on a morphology descriptor instead of a one-hot robot ID. They extended URMA with a Q-function so that offline RL (specifically IQL) could be applied on top of the encoder.

They then compared behavior cloning against IQL, ran leave-one-robot-out pre-training/fine-tuning experiments, and measured pairwise cosine similarity between per-robot actor gradients (from advantage-weighted regression) to quantify conflict. To connect conflict to morphology, they represented each robot as a graph whose nodes are the torso, joints, and feet, with node features capturing relative positions and control parameters, and computed pairwise Fused Gromov–Wasserstein distances between graphs. Hierarchical clustering on that distance matrix produces fixed robot groups, and training then alternates one global critic update with sequential group-wise actor updates.

Why This Matters

Impact on research. The paper opens a largely unexplored combination — offline RL plus cross-embodiment pre-training — which matters because robot demonstration data are scarce and expensive while suboptimal interaction data are comparatively abundant. It also reframes negative transfer in multi-robot training as a measurable, morphology-structured gradient-conflict phenomenon, giving the field a diagnostic tool (pairwise gradient cosine similarity, FGW-based morphology similarity) rather than only anecdotal evidence.

Real-world applications:

• Fleet-scale robot deployment. A manufacturer with several robot models could pre-train a shared policy from mixed-quality logs and adapt it to a new model with modest additional data, instead of collecting expert teleoperation for each platform.
• Leveraging existing teleoperation and failure logs. Companies already hold large archives of imperfect rollouts; offline RL plus EG gives a principled way to extract value from them rather than discarding them.
• Low-data morphologies. Robots with few similar counterparts in a dataset (the paper's bipeds) are precisely where the method aims to prevent catastrophic degradation, which matters for niche or newly launched hardware.
• Locomotion stack pre-training. Legged-robot vendors can use grouped pre-training as an initialization step before task-specific or real-robot fine-tuning.

Industry relevance. The proposed grouping is static and cheap — clustering is done once before training, and small group counts (M = 2 to 4) are recommended — so it is straightforward to bolt onto existing offline RL pipelines without a large compute penalty.

Future Directions

• Sim-to-real and broader domains. The evaluation is restricted to MuJoCo locomotion; sim-to-real transfer, manipulation, and mobile manipulation remain untested.
• Dynamic rather than static grouping. The current grouping is fixed and derived from morphology graphs via FGW distances, and may not adapt when learning dynamics or data quality change, as in offline-to-online RL.
• Group-specific representations. Once compatible groups are identified, learning group-specific embodiment or task representations inside each group — for example with contrastive objectives — could strengthen within-group sharing.
• Validating group structure against alternatives. The paper notes that coarse leg-count heuristics actually hurt performance, which raises the question of how grouping criteria should be designed as robot fleets and task distributions grow.

Target Audience

Graduate students and researchers working on robot learning, offline RL, and robot foundation models; engineers at robotics companies building shared policy stacks for multiple hardware platforms; and readers already familiar with behavior cloning, IQL, and transformer-based policy architectures who want a systematic account of what breaks when heterogeneous robot data is pooled.

Authors’ abstract

Scalable robot policy pre-training has been hindered by the high cost of collecting high-quality demonstrations for each platform. In this study, we address this issue by uniting offline reinforcement learning (offline RL) with cross-embodiment learning. Offline RL leverages both expert and abundant suboptimal data, and cross-embodiment learning aggregates heterogeneous robot trajectories across diverse morphologies to acquire universal control priors. We perform a systematic analysis of this offline RL and cross-embodiment paradigm, providing a principled understanding of its strengths and limitations. To evaluate this offline RL and cross-embodiment paradigm, we construct a suite of locomotion datasets spanning 16 distinct robot platforms. Our experiments confirm that this combined approach excels at pre-training with datasets rich in suboptimal trajectories, outperforming pure behavior cloning. However, as the proportion of suboptimal data and the number of robot types increase, we observe that conflicting gradients across morphologies begin to impede learning. To mitigate this, we introduce an embodiment-based grouping strategy in which robots are clustered by morphological similarity and the model is updated with a group gradient. This simple, static grouping substantially reduces inter-robot conflicts and outperforms existing conflict-resolution methods.