MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulation

AI summary

Overview

Research area: Robotics — automated demonstration generation for imitation learning, applied to multi-step bimanual mobile manipulation (a robot with a movable base, a torso, and two arms).

Technical level: Advanced. The paper builds on motion planning, inverse kinematics, constrained optimization, and visuomotor behavior cloning, and assumes familiarity with the X-Gen family of data generation methods.

Scope: MoMaGen is a simulation-based framework that turns a single human teleoperated demonstration into a large dataset of valid robot trajectories by treating data generation as a constrained optimization problem with hard and soft constraints.

What This Paper Is About

Teleoperating a robot with a mobile base and two high-degree-of-freedom arms is a severe burden on human operators, so human demonstrations for these tasks are expensive to collect. Prior automated data generation methods (the X-Gen family) can multiply a few demonstrations for fixed-base, table-top manipulation, but they fail on mobile manipulators: replaying a source demo's navigation leads to base poses from which the randomized objects are unreachable, and the movable camera may lose sight of the task-relevant objects. MoMaGen's goal is to generate large, diverse, valid datasets for such robots from just one source demonstration.

Key Contributions

1. A unified formulation of automated demonstration generation as a constrained optimization problem, where hard constraints must be strictly satisfied (task success, kinematic feasibility, collision avoidance, visibility during manipulation) and soft constraints are costs to be minimized (visibility during navigation, retraction of torso and arms). The authors show that prior X-Gen methods can be expressed in this same framework using different, and insufficient, constraint sets (Table 1).
2. Novel constraints specific to mobile manipulation: reachability as a hard constraint (sampled base poses must keep required end-effector trajectories within reach), object visibility during manipulation as a hard constraint, object visibility during navigation as a soft constraint, and retraction as a soft constraint.
3. A generation algorithm (Algorithm 1) that, per subtask, verifies grasp state, transforms end-effector poses into the new object frame, samples base and head camera poses until reachability and visibility are satisfied, plans base/torso and arm motion with cuRobo, and replays contact-rich segments in task space. It jointly considers end-effector pose, head camera pose, and base pose, and speeds up generation by prioritizing fast inverse-kinematics checks over full motion planning and decomposing the robot configuration into torso and arm subspaces.
4. Empirical evaluation on four multi-step bimanual mobile manipulation tasks with a single source demo, covering data diversity, generation success, visibility, policy learning, cross-embodiment generation, failure analysis, and real-world deployment.

Main Findings

• Broader data diversity: For the Tidy Table task, MoMaGen covers a much wider range of object poses than SkillMimicGen, and MoMaGen is the only method that can generate data under D1 randomization at all; baselines succeed only under D0. Even under identical object pose distributions (D0), sampling feasible base placements gives MoMaGen far greater action diversity in base, end-effector, and joint space (Figure 4, 50 trajectories, subsampled).
• Generation success: MoMaGen achieves a 63% average data generation success rate for D0 and can generate data for all tasks at all three randomization levels. Table 2 D0 success rates for MoMaGen are 0.86 (Pick Cup), 0.80 (Tidy Table), 0.38 (Put Dishes Away), and 0.51 (Clean Frying Pan), versus SkillMimicGen at 1.00, 0.69, 0.38, 0.40 and DexMimicGen at 1.00, 0.72, 0.38, 0.35. Baselines are omitted for D1/D2 because they reach zero success rates.
• Visibility: MoMaGen achieves over 75% task-relevant object visibility even under D1 and D2, and Table 3 shows it "often double" the visibility of baselines and ablations, especially for multi-step tasks. Removing both hard and soft visibility constraints drops D0 visibility to 0.90, 0.46, 0.40, and 0.35 across the four tasks.
• Both constraint types matter for visibility: The Tidy Table analysis (Figure 5, 1000 trajectories) shows that hard and soft visibility constraints are each essential; the ablated variants perform worse than the full method.
• Better policies from generated data: With 1000 generated demonstrations, MoMaGen matches baselines on Pick Cup (D0), where the 0.3m × 0.3m randomization is small enough for replayed navigation, but significantly outperforms them on Tidy Table (D0). On Pick Cup (D1) with a 1.3m × 0.8m randomization, only MoMaGen data lets WB-VIMA reach a 0.25 success rate, while baselines trained on D0 data fail completely.
• Works across imitation learning algorithms: Fine-tuning π0 (LoRA rank 32) on MoMaGen data for Pick Cup (D0/D1) and Tidy Table (D0) with 1000 demonstrations achieves success rates comparable to WB-VIMA, so the data benefits both from-scratch and pretrained-plus-fine-tuned policies.
• Visibility ablations hurt policy performance: With WB-VIMA on Pick Cup (D0), ablation variants reach 0.45 to 0.65 success, below MoMaGen's 0.75. On Tidy Table (D0), ablations peak at 0.05 while MoMaGen reaches 0.40. Success rates are averaged over 20 unseen evaluation episodes.
• Data scaling: π0 fine-tuned with 500, 1000, and 2000 demonstrations for 50K steps shows promising scaling trends, particularly under D1 randomization.
• Real-world transfer: For Pick Cup on the same robot platform in a similar setup (white table, green cup), 40 real-world demos plus pretraining on 1,000 synthetic demos gave WB-VIMA a 10% success rate versus 0% when trained on real data alone, and gave π0 60% success versus 0% for the baseline.
• Cross-embodiment generation: A single source demo from a Galexea R1 robot was used to generate Pick Cup demos on a TIAGo robot, which the authors attribute to planning and replaying dense end-effector trajectories in task space.
• Failure breakdown: Simulation instabilities account for 35% of failures across all three randomizations. Arm-level motion planning contributes the largest share of planner-related failures (40% on average), exceeding base-level planning failures (26% on average). In D2, navigation-related failures increase significantly due to floor obstacles. (No value is reported for the remaining share.)
• Compute cost: Simulation execution dominates wall-clock time. Base motion planning averages 18 seconds, whereas executing the resulting motion in simulation takes 100 seconds. Base sampling is highly variable because it currently samples randomly within a ring-shaped region around the target.

Methodology in Plain English

The researchers start from one human teleoperated demonstration per task, which they segment into ordered subtasks (single-arm or bimanual) and annotate with the target object, the held object, the timestep before contact, the end timestep, and the retraction type. Each subtask is classified as free-space (moving through open space) or contact-rich (manipulating objects through contact).

To create a new demonstration, they randomize the scene and transform the source end-effector poses into the new object's frame of reference, so contact-rich motion is preserved relative to the object. For each subtask they then search for a robot base and head camera pose that satisfies hard constraints: the target must be reachable by the arm, and the target must be visible without occlusion. The search uses inverse kinematics as a quick filter before running full motion planning, and decomposes the robot configuration into torso and arm subspaces to keep sampling efficient. Once a valid pose is found, the robot plans a base and torso trajectory to it, trying to keep the target object in view along the way (a soft constraint), plans an arm trajectory to the pregrasp pose, replays the contact-rich segment in task space, and finally retracts into a compact configuration. Motion planning and inverse kinematics are handled by cuRobo. Because generation uses privileged information such as ground-truth object poses, a separate visuomotor policy must still be trained on onboard RGB images for real deployment.

Why This Matters

Impact on research. The paper provides a single formulation that subsumes the X-Gen line of work as special cases with weaker constraint sets, giving the field a principled way to reason about what a data generation method enforces and where it will fail. It is, per the authors, the first work to tackle automated data generation for bimanual mobile manipulation, and it demonstrates that a single source demonstration plus 40 real-world demonstrations can produce deployable policies.

Real-world applications.

• Household chores such as moving cups, stacking plates onto shelves, and scrubbing a pan, which the paper's four tasks directly represent.
• Any setting where an assistive or service robot must navigate to a surface and then manipulate objects (kitchens, labs, offices).
• Industrial and warehouse tasks that combine long-range base travel with contact-rich arm work.
• Teleoperation-heavy domains where human demonstration time is the bottleneck, since the method reduces required source demos to one and real-world fine-tuning to 40.

Industry relevance. The work lowers the cost of building robot learning datasets for mobile manipulators, which is currently a major scaling bottleneck. It also shows cross-embodiment transfer from a Galexea R1 to a TIAGo, suggesting generated datasets may be reusable across hardware. The stated dependence on sizable GPU resources for GPU-accelerated motion generation is a practical constraint for anyone adopting the pipeline.

Future Directions

• Removing the reliance on full scene knowledge during generation, for example by estimating object poses relative to the robot with vision models such as SAM2.
• Extending the framework beyond alternating phases of navigation and manipulation to whole-body manipulation tasks such as opening doors, which the authors leave for future work.
• Reducing base sampling variance with more informed sampling strategies, for instance biasing toward regions with more free space, to cut the computational cost of generation.
• Improving controller robustness and simulation fidelity to reduce the 35% share of failures attributed to simulation instabilities, and further reducing arm-level motion planning failures, which account for 40% of planner-related failures on average.
• Closing the sim-to-real gap, since pretrained policies on real hardware reached 10% (WB-VIMA) and 60% (π0) success on Pick Cup after fine-tuning on 40 real demonstrations.

Target Audience

Robotics researchers and engineers working on imitation learning, robot data generation, motion planning, or mobile manipulation who need a way to scale demonstration datasets cheaply. It is most useful to readers who already know behavior cloning and motion planning terminology; the constrained-optimization framing and the comparison table of prior X-Gen methods make it valuable for practitioners deciding which data generation approach to build on, and for students studying how to formalize augmentation as an optimization problem.

Authors’ abstract

Imitation learning from large-scale, diverse human demonstrations has been shown to be effective for training robots, but collecting such data is costly and time-consuming. This challenge intensifies for multi-step bimanual mobile manipulation, where humans must teleoperate both the mobile base and two high-DoF arms. Prior X-Gen works have developed automated data generation frameworks for static (bimanual) manipulation tasks, augmenting a few human demos in simulation with novel scene configurations to synthesize large-scale datasets. However, prior works fall short for bimanual mobile manipulation tasks for two major reasons: 1) a mobile base introduces the problem of how to place the robot base to enable downstream manipulation (reachability) and 2) an active camera introduces the problem of how to position the camera to generate data for a visuomotor policy (visibility). To address these challenges, MoMaGen formulates data generation as a constrained optimization problem that satisfies hard constraints (e.g., reachability) while balancing soft constraints (e.g., visibility while navigation). This formulation generalizes across most existing automated data generation approaches and offers a principled foundation for developing future methods. We evaluate on four multi-step bimanual mobile manipulation tasks and find that MoMaGen enables the generation of much more diverse datasets than previous methods. As a result of the dataset diversity, we also show that the data generated by MoMaGen can be used to train successful imitation learning policies using a single source demo. Furthermore, the trained policy can be fine-tuned with a very small amount of real-world data (40 demos) to be succesfully deployed on real robotic hardware. More details are on our project page: momagen.github.io.