MobileManiBench: Simplifying Model Verification for Mobile Manipulation

AI summary

Overview

Research area: Robotics — vision-language-action (VLA) models, mobile manipulation, and simulation-based benchmark/dataset construction.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning, diffusion/flow-matching action heads, vision-language backbones, and robot kinematics, though the motivation and headline results are readable without that background.

Scope: The paper introduces MobileManiBench, a simulation-based benchmark built on NVIDIA Isaac Sim in which reinforcement-learning policies generate 300K annotated mobile-manipulation trajectories across 2 robots, 630 objects, 5 skills, and 100 realistic scenes, and uses it to train and compare a universal VLA model (MobileManiVLA) against existing VLA baselines.

What This Paper Is About

Current VLA models for robot manipulation depend heavily on large teleoperated datasets such as openX-Embodiment, which are dominated by static, tabletop scenes with gripper-equipped robots. Changing anything about the hardware — adding depth or wrist cameras, moving to a mobile base, or swapping in a dexterous hand — typically forces recollecting data from scratch at high cost. The authors propose verifying VLA architectures in simulation first, and build MobileManiBench: a benchmark that autonomously generates diverse mobile-manipulation trajectories with language, multi-view RGB-depth-segmentation, robot-state, and action annotations, then benchmarks representative VLA models on it.

Key Contributions

1. A simulation-first pipeline for VLA verification. Built on NVIDIA Isaac Sim and driven by reinforcement learning, the pipeline makes it cheap to scale objects, tasks, and scenes while flexibly changing robot embodiment, camera placement, and sensing modalities before committing to real-world data collection.

2. MobileManiRL, a universal state-based RL policy. Parameterized by keypoint-based displacements of robot gripper/hand points, object grasp points, and goal points, a single policy formulation is trained on each robot-object-skill combination — 1,182 combinations per robot — instead of hand-designing or teleoperating per task.

3. MobileManiDataset and MobileManiBench. A dataset of 300K trajectories with 3 data modalities (language instructions, multi-view RGB-depth-segmentation images, synchronized object/robot states and actions) covering 630 objects in 20 categories, 5 skills, over 100 tasks, 100 scene placements, 2 mobile robots (G1 with a parallel gripper, XHand with a 12-DOF dexterous hand), and 2 synchronized head/wrist cameras.

4. MobileManiVLA and a systematic model study. A 3B universal VLA model built on PaliGemma-2 vision-language components with a diffusion-transformer action module, together with comparisons against OpenVLA, CogACT, π0, and π0.5, plus ablations over visual inputs, state inputs, seen/unseen generalization, and base mobility.

Main Findings

• MobileManiRL reaches high success, MobileManiVLA much lower: On the 5 skills, mean success rates are 89.6% (G1) and 92.9% (XHand) for MobileManiRL, versus 56.7% and 57.3% for MobileManiVLA. Per-skill MobileManiVLA rates are Open 42.9/34.5, Close 75.8/77.5, Pull 34.4/29.4, Push 85.1/90.0, Pick 28.8/40.2 (G1/XHand).

• Object structure and skill complexity drive variance (Takeaway 1): Categories like toilet, laptop, and dishwasher score higher because they mainly require lid-flipping; car, door, and table are harder because they demand precise handle localization and stable grasps. Skills requiring accurate, stable grasping — open, pull, pick — are harder than close and push.

• Generalization is harder for implicit-input VLA models (Takeaway 2): MobileManiRL trains and evaluates per seen robot-object-skill combination with explicit geometric state inputs, while MobileManiVLA is a universal model evaluated on unseen objects and unseen scenes using only language, multi-view RGB-D images, and wrist pose in the mobile base frame.

• Dexterous hands help precision but trade off on some skills (Takeaway 3): XHand beats G1 under MobileManiRL (92.9% vs 89.6%), and under MobileManiVLA the two are comparable (57.3% vs 56.7%), with XHand worse on open and pull (finger collisions with object surfaces) but better on pick (force-closure grasps on holistic objects).

• Multi-view and multi-modality images matter a lot (Takeaway 4): Head-view RGB alone yields 7.9% success; adding head depth gives 14.1%; adding wrist RGB gives 14.9%; combining head and wrist RGB-D reaches 28.2%.

• State inputs add further gains (Takeaway 5): Encoding wrist pose in the mobile base frame raises success from 22.4% to 28.2%; adding pseudo-observations (first-frame grasp-point and goal-point positions in the base frame) pushes it to 32.3% and 36.6%.

• Unseen objects are harder than unseen scenes (Takeaway 6): The model scores 59.6% with both objects and scenes seen, 51.3% with seen objects in unseen scenes, 39.2% with unseen objects in seen scenes, and 28.2% when both are unseen.

• A unified comparison protocol favors multi-modal inputs (Takeaway 7): With default architectures and inputs, OpenVLA (7B) scores 4.5% and CogACT (7B) 6.8% (head RGB only); π0 (3B) and π0.5 (3B) score 11.2% and 18.8% using head and wrist RGB plus wrist pose; MobileManiVLA (3B) reaches 28.2% using head and wrist RGB-D plus wrist pose.

• Base mobility is essential (Takeaway 8): Fixing the G1 base (initialized 0.5 m to 1.0 m from the grasp point) on six combinations — open laptop, open cabinet, open faucet, open table, pull cart, pick holistic YCB objects — drops MobileManiRL success drastically relative to the mobile-base setting (Table 7 reports 25.4% fixed-base versus 82.8% mobile-base).

Methodology in Plain English

The authors deliberately generate data in simulation before touching real hardware, so that any change to sensors, cameras, or robot body can be tested cheaply.

First, they define each manipulation task by three points: a point on the robot's gripper or hand, a point on the object where it should be grasped, and a goal point. A single reinforcement-learning policy, MobileManiRL, learns to move the gripper point to the grasp point and then to the goal point, shaped by a reward that combines a global distance term, an approach term, a grasp bonus, a move term, and a success bonus. The policy is a small multilayer perceptron (four layers with hidden dimensions {1024, 1024, 512, 512} plus a linear action head) that consumes time embedding, object state, robot proprioception, robot-object distances, and the previous action, and outputs a 7-dimensional action for the G1 gripper or an 18-dimensional action for the XHand hand. Each robot–object–skill combination is trained in simple "ground" or "tabletop" scenes with randomized robot starting poses, reaching mean success rates of 89.6% (G1) and 92.9% (XHand).

Second, those trained policies are deployed in 5 types of realistic scenes (space, wall, door, outdoor, tabletop) — 80 seen placements for training and 20 unseen for testing — to roll out successful trajectories. Each trajectory stores one language instruction such as "open faucet", synchronized 520×520 RGB, depth, and segmentation images from head and wrist cameras, object and robot states, and the executed action, at 30 FPS with an average length of 160 frames.

Third, all trajectories are aggregated to train MobileManiVLA, a universal model per robot. It reuses a pretrained vision-language model (PaliGemma-2, with a SigLIP vision encoder and Gemma-2 language model), takes two RGB and two depth images reshaped to 224×224×3 plus a "<skill> <object>" instruction, and feeds a cognition token plus a state feature (a 6-D wrist pose encoded by a small MLP) into a diffusion-transformer action module. Training minimizes mean squared error against ground-truth Gaussian noise; at inference the model predicts action chunks of length 16 using CogACT's adaptive ensemble strategy with window size 4.

Compute and settings: RL training used 1,024 simulation environments, learning rate 1e-3, and 4K iterations on 32 NVIDIA V100 GPUs (about 4 days per robot); trajectory generation used 8 NVIDIA RTX A6000 GPUs (about 6 days per robot); VLA training used batch size 480, learning rate 2e-5, and 320K iterations on 8 NVIDIA B200 GPUs (about 12 days per robot).

Why This Matters

Impact on research. MobileManiBench targets a specific bottleneck in embodied AI: verifying architectural choices is currently gated by expensive teleoperation. By making sensors, camera placement, robot embodiment (parallel gripper versus dexterous hand), and object/scene diversity controllable variables, the benchmark lets researchers test whether a new input modality or design actually helps before investing in real-world collection. The paper reports that its objects come from PartNet-Mobility, UniDoorManip, and YCB, and that a universal RL policy was trained inside the benchmark — two properties the authors highlight as differentiators from prior real-world and simulation benchmarks in their Table 1 comparison.

Real-world applications:

• Warehouse and logistics robots that must navigate to a shelf and then open, pull, or pick items, rather than operating from a fixed tabletop position.
• Service and healthcare robots in environments such as hospitals and offices, where the paper's scenes are drawn from kitchen, bedroom, hospital, office, warehouse, and parking lot settings.
• Household assistants that manipulate articulated objects such as fridges, cabinets, dishwashers, ovens, and doors, plus small everyday objects from the YCB set.
• Dexterous-hand platforms where grasp reliability and finger collisions are the limiting factor, informing hardware and control choices before purchase or deployment.

Industry relevance. The paper releases code, datasets, and models publicly (project website: https://dexhand.github.io/MobileManiBench/), and the authors' affiliations span Microsoft Research Asia, the University of Sydney, and Tsinghua University. For companies building general-purpose manipulation policies, the benchmark offers a standardized training and evaluation protocol across robots, tasks, scenes, and modalities, and the reported baseline numbers (for example, π0.5 at 18.8% versus MobileManiVLA at 28.2% on the challenging-skill subset) give a concrete yardstick for how far current VLA models are from reliable mobile manipulation.

Future Directions

• Closing the simulation-to-policy gap. MobileManiVLA trails MobileManiRL substantially (56.7%/57.3% versus 89.6%/92.9% mean success), so improving how implicit sensory inputs substitute for explicit geometric state is an open problem. The paper states that real-world inference of MobileManiVLA is demonstrated in the Appendix, but real-world success rates and sim-to-real transfer numbers are not reported in the main content.
• Better dexterous-hand policies for grasp-critical skills. XHand's finger collisions with object surfaces hurt open and pull under MobileManiVLA; resolving this collision-aware grasping is a clear next step.
• Improving generalization to unseen object structures. Unseen objects cost more performance than unseen scenes (39.2% versus 51.3% in the reported ablation), suggesting object-level generalization deserves targeted methods.
• Expanding skills, objects, and embodiments. The current benchmark covers 5 skills and 630 objects in 20 categories; scaling to richer task vocabularies, more robots, and additional scene types follows directly from the framework's design goal of making such changes cheap.

Target Audience

Robotics and embodied-AI researchers working on VLA models, manipulation policies, or dexterous and mobile robot platforms; benchmark and dataset builders who need a standardized protocol for comparing architectures and input modalities; and practitioners in industrial or service robotics evaluating whether mobile manipulation or dexterous hands are worth the hardware investment. Readers without a background in reinforcement learning or diffusion policies will find the motivation and headline results accessible, but the method sections are aimed at specialists.

Authors’ abstract

Vision-language-action models have advanced robotic manipulation but remain constrained by reliance on the large, teleoperation-collected datasets dominated by the static, tabletop scenes. We propose a simulation-first framework to verify VLA architectures before real-world deployment and introduce MobileManiBench, a large-scale benchmark for mobile-based robotic manipulation. Built on NVIDIA Isaac Sim and powered by reinforcement learning, our pipeline autonomously generates diverse manipulation trajectories with rich annotations (language instructions, multi-view RGB-depth-segmentation images, synchronized object/robot states and actions). MobileManiBench features 2 mobile platforms (parallel-gripper and dexterous-hand robots), 2 synchronized cameras (head and right wrist), 630 objects in 20 categories, 5 skills (open, close, pull, push, pick) with over 100 tasks performed in 100 realistic scenes, yielding 300K trajectories. This design enables controlled, scalable studies of robot embodiments, sensing modalities, and policy architectures, accelerating research on data efficiency and generalization. We benchmark representative VLA models and report insights into perception, reasoning, and control in complex simulated environments, with all code, datasets, and models publicly released.