RGMP: Recurrent Geometric-prior Multimodal Policy for Generalizable Humanoid Robot Manipulation

AI summary

Overview

Research area: Robotics — specifically humanoid robot manipulation, combining vision-language models (VLMs), imitation learning, and generative action modeling.

Technical level: Advanced (assumes familiarity with imitation learning, attention mechanisms, Gaussian Mixture Models, and vision-language models).

Scope: This paper introduces RGMP, an end-to-end framework that unifies geometric-semantic skill selection with data-efficient visuomotor control to let humanoid and dual-arm robots manipulate objects in unseen scenes from very few demonstrations.

What This Paper Is About

Humanoid robots currently depend on huge training datasets to handle multimodal decision-making and generalize to new objects or scenes. Existing methods also neglect geometric reasoning — the shape, position, and spatial affordances of objects — when deciding which manipulation skill to use (e.g., grasping versus pinching). The authors propose RGMP, which couples a geometry-aware skill selector with a recursive, Gaussian-based motion synthesizer so a robot can pick the right skill and execute it accurately using only 40 demonstration samples.

Key Contributions

1. Geometric-prior Skill Selector (GSS): A vision-language model augmented with low-rank geometric adapters and roughly 20 rule-based geometric constraints. It decomposes scenes into object-shape and relative-position priors, then dispatches parameterized skills from a pretrained library without task-specific fine-tuning.
2. Adaptive Recursive Gaussian Network (ARGN): A plug-and-play visuomotor policy that uses Rotary Position Embedding (RoPE), recursive Spatial Mixing Blocks with an Adaptive Decay Mechanism, and a Gaussian Mixture encoder (6 components for a 6-DoF arm) to model directional spatial dependencies and factorize trajectories into full-covariance Gaussians.
3. Real-robot validation across two platforms: Evaluation on both a humanoid robot and a desktop dual-arm robot, demonstrating cross-embodiment generalization and 87% task success in generalization tests.
4. 5× data efficiency gain: RGMP reaches 0.98 success with 40 training samples — matching Diffusion Policy's performance, which requires 200 samples.

Main Findings

• Strong generalization from minimal data: Trained on only 40 Fanta-can grasping demonstrations, RGMP successfully grasped a Coke can, a spray bottle, and a human hand at randomized positions — objects never seen during training.
• 87% average success rate: RGMP achieved 0.98 (Fanta), 0.78 (Coke), 0.81 (spray), and 0.90 (human hands), beating Dex-VLA (0.77 avg), Diffusion Policy (0.70), ManiSkill2-1st (0.64), Octo (0.60), OpenVLA (0.62), and RDT-1b (0.64).
• GSS improves skill selection: Compared to vanilla Qwen-VL, GSS raised skill-selection accuracy (Acc_s) by 15–25% across all tested objects and backbones.
• GMM improves execution precision: For Diffusion Policy, adding GMM raised final accuracy from 0.49 to 0.55 on squashed Coke; ARGN with GSS+GMM reached 0.69 versus 0.60 without GMM.
• All ARGN components matter: Ablation shows RoPE, Spatial Mixing Blocks, and Channel Mixing Blocks combined yield the best accuracy (0.98/0.78/0.81/0.90), with each component contributing measurably.
• Cross-task transfer: On five ManiSkill2 simulator tasks (e.g., plugging chargers via pinch, opening cabinets via grasp), RGMP outperformed all baselines, showing that atomic primitives compose into complex non-grasp behaviors.

Methodology in Plain English

RGMP splits the problem into "which skill should I use?" and "how do I move my arm?"

Skill selection (GSS): When the robot hears a command, a VLM identifies and draws a bounding box around the target object. A segmentation model then extracts the object's shape, and the system combines that with the object's relative position in the scene. Using ~20 hand-coded geometric rules plus shape/affordance commonsense, the VLM outputs an executable skill (grasp, pinch, lift-up) from a pretrained library. Because the geometric priors are modular and minimal, the selector adapts to unseen scenes without retraining.

Motion synthesis (ARGN): Instead of reconstructing 3D geometry explicitly, the policy learns spatial relationships implicitly from RGB images. The image is split into 16×16 patches. A recursive "spatial memory" accumulates relationships from the first patch to the last, with an adaptive decay mechanism that controls how much past memory persists — this prevents vanishing gradients while keeping task-relevant patches weighted heavily. RoPE encodes positional/directional information. Multi-scale features are fused, then a Gaussian Mixture Model with 6 components (one per joint) captures distinct motion modes. At inference, the initial prediction is matched to the nearest Gaussian cluster via Mahalanobis distance, producing the final action.

Training and evaluation: Data consists of (joint space, RGB observation) pairs. The network minimizes MSE between predicted and ground-truth actions. Evaluation counts success when the robot both picks the correct skill and executes it accurately, formalized as Acc = Acc_s × Acc_t.

Why This Matters

Impact on research: The work challenges the assumption that generalizable robot manipulation requires massive datasets. By combining neuro-symbolic geometric reasoning with recursive Gaussian modeling, it shows that carefully designed inductive biases can substitute for scale — a direction that could reshape how the field approaches embodied AI and vision-language-action policies. The 5× data efficiency gain is particularly relevant given that real robot data collection is slow, expensive, and labor-intensive.

Real-world applications:

• Household assistance: Fetching and delivering everyday items (bottles, tissues, cans) in cluttered, changing homes.
• Elder care and service robotics: Handing objects safely to humans (validated on human hand grasping) — a critical capability for assistive robots.
• Warehouse and logistics: Grasping diverse, irregularly shaped packages without extensive per-object training.
• Manufacturing and inspection: Manipulating deformed or atypical components (the paper tests squashed Coke cans as a stand-in for irregular objects).
• Cross-embodiment deployment: The same trained model transfers to a desktop dual-arm platform, enabling manufacturers to reuse policies across robot form factors.

Industry relevance: Reduced data requirements translate directly to lower deployment costs and faster time-to-market for commercial robots. The plug-and-play design of both GSS and ARGN means companies can swap in their own skill libraries or cameras without retraining the full stack.

Future Directions

• Functional generalization: The authors propose inferring trajectories for new objects after teaching just one primary object function, eliminating exhaustive per-object demonstration.
• Scaling beyond three primitives: Extending the skill library to more fine-grained manipulation primitives (e.g., twisting, pushing, tool use) and validating whether geometric priors scale beyond basic shape heuristics.
• Reducing reliance on hand-coded rules: The GSS uses ~20 rule-based geometric constraints; whether these can be learned or meta-learned without sacrificing data efficiency remains open.
• Longer-horizon and dynamic tasks: Current evaluations focus on short manipulation episodes; extending to multi-step tasks in non-static environments is untested.
• Broader cross-embodiment validation: Testing whether the same policy transfers to morphologically very different platforms (legged, aerial, or gripper-diverse arms).

Target Audience

Robotics and embodied AI researchers working on manipulation, imitation learning, or vision-language-action models. Industry engineers building humanoid or dual-arm systems who care about data efficiency and real-world deployment. Graduate students in robotics, computer vision, or multimodal learning who want a concrete example of combining geometric priors with generative action models. Readers without a background in imitation learning, attention mechanisms, or probabilistic modeling will need to consult the referenced background material (Diffusion Policy, RoPE, GMMs) to follow the methodology fully.

Authors’ abstract

Humanoid robots exhibit significant potential in executing diverse human-level skills. However, current research predominantly relies on data-driven approaches that necessitate extensive training datasets to achieve robust multimodal decision-making capabilities and generalizable visuomotor control. These methods raise concerns due to the neglect of geometric reasoning in unseen scenarios and the inefficient modeling of robot-target relationships within the training data, resulting in significant waste of training resources. To address these limitations, we present the Recurrent Geometric-prior Multimodal Policy (RGMP), an end-to-end framework that unifies geometric-semantic skill reasoning with data-efficient visuomotor control. For perception capabilities, we propose the Geometric-prior Skill Selector, which infuses geometric inductive biases into a vision language model, producing adaptive skill sequences for unseen scenes with minimal spatial common sense tuning. To achieve data-efficient robotic motion synthesis, we introduce the Adaptive Recursive Gaussian Network, which parameterizes robot-object interactions as a compact hierarchy of Gaussian processes that recursively encode multi-scale spatial relationships, yielding dexterous, data-efficient motion synthesis even from sparse demonstrations. Evaluated on both our humanoid robot and desktop dual-arm robot, the RGMP framework achieves 87% task success in generalization tests and exhibits 5x greater data efficiency than the state-of-the-art model. This performance underscores its superior cross-domain generalization, enabled by geometric-semantic reasoning and recursive-Gaussion adaptation.