DextER: Language-driven Dexterous Grasp Generation with Embodied Reasoning

AI summary

Overview

• Research area: Robotics, specifically language-conditioned dexterous grasp generation with vision-language-action models.
• Technical level: Advanced (assumes familiarity with VLMs, tokenization, autoregressive generation, and grasp simulation benchmarks).
• Scope: A single paper proposing contact prediction as an intermediate "embodied reasoning" step for multi-fingered grasp generation, evaluated on the DexGYS and Dexonomy benchmarks.

What This Paper Is About

Dexterous robot hands with 20+ degrees of freedom must grasp objects in ways that satisfy a task stated in language, such as "grasp the mug by its handle." Existing vision-language approaches map what the camera sees and what the instruction says directly to hand joint values, skipping any explicit reasoning about where the hand physically touches the object. The authors propose DextER, which first predicts which hand links contact which points on the object surface, then generates the grasp, and they show this intermediate step improves both task alignment and grasp stability.

Key Contributions

1. Contact-centric embodied reasoning. Contact-centric reasoning is proposed as an "embodied thinking" process for language-driven dexterous grasp generation, explicitly modeling the physical interaction structure between multi-fingered hands and objects.
2. Large-scale contact and language annotation. The DexGYS and Dexonomy datasets were annotated with physics-based contact data using the MuJoCo simulator, and Dexonomy was additionally annotated with natural language grasp descriptions generated by a vision-language model.
3. An autoregressive framework enabling steerable generation. Users can specify partial contact constraints (a few finger links and their 3D coordinates), and the model completes the remaining sequence while respecting those constraints.
4. Demonstrated gains over state of the art. DextER reports 67.14% grasp success on DexGYS, 3.83 percentage points above the prior state of the art, with 96.4% improvement in intention alignment.

Main Findings

• DexGYS success rate: DextER achieves 67.14% grasp success on the DexGYS validation set, outperforming the previous state of the art (DexGYSNet at 63.31%) by 3.83 percentage points.
• Intention alignment: DextER achieves a P-FID of 0.20, described as a 96.4% improvement over the previous state-of-the-art DexGYSNet, whose P-FID is listed as 5.60.
• Diversity: DextER reports δq = 13.63 versus 6.12 for DexGYSNet, roughly 2× higher diversity, which the authors interpret as broader mode coverage rather than collapse onto dense ground-truth modes. δr is 77.98 and δt is 8.84.
• Embodied reasoning matters: Removing the contact reasoning step (w/o ER) degrades P-FID from 0.20 to 0.30 (a 50% increase), drops success from 67.14% to 62.37%, lowers Q1 from 0.89 to 0.66, and increases penetration from 0.37 to 0.44.
• Token granularity sweet spot: Both action bins and position bins perform best at 256. Action bins at 128 give P-FID 0.21, CD 1.70, Con. 0.35, success 66.19%; at 512 give P-FID 0.26, CD 1.89, Con. 0.37, success 65.24%. Position bins at 128 give P-FID 0.21, CD 1.67, Con. 0.36, success 67.35%; at 512 give P-FID 0.23, CD 1.69, Con. 0.35, success 66.32%.
• Dropout setting: Contact position dropout at p_drop = 0.5 performs best (P-FID 0.20, consistency 0.34). No dropout gives P-FID 0.22 and success 65.68%; p_drop = 0.7 gives P-FID 0.25 and success 66.54%; p_drop = 1.0 gives P-FID 0.30 and success 63.33%, approaching the no-ECoT model.
• Encoder choice matters more than model scale: PartField gives P-FID 0.20, CD 1.46, Con. 0.34, success 67.14%, while Uni3D gives P-FID 0.52, CD 2.24, Con. 0.46, success 59.07%. Scaling the LLM from Qwen2.5-0.5B to Qwen2.5-1.5B yields only modest improvement (success 67.55% vs 67.14%, P-FID 0.18 vs 0.20), and SmolLM2-360M reaches 64.87% success with P-FID 0.31.
• Contact prediction quality: On the DexGYS validation set, predicted contact links score IoU 0.42, Precision 0.59, Recall 0.63, and F1 0.57; Position Accuracy, the share of predicted contact positions within 1 cm of positions from forward kinematics, is 0.79.
• Zero-shot generalization on Dexonomy: DextER outperforms the DexGYS baseline on all four splits. On the Seen Objects & Grasp Taxonomy split it reaches P-FID 0.44, CD 18.32, success 12.24%. On Unseen Objects: 1.44, 20.07, 10.86%. On Unseen Grasp Taxonomy: 1.04, 19.49, 9.10%. On Unseen Both: 1.23, 20.44, 8.41%. The DexGYS baseline reports success rates of 0.97%, 0.93%, 0.99%, and 1.16% respectively.
• Steerable generation: Adding partial contact constraints improves both alignment and quality. With 1 specified link on the Seen split, P-FID is 0.43, CD 5.51, success 10.40%; with 5 links, P-FID falls to 0.12, CD to 0.73, and success rises to 21.35%. On Unseen Both, 1 link gives 1.03, 6.09, 7.23% and 5 links gives 0.78, 1.98, 8.31%.
• Inference speed: Average inference on a single NVIDIA A6000 is 1.458 s for DextER, 1.105 s for DextER without embodied reasoning, and 1.134 s for DexGYSNet — the reasoning step adds modest overhead.
• Robustness: The authors report that DextER maintains stable performance with only approximately 35% of points visible and added sensor noise (detailed in the appendix).
• Generalization pattern: The model generalizes better to unseen objects than to unseen grasp taxonomies; qualitative analysis of steerable generation on unseen taxonomies found grasps that hold the object but exhibit instability such as shaking.

Methodology in Plain English

DextER takes two inputs: a 3D point cloud of the object (sampled to 10,000 points with XYZ coordinates) and a language instruction. A pretrained 3D encoder, PartField, extracts geometric features, which a small 2-layer MLP projects into the language model's embedding space. The language backbone is initialized from Qwen2.5-0.5B and receives visual tokens (768 of them, from downsampled triplane feature maps of size 3 × 16 × 16) concatenated with text.

The model then generates output as text-like tokens in three stages. Meta-prompts such as "Think step by step: first predict which links contact where on the object, then predict the grasp pose" instruct it to reason about contacts first. Contact tokens encode a hand link name (for example thbase, ffdistal, mfmiddle) followed by discretized x, y, z position bins, wrapped in contact start/end delimiters. Action tokens then encode the 6D wrist pose and finger joint angles, binned into 256 bins per dimension after quantile normalization from the 1st to 99th percentile into [-1, 1], wrapped in action start/end delimiters. All required tokens are registered as special tokens in the pretrained tokenizer.

Training uses standard next-token prediction over the full sequence. Point cloud tokens get bidirectional attention for global geometric context, while language and action tokens use causal attention. Contact position dropout removes position tokens while keeping link tokens at probability p_drop, so the model learns to work from varying levels of contact detail. Training runs with batch size 64 for 100K iterations using AdamW, learning rate 1e-4 with cosine decay, bfloat16 mixed precision, and gradient clipping at 1.0, on 8 NVIDIA A6000 GPUs for approximately 48 hours.

Annotations were generated automatically rather than by hand: grasp and object models are loaded into MuJoCo, forward kinematics are executed, and contacts are read from the physics buffer to yield contact anatomy (which links touch) and contact positions (3D locations on the object surface). Dexonomy lacks language descriptions, so five multi-view renderings per grasp are passed to a vision-language model conditioned on the renderings and contact anatomy, which identifies the object category, infers contacted functional parts such as handle or rim, and writes grasp descriptions. Evaluation uses Isaac Gym-based simulation following Grasp-as-You-Say and DexGraspNet for DexGYS, and DexGraspBench, a MuJoCo-based benchmark, for Dexonomy.

Why This Matters

The paper argues that neither two-stage language-conditioned pipelines (separate intention understanding and grasp synthesis modules, trained disjointly) nor end-to-end approaches (direct multimodal-to-grasp mapping without explicit interaction modeling) explicitly capture the physical principle that a grasp depends on where and how the hand contacts the object. Contact prediction is offered as a representation that is simultaneously embodiment-aware, interpretable, and controllable, and the ablation shows it improves both semantic alignment and physical quality rather than trading one for the other.

Real-world applications:

• Task-specific grasping in unstructured environments where robots work alongside humans and grasp intent (pour, open, hand over) determines success.
• Industrial and logistics manipulation requiring precise contact placement on specific object parts.
• Human-in-the-loop robot control, where an operator specifies a few contact constraints via the steerable interface and the model completes the grasp.
• General vision-language-action systems that need higher-dimensional dexterous control than parallel-jaw grippers provide.

Industry relevance: the work targets multi-fingered anthropomorphic hands, aims for an interface that lets users constrain grasps without hand-tuning joint angles, and reports inference times on a single NVIDIA A6000 that keep the reasoning step's overhead modest relative to a comparable baseline.

Future Directions

• Compounding errors: The authors note that their autoregressive framework is susceptible to compounding errors, which diffusion-based approaches might mitigate, but combining diffusion with embodied reasoning remains underexplored.
• Beyond single static objects: Current evaluation focuses on simulation with single, static objects; extending to complex scenes with occlusions would better reflect real-world scenarios.
• Real-time performance: Sequential token prediction may limit real-time performance.
• Novel grasp taxonomies: Steerable generation on unseen grasp taxonomies produced grasps that held the object but were unstable, indicating that learning stable contact patterns for novel manipulation strategies remains challenging.

Target Audience

Robotics and embodied AI researchers working on dexterous manipulation, vision-language-action models, and embodied chain-of-thought reasoning; engineers building language-driven grasping systems for multi-fingered hands; and readers interested in interpretable, steerable interfaces for robot action generation. The paper assumes background in transformer-based multimodal models and grasp simulation, so it is most accessible to readers already familiar with those areas.

Authors’ abstract

Language-driven dexterous grasp generation requires the models to understand task semantics, 3D geometry, and complex hand-object interactions. While vision-language models have been applied to this problem, existing approaches directly map observations to grasp parameters without intermediate reasoning about physical interactions. We present DextER, Dexterous Grasp Generation with Embodied Reasoning, which introduces contact-based embodied reasoning for multi-finger manipulation. Our key insight is that predicting which hand links contact where on the object surface provides an embodiment-aware intermediate representation, bridging task semantics with physical constraints. DextER autoregressively generates embodied contact tokens specifying which finger links contact where on the object surface, followed by grasp tokens encoding the hand configuration. On DexGYS, DextER achieves 67.14% success rate, outperforming state-of-the-art by 3.83 p.p. with 96.4% improvement in intention alignment. We also demonstrate steerable generation through partial contact specification, providing fine-grained control over grasp synthesis.