CoToGrasp: Contact-Topology-Conditioned Dexterous Grasp Synthesis via Canonical Workspace Learning

AI summary

Overview

Research area: Robotics — dexterous manipulation, task-oriented grasp synthesis, generative modeling for high-DoF robotic hands.

Technical level: Advanced. The paper assumes familiarity with grasp representations (force closure, wrench space), point-cloud learning (DGCNN, Set Transformer), variational autoencoders, and robotic grasp taxonomies.

Scope: A single paper proposing CoToGrasp, an object-agnostic generative framework that synthesizes dexterous grasps conditioned on human grasp taxonomies, evaluated on DexGraspNet and validated on a physical Allegro Hand.

What This Paper Is About

Most dexterous grasp planners optimize only for physical stability — they answer "can this object be held?" but not "how should it be held to serve a task?" When a downstream task requires a specific precision pinch rather than an enveloping power grasp, existing planners must generate and discard enormous numbers of functionally useless candidates. CoToGrasp addresses this by training a generative model that produces grasps strictly conditioned on a requested contact topology (drawn from the Gonzalez human grasp taxonomy), and it does so without ever needing costly datasets that annotate grasp types against specific object meshes.

Key Contributions

1. A contact-topology-conditioned grasp generator. CoToGrasp conditions synthesis on 21 contact templates derived from the Gonzalez human taxonomy, transferred onto a robot hand as semantic masks over 22 anatomical contact zones. The taxonomy acts purely as a labeling interface and can be swapped for any other contact-based representation.

2. A feature-based canonical workspace that decouples semantics from object geometry. Local geometric features from either the gripper (training) or the target object (inference) are extracted with a DGCNN and aggregated onto fixed spatial basis points anchored to the gripper frame via weighted kNN. Because training and inference occur on the same fixed basis, the model learns the gripper's intrinsic contact manifold and generalizes zero-shot to unseen objects.

3. A training set that requires no objects. Pairing 10,000 kinematically valid gripper configurations with all 21 templates yields 210,000 training samples generated purely in the canonical frame — no object meshes, no physics simulation, no human-annotated grasp labels.

4. Novel semantic evaluation metrics plus a cascading validation pipeline. The paper introduces Topology Compliance (TC) and two entropy measures (stability entropy H_SR and semantic entropy H_TC) to quantify functional diversity and mode collapse, and pairs generation with a label-consistency filter, force-closure check with latent resampling, and an energy-based joint optimization that includes a new repulsive term enforcing a 5 mm safety margin on unused fingers.

Main Findings

• Topology-unaware planners suffer severe mode collapse. Baselines (DFC, GenDexGrasp, DRO-Grasp, GOAG) cluster almost entirely around enveloping power grasps, producing low semantic entropy (H_TC between 0.60 and 0.74) and a large "unknown" category of stable-but-functionally-meaningless grasps. CoToGrasp reaches H_TC = 0.83, the highest measured.

• Speed advantage is substantial. CoToGrasp generates a grasp in 0.11 s, compared to 1.72 s for DRO-Grasp, 14.65 s for GenDexGrasp, and over 1800 s for the analytical DFC method.

• A deliberate trade-off on success rate. CoToGrasp's overall SR is 36.94% against 63-78% for unconditioned baselines. The authors attribute this directly to forcing diverse topologies (e.g. precision pinches) onto objects whose geometry does not naturally afford them; the Top-5 average SR rises to 56.18%.

• Superior to the taxonomy-aware baseline. Against Dexonomy on DexGraspNet, CoToGrasp achieves higher average SR (26.72% vs 21.13%), higher topology compliance (17.18% vs 14.28%), and higher semantic entropy (0.84 vs 0.77), with the largest gains on precision grasps.

• Precision grasps are where the gap is widest. For topology M2, CoToGrasp reaches 30.3% SR versus Dexonomy's 10.5%. Dexonomy's apparently inflated scores on M11 (60.5%) and M21 (37.2%) are artifacts of mode collapse — it silently abandons the requested topology and emits an unverified enveloping grasp, which its pipeline records as a success.

• The validation pipeline is load-bearing. Removing the label-consistency check drops TC from 17.18% to 14.45% and SR from 26.72% to 22.97%; removing the force-closure resampling drops TC to 16.26%. Removing both is worse still.

• Physical feasibility confirmed. Grasps were executed on a UR10 arm with an Allegro Right Hand across YCB objects, spanning precision and power categories. Topology M6 was suppressed because the four-fingered Allegro lacks the little finger required for that template.

Methodology in Plain English

The key insight is a symmetry of contact: a point on the gripper touches the object if and only if a point on the object touches the gripper. Existing methods search over the object surface — an unbounded, endlessly varied domain. CoToGrasp instead searches over the gripper's own surface, which is fixed, bounded, and mechanically finite.

Concretely, the object is transformed into the gripper's canonical frame and its surface normals are inverted, turning it into a "negative mold" of the hand's expected contact surface. A point-cloud encoder extracts local geometry features, and those features are aggregated onto a fixed grid of basis points anchored to the palm, weighted by a normal-aware distance so that surface orientation governs contact viability. Because that grid never changes between training and testing, the network sees the same representation whether it is looking at a gripper or an object.

Each populated grid point becomes a token carrying both geometric information and the requested topology embedding. A Transformer encoder models which contact regions co-occur, and a Set Transformer compresses everything into a latent code. A conditional VAE then samples from that code and predicts a per-point contact-zone label across the workspace — essentially a contact mask on the hand. During training this is supervised against templates projected from the taxonomy; at inference, the latent is sampled from a standard Gaussian.

Generated candidates pass through two gates before ever reaching optimization: a label-consistency check that rejects masks deviating by more than one contact zone, and a fast force-closure estimate. If force closure fails, the model simply resamples its latent variable (up to 20 times) to produce a different contact pattern. Survivors go to an energy minimization that pulls the fingers onto the predicted contact zones, penalizes object penetration and self-collision, respects joint limits, and — the new addition — pushes unused fingers and palm areas away from the object by at least 5 mm so they do not accidentally create contacts that violate the requested topology.

Why This Matters

Impact on research. The paper reframes grasp synthesis from a geometry problem to a semantics problem, and shows that the two can be decoupled. Its object-agnostic training regime eliminates the data bottleneck that has constrained taxonomy-conditioned grasping — no object-specific annotations, no simulation rollouts, 210,000 training samples generated in seconds. The new TC and entropy metrics also give the field a way to measure something it previously could not: whether a planner's output distribution is functionally biased.

Real-world applications:

• Humanoid and service robots interpreting high-level instructions from LLM planners, where "hand me the scissors by the handles" requires a specific precision topology rather than any stable grasp.
• Industrial assembly and tool insertion, where a screwdriver or connector must be grasped in a particular way for the downstream motion to succeed.
• In-hand manipulation and finger gaiting, where controllable multi-point contact distributions are a prerequisite for reorienting an object within the hand.
• Handle-based grasping (doors, drawers, pans), where the grip must match the affordance the object offers.

Industry relevance. The 0.11 s generation time makes the method viable inside real-time control loops rather than as an offline planner. Because the pipeline is object-agnostic at training time and the global grasp pose is treated as an external prior rather than inferred end-to-end, it composes cleanly with existing task planners and can support kinematic keyframing for continuous manipulation along a trajectory — an attractive property for system integrators who already have pose-selection logic in place.

Future Directions

1. Autonomous grasp pose selection. The paper deliberately externalizes the 6D pose (R, t) to a heuristic topology-conditioned sampler. End-to-end or learned pose proposal that respects topology constraints would remove the reliance on a two-stage pipeline and likely lift the success rate.

2. Closing the simulation-to-execution gap. Topology compliance is measured before and after physics simulation, and contact drift under dynamic simulation is the main reason absolute TC scores remain modest. Contact patterns that survive real contact dynamics — not just idealized kinematics — remain an open problem.

3. Extending to arbitrary hand morphologies. The taxonomy was transferred onto a four-fingered Allegro by suppressing topologies that require five fingers. A principled mapping of the 22 human contact zones onto non-anthropomorphic or three-fingered hands would broaden applicability.

4. Integration with language and vision models. The paper positions topology conditioning as the bridge between high-level semantic intent and low-level physical interaction, but the actual link from language ("grasp this to pour") to a topology index is left to future work.

Target Audience

Robotics researchers working on dexterous manipulation, grasp synthesis, and humanoid control — particularly those building task-oriented planners or dataset-free generative pipelines. The paper is also relevant to engineers integrating multifingered hands into industrial or service platforms who need grasp generation fast enough for closed-loop execution, and to anyone studying mode collapse and distributional bias in conditional generative models, since the entropy-based evaluation methodology transfers beyond grasping.

Authors’ abstract

Current dexterous grasp planners primarily optimize for physical stability, focusing on whether an object can be grasped rather than how it should be grasped to support downstream functional tasks. However, conditioning grasp synthesis on specific human grasp taxonomies typically requires prohibitively expensive, object-annotated datasets. To address these limitations, we propose CoToGrasp, a novel generative framework that synthesizes diverse, stable grasps strictly conditioned on specific contact topologies. To bypass the data collection bottleneck, CoToGrasp is trained entirely in an object-agnostic manner. We introduce a feature-based canonical workspace that projects local object features into a unified gripper-centric domain, effectively decoupling the semantic functional intent from the arbitrary object geometry. By learning the intrinsic contact manifold of the gripper within this workspace, our model achieves zero-shot generalization to unseen objects at inference. Extensive evaluations on the large-scale DexGraspNet dataset demonstrate that CoToGrasp achieves state-of-the-art performance, outperforming existing taxonomy-guided planners. Finally, we demonstrate the physical viability and kinematic feasibility of our synthesized contact topologies on a physical robot platform. Code is available on our project website https://cea-list.github.io/cotograspweb/ .