Leveraging CVAE for Joint Configuration Estimation of Multifingered Grippers from Point Cloud Data

AI summary

Overview

Research area: Robotics — grasp planning, inverse kinematics (IK), and deep generative modeling for multifingered robotic hands.

Technical level: Advanced. The paper assumes familiarity with variational autoencoders, PointNet-style point cloud encoders, kinematic chains, and grasp metrics.

Scope: The paper proposes a Conditional Variational Auto-Encoder (CVAE) that reconstructs the 16 joint angles of the Allegro Hand directly from point cloud data of the gripper, and evaluates it on the MultiDex grasping dataset.

What This Paper Is About

Determining the joint angles of a multifingered gripper is normally done by solving inverse kinematics from fingertip poses — a process that can produce multiple or infinite solutions and often requires extra rules to pick among them, or numerical approximation for complex mechanisms. This paper replaces that optimization step with a learned model: a CVAE that takes a point cloud of the gripper (or a subset of it, such as contact points) and outputs the joint configuration directly. The goal is a fast, robot-centric pipeline that fits into AI-driven grasp planning and control.

Key Contributions

1. A CVAE-based pipeline that estimates the full joint configuration of a poly-articulated gripper from point cloud data alone, using a PointNet encoder for geometry and fully connected MLPs for joint values and reconstruction.
2. A robot-centric dataset generation procedure that requires only the gripper's URDF (giving access to its kinematic model and CAD files) and produces three alternative point cloud representations: Fully Dense PC, Cluster PC, and Handprint PC.
3. A self-collision filter (Equation 1) that removes invalid sampled configurations before training, using keypoints at link bounding-box centers and a distance threshold.
4. Validation on the MultiDex grasp dataset with the Allegro Hand, reporting joint and Cartesian errors plus inference times under 0.05 ms, with accuracy described as comparable to state-of-the-art CVAE-based IK approaches.

Main Findings

• Inference speed: Joint configuration inference took less than 0.05 ms on a single Nvidia GeForce RTX 4090 GPU with a batch size of 100. Per-configuration times in Table I are 0.042 ms (Fully Dense PC), 0.050 ms (Cluster PC), 0.048 ms (Handprint PC), 0.044 ms (Generic finger) and 0.043 ms (Thumb). The authors state this is compatible with robot controllers that operate within a few milliseconds.

• Joint accuracy across representations: Mean joint error was 0.075 ± 0.042 rad (Fully Dense PC), 0.064 ± 0.029 rad (Cluster PC) and 0.063 ± 0.035 rad (Handprint PC). Expressed relative to the maximum achievable error (full joint range), these correspond to 4.49%, 3.90% and 3.84%.

• Finger-level accuracy: Table I also reports separate columns for the generic finger and the thumb, with mean joint errors of 0.017 ± 0.012 rad (1.04%) and 0.018 ± 0.009 rad (1.10%). The authors note the generic finger and thumb results are of comparable magnitude despite the thumb's more complex kinematic diagram, which they read as evidence that the model is independent of a specific kinematic structure.

• Cartesian accuracy: Mean Cartesian error was 3.889 ± 2.421 mm (Fully Dense PC), 3.967 ± 1.633 mm (Cluster PC) and 3.821 ± 2.023 mm (Handprint PC), with 2.89%, 3.07% and 2.85% relative error. The generic finger and thumb columns report 0.219 ± 0.178 mm (0.64%) and 0.363 ± 0.380 mm (1.09%).

• Best-of-batch potential: The paper also reports the mean of the lowest joint and Cartesian error per batch, which is lower than the mean error. Lowest joint error per batch was 0.022 ± 0.003 rad (1.43%, Fully Dense PC), 0.029 ± 0.003 rad (1.94%, Cluster PC) and 0.022 ± 0.003 rad (1.43%, Handprint PC). Lowest Cartesian error per batch was 1.331 ± 0.176 mm (1.03%), 1.911 ± 0.227 mm (1.54%) and 1.475 ± 0.223 mm (1.13%). The authors attribute this to the CVAE mapping one input to multiple outputs, and note that sampling multiple latents and selecting the best would multiply inference time and be less suitable for real-time use.

• Handprint PC outperformed Cluster PC: The Handprint PC dataset showed improved accuracy over Cluster PC, which the authors attribute to better feature representation since more points describe the gripper configuration. Timing differences between representations were minimal.

• Dataset distribution: The authors' generated dataset had a joint-angle standard deviation of 0.287 versus 0.250 for MultiDex. They note the expected standard deviation of a uniform distribution on [0, 1] is 1/sqrt(12) ≈ 0.2887, so their dataset is close to uniform while MultiDex is not, likely because it focuses on grasp-related poses.

• Collision rate in MultiDex: Of the 37,774 Allegro Hand point clouds in MultiDex, only 64% (24,077) would have been retained by the authors' collision criteria, meaning at least 36% of MultiDex is not represented in their training data. Those configurations were kept in the validation set to test generalization.

• Encoder limitation: The PointNet encoder, though designed to be invariant to geometric transformations, does not fully achieve this invariance in practice. Adding rotations and translations to the dataset's point clouds would require a different encoder architecture capable of distinguishing both the point relationships defining the joint configuration and the gripper's overall 6D pose.

Methodology in Plain English

The authors build their own training data instead of relying on an existing grasp dataset. Starting from the gripper's URDF file, they sample M = 30,000 joint configurations from the range of motion of each joint using a uniform distribution on [0, 1] to keep training stable. Each sampled configuration is checked with a self-collision function inspired by prior work: keypoints at the center of each link's bounding box are compared, and if any two links come closer than a threshold distance, the configuration is discarded as invalid. For the surviving configurations, each link's mesh surface is sampled to build a point cloud. Three variants are produced: Fully Dense PC with 512 points equally spaced around the whole gripper, Cluster PC using a small representative cluster of points per link within a sphere of radius R (set to 50% of each link's radius) centered on the link's inner surface, and Handprint PC which keeps only points on the inner side of the fingers and palm, selected by a dot product between the outward palm normal and each point's normal in the nominal configuration with all joints at 0 radians.

The model itself is a CVAE. The encoder maps the input point subset S, conditioned on the joint configuration Q, into a latent distribution, using a PointNet to extract point features that are concatenated with features from an MLP applied to the joint values. The decoder does the reverse: it extracts point features with a second PointNet, concatenates them with a latent sample z, and generates the joint angles through a fully connected MLP. The prior over z is chosen to be a standard normal independent of the input, to give the model more flexibility. Training maximizes the evidence lower bound, equivalent to minimizing a loss combining a reconstruction term (an RMSE over the joint angles, Equation 4) and a weighted KL divergence between the learned posterior and the prior. The weight β starts at 0.0001 for the first 50 epochs, follows a sigmoid curve from 0.0001 to 1.0 between epochs 50 and 100, then stays at 1.0 to the end of training; the paper states no improvements were seen with a cyclical schedule. The β evolution over 250 epochs is shown in Figure 3.

Training ran for 4000 epochs with batch size 500 on a single Nvidia GeForce RTX 4090 GPU, taking 2 GPU hours on the Fully Dense PC dataset. The dataset was split 80% for training and 20% for testing, and optimized with Adam at a learning rate of 0.0001. Experiments used the Allegro Hand V4, which has N = 16 independent joints across four fingers — three generic fingers and one thumb.

Evaluation used MultiDex, a grasp dataset created by Li et al. containing 5 multifingered grippers (EZGripper, Barrett, Robotiq-3F, Allegro, and ShadowHand), 58 household objects from YCB and ContactDB, and 436,000 diverse grasping poses. The authors used the Allegro Hand poses, giving 37,774 hand point clouds compared with the 30,000 in their training dataset, sampled by the same procedure as their three representations. Metrics were mean joint error in radians, mean Cartesian error in millimetres (the L2 norm between ground truth and predicted keypoints, using the same keypoints as in the collision equation), and inference time in milliseconds measured on a single RTX 4090 with batch size 100 and averaged across batches.

Why This Matters

Impact on research: The work targets the part of the grasp planning pipeline that usually consumes the most computation — converting a desired set of contact points or fingertip poses into joint angles. By learning this mapping implicitly, the model can sidestep the multi-solution and redundancy problems that make analytical IK awkward, and it is trained only from a URDF rather than from expensive physical data collection. The authors position the result as comparable to other CVAE-based IK methods, such as the one reported in reference [12], while being faster to infer.

Real-world applications (as implied by the paper's framing):

• Real-time control of multifingered robotic hands in grasp planning pipelines, where inference must fit within a few milliseconds.
• Generating joint configurations from contact-point sets produced by grasp-planning or generative models, since the Cluster PC dataset is designed around contact regions.
• Vision-based gripper control, where point clouds come from visual sensors rather than simulation.
• Hand pose estimation from point cloud data, which the authors name as a target for expanding the pipeline's applicability.

Industry relevance: The pipeline is robot-centric and needs only the gripper's URDF, so adapting it to a new hand does not require new physical data collection. Combined with inference times reported below 0.05 ms, this makes it a candidate component for industrial pick-and-place, logistics, and any control stack that already produces 3D point clouds of the gripper or contact points. The work was funded by the European Union's Horizon Europe research and innovation program under grant agreement nº 101135708 (JARVIS Project).

Future Directions

• Estimating the gripper's 6D pose: The authors plan to estimate the gripper pose as an element of SE(3) relative to a desired reference frame, which they describe as crucial for mapping object pose to gripper pose in grasping tasks.
• Handling geometric transformations: Because PointNet does not fully achieve invariance in practice, adding rotations and translations to the dataset would require a new encoder architecture that understands both the point relationships defining the joint configuration and the gripper's overall pose.
• Hyperparameter optimization: Future work will extend the architecture while simultaneously optimizing the model's hyperparameters.
• Broader applicability: The authors aim to expand the pipeline to hand pose estimation and improve robustness across various grasping scenarios.

Target Audience

Robotics researchers and engineers working on dexterous manipulation, grasp planning, and IK for multifingered hands; machine learning practitioners interested in conditional generative models applied to point cloud regression; and control engineers who need fast joint-space inference for robot hands. Readers without a background in variational inference, PointNet-style architectures, or kinematic chains will find the method sections demanding, though the evaluation and motivation are accessible.

Authors’ abstract

This paper presents an efficient approach for determining the joint configuration of a multifingered gripper solely from the point cloud data of its poly-articulated chain, as generated by visual sensors, simulations or even generative neural networks. Well-known inverse kinematics (IK) techniques can provide mathematically exact solutions (when they exist) for joint configuration determination based solely on the fingertip pose, but often require post-hoc decision-making by considering the positions of all intermediate phalanges in the gripper's fingers, or rely on algorithms to numerically approximate solutions for more complex kinematics. In contrast, our method leverages machine learning to implicitly overcome these challenges. This is achieved through a Conditional Variational Auto-Encoder (CVAE), which takes point cloud data of key structural elements as input and reconstructs the corresponding joint configurations. We validate our approach on the MultiDex grasping dataset using the Allegro Hand, operating within 0.05 milliseconds and achieving accuracy comparable to state-of-the-art methods. This highlights the effectiveness of our pipeline for joint configuration estimation within the broader context of AI-driven techniques for grasp planning.