GraspLDP: Towards Generalizable Grasping Policy via Latent Diffusion

AI summary

Overview

Research area: Robotics — robot learning, specifically imitation-learned visuomotor policies for robotic grasping, combining latent diffusion models with pretrained grasp-detection priors.

Technical level: Advanced. The paper assumes familiarity with denoising diffusion models, variational autoencoders, action chunking, SE(3) pose geometry, and visual imitation learning.

Scope: The paper proposes GraspLDP, a two-stage latent diffusion policy that injects grasp-pose and graspness priors from a pretrained grasp detector into action generation, and evaluates it on the LIBERO simulator benchmark plus real-robot grasping, cluttered-scene, and dynamic-grasping tasks.

What This Paper Is About

General-purpose imitation-learned manipulation policies (such as Diffusion Policy) tend to grasp less precisely and generalize less well than specialized grasp detectors, because modeling an entire grasping trajectory is harder than predicting a single static grasp pose. The authors' goal is to let a diffusion policy inherit the precision and generalization of a dedicated grasp detector without the enormous data and compute cost of billion-frame Vision-Language-Action training. Their answer is to inject grasp priors directly into the policy's latent action space, rather than treating a grasp pose as just another conditioning input.

Key Contributions

1. Latent-space grasp-pose guidance. Instead of concatenating a grasp pose with observations as an extra condition, GraspLDP compresses action chunks into a compact action latent with a lightweight VAE encoder, pairs the latent with the target grasp pose in that shared space, and reconstructs actions with an asymmetric decoder — decoupling "where to grasp" from "how to move."

2. Visual graspness cue with self-supervised reconstruction. Point-wise graspness scores from a pretrained graspness network are back-projected into pixel space, thresholded, and superimposed on the wrist-camera image. The policy is additionally trained to reconstruct this cued image from intermediate reverse-diffusion representations, forcing it to actually attend to the cue.

3. Heuristic Pose Selector (HPS) for inference. Given multiple grasp pose candidates, HPS discards colliding poses, applies non-maximum suppression, keeps the top-k by grasp score, and then chooses the candidate minimizing a weighted SE(3) geodesic distance to the current end-effector pose — balancing grasp quality against kinematic proximity.

4. A new evaluation metric, Grasp Frame Error (GFE). GFE measures how closely the frame at which the gripper begins to close matches the target grasp pose guidance, quantified with the same weighted SE(3) metric used in HPS.

Main Findings

• In-domain simulation success: GraspLDP reaches 80.3% (261/325) success rate on the LIBERO-based benchmark, versus 62.8% (204/325) for Diffusion Policy, 57.5% (187/325) for fine-tuned OpenVLA, and 50.8% (165/325) for GraspVLA. The authors report this as a 17.5% improvement over diffusion policy.

• Generalization gains: GraspLDP scores 71.1% spatial, 58.2% object, and 64.6% visual generalization, compared to 48.9%, 11.4%, and 16.3% for Diffusion Policy — reported as gains of 22.2%, 46.8%, and 48.3% respectively. Overall average: 68.6% for GraspLDP versus 34.9% for Diffusion Policy, 49.7% for GraspVLA, and 56.9% for the "Ours Baseline" conditioning-only variant.

• Ablation confirms both components matter: Removing the graspness cue drops in-domain SR to 77.4%; replacing latent guidance with simple condition guidance drops it to 73.5%; removing latent guidance entirely drops it to 60.6%; removing both drops it to 55.1%. Grasp Frame Error rises correspondingly as guidance is removed (from 1.33 to 1.49 to 1.58 in-domain).

• HPS beats single-criterion selection: With random selection, in-domain SR falls to 66.8%; highest grasp score, 72.0%; nearest end-effector pose, 69.5% — all below HPS's 80.3%.

• Real-world grasping is competitive with a dedicated detector: On the Franka Research 3 setup, GraspLDP achieves 84.0% on ID&SG, 75.0% on novel objects, 77.0% under visual variation (average 78.7%), close to AnyGrasp's average of 79.7% and far above Diffusion Policy (43.0%) and GraspVLA (26.0%).

• Cluttered-scene strength: In four tabletop scenes with 5 to 8 objects (Scene 4 includes stacked objects), GraspLDP and AnyGrasp both reach the highest Scene Completion Rate of 92.3%; GraspLDP attains a 12.7% higher success rate. GraspLDP clears every object in Scenes 1–3, despite being trained only on single-object demonstrations.

• Dynamic grasping: On moving-object and handover tasks, GraspLDP succeeds in more trials than Diffusion Policy, GraspVLA, and AnyGrasp (which uses a tracker); the paper notes it shortens the action horizon from 8 to 4 for these experiments.

• Latency is modest despite extra components: Graspness inference adds 36 ms and latent decoding adds <1 ms; GraspLDP is about 15% slower than diffusion policy at the same configuration, while GraspVLA's inference latency remains 154 ms higher than GraspLDP even after torch.compile() acceleration.

• Data efficiency and few-shot behavior: Using 12K, 1.2K, and 120 demonstrations, GraspLDP scores 80.3%, 64.6%, and 43.1%, versus Diffusion Policy's 62.8%, 41.5%, and 13.8% — gains of +17.5, +23.1, and +29.3 points.

• Robust to weaker grasp detectors: With GSNet, SBG, and GraspNet as prior sources, Grasp Detection baseline scores are 78.5%, 73.8%, and 70.8%, while GraspLDP improves them to 80.3%, 76.9%, and 75.4% respectively; gains are larger when the initial grasp poses are more suboptimal.

Methodology in Plain English

The authors build on a two-stage latent diffusion framework. In the first stage, action chunks (sequences of end-effector poses) are compressed by a lightweight VAE encoder into compact action latents, and an asymmetric decoder rebuilds the chunk from the latent concatenated with the target grasp pose. The VAE is trained with an L2 reconstruction loss plus a KL penalty, with a KL multiplier of 1e-6.

In the second stage, a diffusion model denoises in this compact latent space rather than directly in action space, conditioned on the current observation (RGB images, depth, and proprioceptive state). Two grasp priors from a pretrained detector enter here. The first is the grasp pose itself, used inside the latent space so it steers generation more strongly than a plain conditioning input would. The second is the graspness map: a graspness network assigns each point in the depth-projected point cloud a score in [0,1], those scores are back-projected to pixels, and pixels above a threshold τ are masked with color onto the wrist-view RGB image. The resulting cued image both conditions denoising and serves as an auxiliary self-supervised target — the model must reconstruct it from intermediate representations at each reverse step.

At inference, the grasp detector proposes many candidates. HPS removes colliding poses, applies non-maximum suppression, keeps the top-k by grasp score, and picks the candidate with the smallest weighted SE(3) geodesic distance (with diagonal weights w_t and w_r making translation and rotation commensurate) to the current end-effector pose. The decoder then turns the denoised action latent plus the selected grasp into the final action chunk.

Training data in simulation was generated by importing GraspNet-1Billion's 3D objects into LIBERO (robosuite/MuJoCo), sampling diverse high-quality grasp poses from the dataset labels with NMS, synthesizing trajectories via SLERP for rotations and linear interpolation for translations, simulating them, and keeping only successful demonstrations, with a heuristic speed de-biasing step.

Why This Matters

Impact on research. The paper argues that current prior-centric grasping policies under-use the mature grasp-detection literature — most only feed in a grasp pose as weak conditioning — and that data-centric alternatives like GraspVLA are uneconomical (Syngrasp-1b uses 1 billion simulated frames and 160 RTX 4090 GPUs for 10 days). GraspLDP shows a comparatively low-cost route to combining a closed-loop diffusion policy with detector priors, and introduces GFE as a way to measure how faithfully a policy follows grasp guidance.

Real-world applications (as evaluated or implied by the paper):

• Bin picking and tabletop clearing in cluttered scenes with 5 to 8 objects, including stacked configurations.
• Catching or tracking moving objects, such as a moving banana or watermelon, and grasping a mug handed over by a person.
• Grabbing everyday household items — the real-world object set includes rigid bodies plus a small number of articulated and deformable objects.
• Operation under degraded lighting, where the illumination-invariant graspness cue is credited with maintaining success.

Industry relevance. Low inference latency matters for real-time grasping; the paper positions GraspLDP as roughly 15% slower than Diffusion Policy but far faster than GraspVLA in absolute terms, which it frames as better suited to dynamic scenes. Few-shot behavior (43.1% with only 120 demonstrations) and robustness to weaker grasp detectors suggest the approach could be layered onto existing detection pipelines rather than requiring large new data collection efforts.

Future Directions

• Tactile and force/torque sensing. The authors state that highly deformable or fragile objects such as eggs or beakers remain a challenge, and plan to incorporate high-frequency tactile and force/torque signals.
• Toward a general grasping and manipulation foundation model. The conclusion frames the method as a foundation for future work on robotic foundation models for grasping and manipulation.
• Extending beyond the grasp pose as the sole prior. The related-work discussion notes that most prior-centric methods are limited to grasp poses; richer detector outputs remain unexplored.
• Sustained dynamic grasping. The dynamic results are partial — several cells in the dynamic task table are failures — leaving headroom for more robust time-consistent grasp tracking.

Target Audience

Robotics and embodied-AI researchers working on imitation learning, diffusion policies, or 6-DoF grasp detection; engineers building closed-loop pick-and-place, bin-picking, or dynamic grasping systems; and graduate students who already understand diffusion models and robot learning and want a concrete case study of injecting structured geometric priors into a generative policy.

Authors’ abstract

This paper focuses on enhancing the grasping precision and generalization of manipulation policies learned via imitation learning. Diffusion-based policy learning methods have recently become the mainstream approach for robotic manipulation tasks. As grasping is a critical subtask in manipulation, the ability of imitation-learned policies to execute precise and generalizable grasps merits particular attention. Existing imitation learning techniques for grasping often suffer from imprecise grasp executions, limited spatial generalization, and poor object generalization. To address these challenges, we incorporate grasp prior knowledge into the diffusion policy framework. In particular, we employ a latent diffusion policy to guide action chunk decoding with grasp pose prior, ensuring that generated motion trajectories adhere closely to feasible grasp configurations. Furthermore, we introduce a self-supervised reconstruction objective during diffusion to embed the graspness prior: at each reverse diffusion step, we reconstruct wrist-camera images back-projected the graspness from the intermediate representations. Both simulation and real robot experiments demonstrate that our approach significantly outperforms baseline methods and exhibits strong dynamic grasping capabilities.