ActiveGrasp: Information-Guided Active Grasping with Calibrated Energy-based Model

AI summary

Overview

Research area: Robotics — robotic manipulation, active perception, next-best-view planning, and generative grasp pose modeling on the SE(3) manifold.

Technical level: Advanced. The paper relies on information theory (conditional entropy, Fisher/Hessian-based information gain), Gaussian Approximation of Posterior, energy-based models, denoised score matching, and 3D Gaussian Splatting.

Scope (one sentence): The paper proposes a calibrated energy-based grasp model and an information-theoretic next-best-view selection rule that chooses the view maximizing entropy reduction of the grasp distribution, evaluated in simulation and on a real robot arm.

What This Paper Is About

Grasping an object in a cluttered scene is hard because the initial camera views do not reveal enough about which grasps are actually feasible, so the robot should actively decide where to look next. Prior active-grasping methods estimate the value of a candidate view using visibility, voxel coverage, or the projection of the grasp distribution onto 2D/3D grids, which biases view selection toward scene completion or grasp position rather than the true grasp success distribution. This paper defines the information gain of a view as the reduction in entropy of the grasp success distribution, estimates that entropy with a calibrated energy-based model over SE(3) grasp poses, and uses it to drive view selection.

Key Contributions

1. An entropy definition for grasp distributions. The authors define grasp entropy as the conditional entropy of the success event given the grasp pose and scene representation, computed as the expectation of the Bernoulli entropy of a single grasp; they distinguish this from the Shannon entropy of the grasp pose density, which they show can actually increase after an observation.
2. Calibrated energy-based grasp generation. They augment an SE(3) energy-based grasp model with scene information, dual score-matching losses over successful and failed grasps, an Average Precision (AP) loss, and a learnable temperature, so that the predicted energy level corresponds to the real grasp success rate.
3. Information-gain-based next-best-view selection. They derive the information gain of a candidate view from the Hessian of the grasp entropy with respect to the scene representation, and select views by maximizing this gain without additional heuristics such as affordance maps or visibility counts.
4. A reproducible active grasping benchmark. They build a physically informed PyBullet simulation benchmark with YCB objects and release it, along with a project website (https://rpfey.github.io/activegrasp/).

Main Findings

• Highest simulated success rate: With the same grasp model (Se3diff Calib), ActiveGrasp reaches a 79.00% success rate (316/400 trials) in simulation, compared with 74.25% for Random, 74.00% for ActiveNGF and ACE, 73.75% for Breyer, 73.50% for FisherRF, and 73.00% for the uncalibrated Se3diff Scene + ActiveGrasp setting.
• Best calibration: ActiveGrasp achieves the lowest Expected Calibration Error at 0.02, versus 0.05–0.07 for the other active methods paired with Se3diff Calib, 0.35 for ACE + ACE, 0.40 for Se3diff + ActiveNGF, and 0.28 for Se3diff Scene + ActiveGrasp.
• Calibration beats grasp-model choice alone: Grasp models with other next-best-view planners score 62.00% (GSNet + ActiveNGF), 58.25% (Se3diff + ActiveNGF), 32.50% (Contact GraspNet + ActiveNGF), and 22.50% (ACE + ACE).
• More views do not substitute for active selection: Random with 8 more views than the other methods improves success from 74.25% to 76.50%, still below ActiveGrasp's 79.00% and at a higher execution time cost.
• Model ablations on ACRONYM: Average precision rises from 34.69 (Se3diff) to 66.00 (+Scene), 83.31 (+Scene+AP), 83.13 (+Scene+FSM+AP), and 87.84 (+Scene+FSM+AP+T). ECE drops from 0.17 to 0.03, and ECE measured after executing grasps in the Bullet simulator drops from 0.35 to 0.05. Adding failure grasps to score matching (FSM) lowers ECE (Bullet) from 0.17 to 0.08 but degrades angular/translational distance metrics; the learnable temperature recovers and improves all metrics.
• Real-robot results: On a Kinova 7 DoF arm, ActiveGrasp succeeds on 9/10, 8/10, and 9/10 trials across three setups (red cup, red chip bottle, red tomato can), compared with Breyer (6/10, 6/10, 8/10), ActiveNGF (7/10, 6/10, 8/10), and ACE (8/10, 4/10, 8/10).
• Faster view selection: ActiveGrasp takes 9.61 ± 0.09 s, versus ACE at 18.24 ± 0.15 s, ActiveNGF at 27.66 ± 0.32 s, and Breyer at 32.97 ± 0.45 s; vanilla FisherRF is faster at 7.43 ± 0.01 s but has a worse success rate and does not account for grasping.
• Qualitative view behavior: Breyer selects views on the side of the object with many obstacles, and ACE and ActiveNGF select views close to already-captured views, whereas ActiveGrasp selects views that observe highly uncertain grasps.

Methodology in Plain English

The system represents the scene as a set of 3D Gaussians (3D Gaussian Splatting), where each Gaussian stores position, rotation, scale, color, and a semantic channel marking the target object. From a set of initial views, this scene representation is built and continuously refined as new images arrive.

An energy-based model takes the scene and a candidate grasp pose in SE(3), plus a noise level, and outputs two logits for grasp success and failure. Those logits are converted into success and failure probabilities, and the energies are simply the negative logs of those probabilities, so the energy value is directly interpretable as a success probability. The model is trained with two score-matching losses — one pulling the energy landscape toward successful grasps, one pushing it away from failed grasps — plus an Average Precision loss that only applies gradients until positives rank above negatives, plus a signed-distance-function loss. A learnable temperature keeps the energy scale numerically stable.

To decide where to look next, the authors define the grasp entropy as the expected Bernoulli entropy of the success probability over the grasp pose distribution. They use a Gaussian approximation of the posterior around the maximum a posteriori scene estimate, expand the entropy to second order, and express the information gain of a candidate view as a trace involving the difference between the scene Hessian before and after taking that view. Candidate views are sampled on a sphere around the foreground object using Spherical Fibonacci Sampling, and the view with the highest gain is executed. The loop repeats until the view budget is exhausted, after which grasp poses are generated and executed.

In simulation, the setup uses a 7 DoF Franka robot with an 800x800 RGBD wrist camera, 10 objects per scene, 20 scenes, 5 target objects per scene, and 4 trials per target (400 trials). Each trial takes 2 fixed views and actively selects 2 more. In the real world, a Kinova 7 DoF arm takes 4 fixed views and actively selects 2. Training uses the Acronym dataset (200k steps, batch size 24, Adam at 1e-3, λ1=1, λ2=0.1, sigma=0.5), with 3DGS trained for 1k steps per new view and 128 candidate view proposals in a 0.3–0.5 m radius.

Why This Matters

Impact on research: The paper argues that information gain for grasping must be computed from the grasp distribution itself, on the SE(3) manifold, and on a calibrated distribution — three criteria the authors claim prior work violates. It introduces the distinction between the Shannon entropy of the pose density and the conditional entropy of grasp success, and it connects energy-based grasp models to next-best-view planning in a single framework. The released PyBullet/YCB benchmark with fixed trial counts aims to make active-grasping comparisons reproducible.

Real-world applications:

• Bin picking and order fulfillment, where objects are piled on top of each other and initial camera views are heavily occluded.
• Warehouse and logistics robotics that must operate under a limited sensing budget or cycle-time constraint.
• Household and service robots retrieving objects from cluttered shelves, counters, or drawers.
• Industrial feeding and assembly, where a manipulator must pick parts from a randomly arranged bin with limited camera coverage.

Industry relevance: The method targets a practical constraint — a fixed budget of camera views and limited execution time. Its 9.61 s view-selection time is roughly 2x to 3.4x faster than the compared active methods, and its calibration property matters in deployment because the robot executes the highest-scoring predicted grasp, so a calibrated score directly supports that decision. The reported real-robot evaluation on a Kinova arm with a RealSense camera indicates the pipeline is not simulation-only.

Future Directions

• Better second-order approximation: The authors state as a limitation that the information gain relies on an approximation of the second-order term of the grasp entropy with respect to the scene representation. Improving this approximation is an explicit open direction.
• Calibration under domain shift: The paper notes that calibration is a requirement of the method and calls for better calibration techniques when the model faces domain shift between training and deployment.
• Broader benchmarking: The released simulation benchmark is intended to enable follow-up reproducible comparisons; extending it to more object sets, clutter configurations, and robots follows naturally from that goal.
• Grasp model independence: The pipeline is demonstrated with SE(3) Diffusion Fields as the base energy model and compared against Contact GraspNet, ACE, GSNet, and Se3diff; whether the information-gain estimator transfers to other SE(3) generative grasp models remains an open question.

Target Audience

Researchers and graduate students working on robotic grasping, active perception, and next-best-view planning; practitioners building manipulation systems for cluttered environments with limited sensing budgets; and anyone interested in calibrated energy-based models, SE(3) manifold learning, or 3D Gaussian Splatting as a scene representation for manipulation.

Authors’ abstract

Grasping in a densely cluttered environment is a challenging task for robots. Previous methods tried to solve this problem by actively gathering multiple views before grasp pose generation. However, they either overlooked the importance of the grasp distribution for information gain estimation or relied on the projection of the grasp distribution, which ignores the structure of grasp poses on the SE(3) manifold. To tackle these challenges, we propose a calibrated energy-based model for grasp pose generation and an active view selection method that estimates information gain from grasp distribution. Our energy-based model captures the multi-modality nature of grasp distribution on the SE(3) manifold. The energy level is calibrated to the success rate of grasps so that the predicted distribution aligns with the real distribution. The next best view is selected by estimating the information gain for grasp from the calibrated distribution conditioned on the reconstructed environment, which could efficiently drive the robot to explore affordable parts of the target object. Experiments on simulated environments and real robot setups demonstrate that our model could successfully grasp objects in a cluttered environment with limited view budgets compared to previous state-of-the-art models. Our simulated environment can serve as a reproducible platform for future research on active grasping. The source code of our paper will be made public when the paper is released to the public.