TOSC: Task-Oriented Shape Completion for Open-World Dexterous Grasp Generation from Partial Point Clouds

AI summary

Overview

Research area: Robotics — task-oriented dexterous grasping and 3D shape completion from partial observations.

Technical level: Advanced. The paper assumes familiarity with point cloud learning, masked autoencoders, diffusion/flow-matching generative models, ControlNet, and robot grasp evaluation metrics.

Scope: The paper introduces "Task-Oriented Shape Completion" (TOSC), a pipeline that completes only the task-relevant contact regions of a partially observed object and then generates a dexterous grasp for a specified manipulation task.

What This Paper Is About

Robots that only see a partial point cloud of an object (due to occlusion, clutter, or sensor noise) struggle to generate grasps that suit a specific downstream task, such as pouring or hammering. The authors argue that generic shape completion fails in these conditions because missing data is ambiguous, so they instead complete only the regions likely to be contacted during the task. The goal is to produce stable, semantically appropriate dexterous grasps for open-world objects directly from partial observations.

Key Contributions

1. A new task formulation — Task-Oriented Shape Completion. Rather than reconstructing the entire object geometry, the method reconstructs only contact-relevant regions, conditioned on the manipulation task rather than on geometry alone.

2. A candidate generation plus discriminative selection pipeline. Multiple plausible completed shapes are synthesized using pre-trained foundation models (ControlNet for RGB synthesis, a 3D shape generation network, SAM and a multi-modal large model for task-region detection), then a 3D discriminative autoencoder (DAE) scores plausibility and restores the best candidate from a global perspective.

3. FlowGrasp, a constraint-aware conditional flow-matching grasp generator. Geometric and semantic constraints are injected into training through a single-step, input-side gradient correction applied to each predicted velocity, requiring no additional explicit losses or inference-time optimization.

4. State-of-the-art results on both tasks. The method improves Grasp Displacement by 16.17% and Chamfer Distance by 55.26% over the state of the art, and shows generality on open-set categories and tasks.

Main Findings

• Grasp quality on OakInk-PartialPC. The method reports a Grasp Displace mean of 3.11 cm and variance of 3.54 cm, Contact Ratio of 98.30%, Penetration Volume of 6.87 cm³, Penetration Depth of 0.090 cm, P-FID of 21.60, and an LLM score of 88.3. This is the best reported Grasp Displace result in Table 1 among the compared methods (GraspCVAE, GraspTTA, SceneDiffuser, DexTOG, DexGYSGrasp).

• Penetration trade-off. SceneDiffuser reports the lowest Penetration Volume (6.52 cm³) and Depth (0.090 cm), which the authors attribute to a conservative strategy that keeps unnecessary distance between hand and object; the paper claims its own method balances penetration avoidance against grasp stability better.

• Perceptual user study. Human evaluators each scored 100 distinct grasps on a 0–5 scale across Semantic Consistency, Physical Plausibility, and Interaction Stability. The method scores 4.38, 3.84, and 3.80 respectively, the highest in the table.

• Shape completion superiority. On OakInk-PartialPC, the method reports CD-ℓ2 of 1.66 × 10⁻⁴, F-Score@1 of 0.860, and DCD of 0.488, versus PointAttn (4.58, 0.512, 0.698), SVDFormer (3.71, 0.643, 0.603), and SymmCompletion (3.94, 0.618, 0.611).

• Zero-shot generality on novel categories. Evaluated on 9 novel object categories, each containing about 100 objects plus novel language instructions, the method reports the best performance across all metrics in Table 4 — for example Grasp Displace mean 4.21 cm and P-FID 42.97, compared with GraspCVAE, GraspTTA, SceneDiffuser, DexTOG, and DexGYSGrasp.

• Ablation: every component matters. Removing TOSC Candidate Generation, TOSC Selection and Restoration, the task-oriented (versus generic) completion, token masking, or gradient guidance each degrades performance relative to the full method. Notably, the "w/o gradient guidance" variant reaches a higher Contact Ratio (98.83%) than the full method (98.30%) but much lower perceptual scores (3.72 / 2.18 / 3.63 versus 4.38 / 3.84 / 3.80).

• Training data scale. The DAE's plausible-shape training set contains 72,524 objects drawn from 6 datasets: ModelNet40, ShapeNetCore, ScanObjectNN, OmniObject3D, DexGraspNet, and AffordPose.

• Not reported: real-robot hardware experiments, inference latency, and comparison against methods not listed in the tables are not reported in the provided content.

Methodology in Plain English

The pipeline takes three inputs: a partial point cloud, the object's category label, and a natural-language description of the manipulation task.

Step 1 — Generate candidates. The partial point cloud is rendered into a depth image using a viewpoint chosen by Hidden-Point-Removal. ControlNet then turns that depth image into several RGB images, using the category as a prompt and varying a control scale so that some images stick closely to the observed geometry while others let the model "imagine" the missing parts. A 3D generative model (Hunyuan3D-DiT-v2-mini-Fast) converts each RGB image into a mesh, which is sampled into a point set. SAM segments the images, a multi-modal large model (GPT-4o) identifies task-relevant regions, and those regions are projected onto both the observed and generated clouds. An ICP-style alignment jointly minimizes Chamfer distance on the whole object and, with extra weight, on the task-relevant regions — producing several fused "TOSC candidates."

Step 2 — Score and restore. A 3D discriminative autoencoder decides which candidate is plausible. Its encoder tokenizes the cloud into patches (via farthest point sampling and KNN, with a lightweight PointNet), and predicts a Gaussian distribution. Plausible shapes are trained toward N(0,1) and deliberately sabotaged ones toward N(1,1), so at inference a likelihood comparison gives a plausibility score. The decoder then reconstructs a clean, complete shape. Training data is generated by pairing objects with random tasks, locating task-relevant parts with PartSlip, and then destroying those parts through removal, noise, and local perturbation (distinction between the paper's "OakInk-PartialPC" and "OakInk-PartialIPC" naming appears inconsistent in the text).

Step 3 — Generate the grasp. FlowGrasp is a conditional flow-matching model conditioned on a PointNet++ feature of the restored shape concatenated with a CLIP embedding of the task text. Instead of adding penalty losses or running gradient steps at inference, it corrects the training velocity target in a single step using the gradient of weighted constraint functions, and regresses that corrected target. At inference it simply integrates the learned ODE from t=0 to t=1.

Why This Matters

Impact on research. The paper reframes shape completion as a task-conditioned problem rather than a purely geometric one, and shows that throwing away irrelevant geometry is a feature, not a bug. It also offers a lightweight way to bake constraints into flow matching — one gradient correction at training time rather than test-time optimization — which is transferable to other constrained generative robotics problems.

Real-world applications

• Household service robots that must grasp occluded or cluttered items to pour, cut, or hand them over.
• Warehouse and logistics pick-and-place, where a bin blocks most of the object from view.
• Industrial automation cells where a tool must be grasped by a specific functional part (the handle, not the blade).
• Assistive robot arms operating from a single fixed camera in unstructured homes.

Industry relevance. The method is built almost entirely on off-the-shelf foundation models, so improvements in those base models can raise performance without retraining the grasp policy. FlowGrasp avoids inference-time gradient optimization, which matters for latency-sensitive robotic control, though the paper does not report runtime numbers. The reported Chamfer Distance gain of 55.26% and Grasp Displacement gain of 16.17% suggest a meaningful step toward partial-observation grasping that does not require expensive task-specific supervision.

Future Directions

• Real-robot validation. All reported results are on datasets and in simulation-like evaluation; physical deployment with a real dexterous hand is not demonstrated.
• Efficiency and deployment cost. The pipeline chains ControlNet, a 3D generative model, SAM, GPT-4o, a 3D DAE, and FlowGrasp; inference latency and compute budget are not reported and would be a natural follow-up.
• Beyond single-object grasps. Extending to multi-object clutter, articulated objects, or bimanual manipulation would test how well task-oriented completion scales.
• Closed-loop sensing. Whether the completed contact region can be refined using tactile feedback after initial contact — rather than committed to before grasping — remains an open question.

Target Audience

Robotics and embodied-AI researchers working on grasping, manipulation, and 3D perception; graduate students studying shape completion or generative models for robot action; and applied engineers building perception pipelines for robot arms who need to understand both the promise and the current dataset/simulation-bound limitations of foundation-model-driven grasp generation.

Authors’ abstract

Task-oriented dexterous grasping remains challenging in robotic manipulations of open-world objects under severe partial observation, where significant missing data invalidates generic shape completion. In this paper, to overcome this limitation, we study Task-Oriented Shape Completion, a new task that focuses on completing the potential contact regions rather than the entire shape. We argue that shape completion for grasping should be explicitly guided by the downstream manipulation task. To achieve this, we first generate multiple task-oriented shape completion candidates by leveraging the zero-shot capabilities of object functional understanding from several pre-trained foundation models. A 3D discriminative autoencoder is then proposed to evaluate the plausibility of each generated candidate and optimize the most plausible one from a global perspective. A conditional flow-matching model named FlowGrasp is developed to generate task-oriented dexterous grasps from the optimized shape. Our method achieves state-of-the-art performance in task-oriented dexterous grasping and task-oriented shape completion, improving the Grasp Displacement and the Chamfer Distance over the state-of-the-art by 16.17\% and 55.26%, respectively. In particular, it shows good capabilities in grasping objects with severe missing data. It also demonstrates good generality in handling open-set categories and tasks.