F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation

AI summary

Overview

Research area: Computer vision — category-level 6D object pose and size estimation from RGB-D data, with a secondary focus on object shape completion and robotic manipulation.

Technical level: Advanced. The paper assumes familiarity with point cloud networks (DGCNN), Transformer/Mixture-of-Experts architectures, SE(3) rotation parameterizations, and standard category-level pose benchmarks (NOCS-style metrics, SOPE/ROPE).

Scope (one sentence): The paper presents F2G-Pose, a single-pass RGB-D network that jointly predicts category-level 6D pose, metric size, and camera-frame completed shape by lifting frozen visual foundation-model features onto partial point clouds, and evaluates it on synthetic benchmarks, real-world datasets, and a physical robot.

What This Paper Is About

Category-level pose estimation must recover an object's rotation, translation, and size without an instance-specific CAD model, which is hard because a single-view depth crop shows only part of the object, symmetric shapes create rotational ambiguity, and objects within one category differ in shape and size. Existing approaches either rely on intermediate structures (canonical coordinates, correspondences, shape priors) that propagate errors, or use diffusion-style sampling that is slow and needs a separate step to pick one pose out of many candidates. F2G-Pose instead predicts the final pose, size, and a completed shape directly in one feed-forward pass.

Key Contributions

1. A direct, real-time RGB-D framework. F2G-Pose predicts category-level 6D pose, metric size, and camera-frame completed shape in a single feed-forward pass, sharing one representation across all three outputs and avoiding test-time sampling, candidate scoring, or canonical-shape alignment.

2. Geometry-aware foundation feature lifting. Dense RADIOv2.5 features (fused from layers 8, 16, and 23) are projected onto the observed 3D points, encoded with DGCNN tokenization, and integrated by a Mixture-of-Experts Transformer (8 experts, top-2 routing) to give conditional capacity across diverse shape regimes.

3. Shape completion as dense geometric supervision. A parallel shape branch reconstructs complete camera-frame geometry through coarse-to-fine refinement, which both serves as a usable output and regularizes the shared representation used for metric pose and size prediction.

4. Broad empirical validation including physical deployment. State-of-the-art accuracy is reported on SOPE and ROPE at 28 FPS for pose-only inference, with additional camera-frame geometry results on HouseCat6D, real-object transfer on HANDAL, latency and robustness analyses, and pick-and-place experiments on an ARX robot arm.

Main Findings

• Accuracy on SOPE: F2G-Pose reaches AUC IoU25/50/75 of 59.6 / 43.7 / 16.5, compared with GenPose++ at 50.1 / 31.9 / 6.4. Mean rotation error drops from 19.9° to 15.7° and translation error from 1.14 cm to 0.88 cm.

• Accuracy on ROPE: AUC IoU25/50/75 of 46.5 / 27.8 / 5.8, versus GenPose++ at 39.0 / 19.1 / 2.0 and RFMPose (report-only) at 42.1 / 21.0 / 2.2. VUS at 5°/5cm and 10°/2cm is 15.9 and 21.8, versus GenPose++ at 15.1 and 29.4 — the paper describes this as matching or exceeding available high-precision VUS results.

• Speed: 28 FPS in pose-only mode and 23.7 FPS with the shape head enabled, versus 4 FPS for GenPose++, 6 FPS for GenPose and SGPA, 5 FPS for NOCS, 35 FPS for IST-Net, and 50 FPS for HS-Pose.

• Candidate-to-decision gap in GenPose++: On ROPE, GenPose++'s sampled candidates have a median rotation error of only 8.2° but a mean of 39.3°, with 26.9% exceeding 45°. A best-of-50 oracle reaches 9.0° mean and selects a candidate within 5° for 79.3% of instances, yet the deployed pipeline ends at 30.7° mean and 48.3% within 5°.

• Shape completion: Chamfer-L1 (×10⁻³) of 5.42 on SOPE and 7.1 on HouseCat6D, versus AdaPointr at 24.41 and 32.3, Pointr at 29.87 and 40.2, and FoldingNet at 62.72 and 73.2.

• Ablations (ROPE, IoU25/50/75): Removing RGB foundation features causes the largest drop (34.5 / 16.2 / 2.2); removing MoE gives 41.0 / 24.0 / 3.9; removing shape supervision gives 39.5 / 23.2 / 3.5; the full model gives 46.5 / 27.8 / 5.8.

• Backbone is not the main driver: Swapping RADIO for DINOv2 yields 45.5 IoU25 / 27.0 IoU50 / 13.7 VUS(5°/5cm), and CLIP yields 44.8 / 26.3 / 12.5, against the full model's 46.5 / 27.8 / 15.9.

• Routing trade-off: Dense FFN achieves 24.0 IoU50 and 8.7 VUS at 19.2 FPS; top-1 MoE gives 25.6 and 13.5; top-2 without balancing 26.0 and 14.2; balanced top-2 27.8 and 15.9 at 23.7 FPS; top-4 28.0 and 16.1 at 18.5 FPS. Load balancing raises routing entropy from 0.71 to 0.95 (normalized by log 8).

• Challenging materials: On ROPE, transparent objects score 43.5 IoU25 / 4.7 IoU75 / 10.4 VUS(5°/2cm) with 25.3° rotation error and 1.19 cm translation error; reflective objects score 42.6 / 4.5 / 9.8 with 25.2° and 1.25 cm. Chamfer-L1 on HouseCat6D is 6.2 for transparent and 10.1 for reflective objects.

• Real-world transfer: On HANDAL, F2G-Pose reaches 35.0 / 11.7 / 0.3 AUC IoU25/50/75, versus Any6D (0 ref) at 14.5 / 3.8 / 0.0 and GenPose++ (0 ref) at 16.7 / 4.3 / 0.1. On ROPE it exceeds the single-reference methods FoundationPose (35.0 / 18.0 / 3.1) and Any6D (37.2 / 19.4 / 3.5) without any reference.

• Robot manipulation (80 trials per method): The full model achieves 74/80 successful picks and 68/80 placements at 36 ms latency, compared with 66/80 and 57/80 for the full GenPose++ pipeline at 256 ms, which returned 0% of usable decisions within the 50 ms budget. Depth-only F2G-Pose achieves 61/80 and 50/80; without shape completion, 68/80 and 57/80.

• Failure modes: Errors concentrate on corrupted depth that removes thin or asymmetric parts, severe occlusion hiding functional geometry, and symmetric low-texture objects that admit 90°/180° alternatives. High-precision IoU75 remains weak under cross-dataset shift.

Methodology in Plain English

The system takes a cropped RGB image and the matching segmented depth points of one object and produces three things at once: where the object is and how it is oriented, how big it is in meters, and a filled-in 3D shape of the whole object as seen from the camera.

To build a rich description of the object, the authors run a frozen, externally pretrained vision backbone (RADIOv2.5) on the image crop and then, for each 3D depth point, look up the corresponding image feature at that point's pixel location and attach it. This gives every point both its measured position in space and a semantic/appearance descriptor. A graph-based point network (DGCNN) then compresses the 1024 points into 128 tokens that capture local surface structure.

These tokens go through eight Transformer blocks. The key twist is that each block's ordinary feed-forward layer is replaced by a Mixture-of-Experts layer with eight experts, where each token is routed to its top two. A load-balancing penalty keeps the experts from collapsing onto a few favorites. The intuition is that different experts can specialize for different shape families, which helps when one taxonomy spans many object types.

Two small decoder heads sit on top of the pooled object representation. The pose-size head regresses translation, metric size, and rotation using a continuous 6D rotation representation (two columns of the rotation matrix, later orthogonalized), which is more stable to optimize than raw Euler angles or quaternions. The shape head first guesses 512 points, mixes them with 256 sampled observed points, scores all 768 to keep the 512 most reliable, and then predicts four offsets per point to produce a 2048-point completed shape — directly in the camera frame rather than in a canonical space.

Training uses Smooth L1 losses for rotation, translation, and size, with symmetry handling: for each axis the object is labeled as having no symmetry, 90°, 180°, or arbitrary rotational symmetry, and the loss compares against whichever valid ground-truth rotation is closest. Chamfer distance (L1 version) supervises both the intermediate and final shape outputs. The whole model is trained on the synthetic SOPE training split only (5K object instances, 149 categories), for 100 epochs with batch size 512 using AdamW, and the vision backbone stays frozen.

Why This Matters

Impact on research. The paper reframes category-level pose estimation as a direct prediction problem rather than a sampling-and-selection problem. Its candidate diagnostics quantify the "candidate-to-decision gap" in diffusion-based pose estimators — showing that a generative model can contain accurate hypotheses that the deployed pipeline fails to extract — which is a methodological argument that applies beyond this specific system. The finding that camera-frame shape completion acts as dense supervision that improves pose and size accuracy (removing it drops ROPE IoU50 from 27.8 to 23.2) is a reusable design lesson about multi-task regularization.

Real-world applications:

• Robotic pick-and-place and bin manipulation, where the reported 36 ms latency and 100% readiness within a 50 ms control budget matter more than marginal accuracy gains.
• Augmented reality, where placing virtual objects requires knowing metric size as well as pose, and where the completed shape gives a usable collision volume.
• Warehouse and logistics automation needing to handle novel instances of known categories without per-object CAD models.
• Assistive or collaborative robots operating on household objects (bottles, mugs, pans, tools — the categories tested on HANDAL and in the manipulation trials).

Industry relevance. The 28 FPS / 23.7 FPS throughput on a single RTX 4090, combined with training on only synthetic SOPE data and a frozen off-the-shelf backbone, makes the approach attractive for deployment pipelines where retraining a large foundation model is impractical. The requirement for pre-segmented RGB-D input is a realistic integration point, since most robot stacks already run an instance segmenter upstream.

Future Directions

• Mask-free operation. The method assumes a segmented RGB-D crop at inference; the authors explicitly list removing this dependency as future work, since mask failures propagate into pose, size, and shape.
• Calibrated uncertainty and abstention. The paper notes the model provides no calibrated uncertainty or abstention mechanism — notable given that the paper's own analysis shows how much accuracy hinges on selecting a reliable prediction.
• Articulated, deformable, and broader real-object coverage. The authors name articulated or deformable categories and wider real-object evaluation as next steps, extending beyond the rigid, fixed-taxonomy setting tested here.
• Active perception for heavy occlusion. Severe occlusion is identified as a dominant failure mode (alongside thin parts and symmetric low-texture objects), and the authors propose active perception as a route to acquire better viewpoints rather than only improving single-view inference.

Target Audience

Researchers and graduate students working on 6D object pose estimation, category-level 3D vision, or point cloud learning who want to understand where direct feed-forward prediction stands against generative/diffusion approaches. It is also relevant to robotics engineers evaluating whether a perception module can meet a real-time control budget under segmented RGB-D input, and to practitioners interested in how frozen foundation-model features can be lifted onto 3D observations rather than used as a standalone image prior. Beginners will find the introduction and method overview accessible, but the experiments and ablations assume familiarity with AUC IoU, VUS, and Chamfer distance metrics.

Authors’ abstract

This paper studies category-level RGB-D object pose estimation, which recovers an object's 3D rotation and translation without instance-specific CAD models or reference views. This task is highly ambiguous due to intra-class variations, symmetries, and partial observations. Current solutions struggle to balance reliability and efficiency: methods based on intermediate structures are prone to error propagation, while diffusion models suffer from high inference latency and complex post-hoc candidate extraction. To address this, we present F2G-Pose, a real-time direct, single-pass framework that predicts category-level pose, metric size, and camera-frame completed shape from a single RGB-D observation in one feed-forward pass. F2G-Pose lifts dense visual foundation features to partial point clouds, encodes local geometry with DGCNN tokenization, and integrates object-level structure with a Mixture-of-Experts Transformer. Joint size prediction and camera-frame shape completion encourage metric consistency through dense geometric supervision. With its task-specific network trained only on pose-annotated synthetic SOPE data and its frozen RADIO backbone externally pretrained, F2G-Pose achieves state-of-the-art accuracy on SOPE and ROPE while running at 28 FPS for pose-only inference, and further demonstrates strong camera-frame geometry on HouseCat6D and real-object transfer on HANDAL. Analyses of latency, robustness, shape completion, and GenPose++ candidate behavior further demonstrate the efficiency and reliability of F2G-Pose, with real-world manipulation experiments demonstrating deployment potential under segmented RGB-D inputs. The project page, code, and pretrained models are made available at https://jrryzh.github.io/F2G-Pose/.