LabelAny3D: Label Any Object 3D in the Wild

AI summary

Overview

• Research area: Monocular 3D object detection and automatic 3D data annotation (computer vision).
• Technical level: Intermediate. The paper assumes familiarity with monocular 3D detection, depth estimation, and 3D reconstruction, but its pipeline is described step by step.
• Scope: The paper introduces LabelAny3D, an analysis-by-synthesis pipeline that turns single in-the-wild RGB images into 3D bounding box annotations, and uses it to build COCO3D, an open-vocabulary monocular 3D detection benchmark derived from MS-COCO.

What This Paper Is About

Monocular 3D detection from a single image works well in indoor and autonomous driving settings, but struggles on ordinary internet-style photos because almost no 3D-annotated in-the-wild data exists, and manually drawing 3D boxes is expensive. The paper's goal is to generate high-quality 3D bounding box labels for arbitrary object categories in natural images with minimal human supervision, by reconstructing the whole 3D scene from a 2D image rather than guessing depth directly. It then uses those labels to train a detector and to build a new benchmark.

Key Contributions

1. LabelAny3D, an efficient 3D annotation pipeline that produces 3D bounding boxes on in-the-wild images in an analysis-by-synthesis manner.
2. A demonstration that 3D annotations from LabelAny3D consistently improve monocular 3D detection performance, surpassing existing auto-labeling approaches.
3. COCO3D, a new benchmark for open-vocabulary monocular 3D detection, featuring diverse object categories beyond those covered in existing 3D datasets.
4. A new evaluation metric, Relative Layout AP₃D, which scores how consistent the predicted and ground-truth relative spatial layouts are by optimizing a global scale factor, since metric depth in the benchmark comes from model predictions.

Main Findings

• Pseudo-label quality far exceeds the prior auto-labeling baseline. On COCO3D, LabelAny3D reaches AP₃D 64.17, AP₃D at IoU 0.15 of 82.11, AP₃D at IoU 0.25 of 74.47, AP₃D at IoU 0.50 of 57.34, AR₃D 73.57, relative AP₃D 64.17, and relative AR₃D 73.57. The comparison method OVM3D-Det (given the same depth, marked with *) scores 10.03, 16.88, 9.03, 1.44, 17.82, 10.04, and 17.84 respectively.
• Training on LabelAny3D labels helps even from scratch. Training OVMono3D from scratch on OVM3D-Det labels fails to converge, and fine-tuning on OVM3D-Det labels alone performs poorly. Training from scratch on LabelAny3D pseudo-labels reaches 7.78 AP₃D and 15.41 AR₃D on COCO3D.
• Combining Omni3D with LabelAny3D gives the best COCO3D result. Fine-tuning the pretrained OVMono3D on Omni3D plus LabelAny3D yields 10.92 AP₃D and 20.10 AR₃D on COCO3D (a 5.05 AP₃D increase over the 5.87 AP₃D Omni3D-only baseline), and 32.02 relative AP₃D / 43.82 relative AR₃D.
• Omni3D plus OVM3D-Det only marginally helps. That combination reaches 6.82 AP₃D on COCO3D — a 0.95 AP₃D gain over the baseline — and degrades performance on novel categories, which the authors attribute to excessive label noise.
• Better out-of-domain generalization on novel categories. The model trained solely on COCO3D pseudo labels achieves 8.47 AP₃D on OVMono3D's out-of-domain novel categories, and the Omni3D + LabelAny3D model reaches 16.98 AP₃D on Omni3D novel categories versus 16.05 for the Omni3D-only baseline.
• Base-category degradation is acknowledged. All fine-tuned models show some drop on OVMono3D's base categories (Omni3D + LabelAny3D scores 22.74 AP₃D versus 24.77 for the baseline). The authors attribute this to catastrophic forgetting from added scene and category diversity without more model capacity, and to label noise.
• KITTI annotation comparison. LabelAny3D achieves a higher overall AP₃D of 13.6 on KITTI compared to 12.39 from OVM3D-Det, and on the truck category 32.74 versus 13.46.
• Ablations confirm each component matters. On a subset of COCO3D, the vanilla 3D scene reconstruction model Gen3DSR scores AP₃D 1.95 versus 43.17 for the full pipeline. Removing super-resolution drops it to 28.13, removing amodal completion to 39.22, replacing MoGe with metric depth alone to 22.77, switching TRELLIS to DreamGaussian to 36.84 (a 6.33-point drop), and swapping 2D matching plus Perspective-n-Point for ICP to 24.28.
• Human refinement is cheap. Of the 5,373 annotations in COCO3D, 3,146 were accepted by annotators without modification, 2,227 needed only minor refinement, and 466 were rejected for reasons such as reflections, 2D object representations, or insufficient point cloud quality. Average IoU₃D exceeds 0.40 for the majority of categories.

Methodology in Plain English

Instead of estimating depth for each object and guessing its size, the pipeline rebuilds the entire scene in 3D and reads the labels off that reconstruction:

1. Upscale the image. A diffusion-based super-resolution model called InvSR enlarges the input image by 4x, so small or blurry objects can be reconstructed later.
2. Get clean 2D masks. The pipeline uses refined masks from the COCONut dataset instead of raw MS-COCO masks, removes objects that touch the image border (treated as truncated), removes masks below a size threshold, and upscales the masks to match the enlarged image.
3. Complete and reconstruct each object. An amodal completion diffusion model from Gen3DSR inpaints occluded parts of the object crop, then a single-view reconstruction method (for example TRELLIS) recovers a full 3D mesh in a canonical, normalized pose.
4. Estimate scene geometry. A relative depth model (MoGe) provides the scene's geometry and camera intrinsics; these are aligned to the scale and perspective of a metric depth model such as Depth Pro. The aligned depth map is unprojected into 3D space to recover the scene structure.
5. Place each object in the scene. Dense 2D-2D correspondences between the real image and rendered views of the mesh are computed with MASt3R; matched keypoints are unprojected to 3D points on the mesh, and a PnP solver with RANSAC estimates the camera pose.
6. Recover metric scale. The scale factor is the median ratio of real to rendered depth over the overlap region between the segmentation mask and the rendered mask.
7. Extract the box. Points are sampled from the mesh surface, the vertical axis is aligned to the canonical upward direction from TRELLIS, PCA on the projected points determines yaw, and a tight 3D bounding box is fitted.

For the benchmark, five annotators with prior 3D vision experience refined the pipeline's output by adjusting box dimensions, rotation, and center using the generated point maps, and removed noisy or invalid boxes and entire images with incorrect relative geometry.

For detection training, the authors build on OVMono3D, which detects objects in 2D with an open-vocabulary detector and then lifts 2D boxes to 3D cuboids using features from a pretrained vision transformer. Only the lifting head is trained, using ground-truth 2D boxes and a loss that sums separated attribute losses for 2D center shift, depth, dimensions, and rotation, plus a Chamfer loss comparing the full predicted box to the ground truth. Training used 15,869 MS-COCO training images annotated by LabelAny3D with no human refinement.

Why This Matters

• Research impact: The paper reframes 3D annotation as a scene reconstruction problem rather than a depth-regression problem, and shows that this yields substantially cleaner supervision than metric-prior approaches. It also supplies an evaluation benchmark and a scale-invariant metric for a setting — in-the-wild monocular 3D detection — that previously had no quantitative test bed. The authors note the pipeline may also help amodal 3D reconstruction, 6D pose estimation, and scene completion.
• Real-world applications:
  • Robotics and embodied AI, where a single camera is far cheaper than LiDAR rigs or multi-view stereo.
  • Autonomous driving and outdoor scene understanding.
  • AR/VR wearables, which the paper cites as well suited to lightweight monocular methods.
  • Large-scale dataset construction for general scene understanding where 3D sensors cannot be deployed.
• Industry relevance: The pipeline is modular, so improvements in depth estimation or amodal completion can be dropped in directly. Reducing 3D labeling cost matters to any organization that needs 3D supervision at scale, and the results suggest pseudo-labels from this pipeline are usable both for fine-tuning and for training from scratch, with human annotators spending time only on refinement.

Future Directions

• Condition 3D generation on RGBD data. The authors note that TRELLIS can produce meshes with ambiguous depth along the viewing direction, causing misalignment with RGBD point clouds; they point to Hunyuan3D-Omni as a possible direction.
• Develop robust training strategies for noisy pseudo-annotations, since label noise is cited as one cause of the base-category degradation seen in fine-tuned models.
• Improve robustness in the failure cases identified: heavy occlusion, textureless regions, and small objects all inject noise into the final 3D bounding boxes.
• Build a larger benchmark. The paper states that a larger version 2 of COCO3D will be released soon; version 1 covers 2,039 human-refined images and 5,373 instances. The authors also note the benchmark is not exhaustively annotated because objects with erroneous depth, severe occlusion, or truncation were excluded.

Target Audience

Researchers and engineers working on monocular 3D object detection, 3D data annotation, and open-vocabulary 3D perception; practitioners who need to build 3D datasets without LiDAR or depth sensors; and teams evaluating whether foundation-model-driven auto-labeling is good enough to replace or accelerate manual 3D labeling. Readers already familiar with Cube R-CNN, OVMono3D, and Omni3D will get the most out of it, though the pipeline description is self-contained enough for those newer to the area.

Authors’ abstract

Detecting objects in 3D space from monocular input is crucial for applications ranging from robotics to scene understanding. Despite advanced performance in the indoor and autonomous driving domains, existing monocular 3D detection models struggle with in-the-wild images due to the lack of 3D in-the-wild datasets and the challenges of 3D annotation. We introduce LabelAny3D, an \emph{analysis-by-synthesis} framework that reconstructs holistic 3D scenes from 2D images to efficiently produce high-quality 3D bounding box annotations. Built on this pipeline, we present COCO3D, a new benchmark for open-vocabulary monocular 3D detection, derived from the MS-COCO dataset and covering a wide range of object categories absent from existing 3D datasets. Experiments show that annotations generated by LabelAny3D improve monocular 3D detection performance across multiple benchmarks, outperforming prior auto-labeling approaches in quality. These results demonstrate the promise of foundation-model-driven annotation for scaling up 3D recognition in realistic, open-world settings.