MixRI: Mixing Features of Reference Images for Novel Object Pose Estimation

AI summary

Overview

Research area: Computer vision — six degrees-of-freedom (6DoF) object pose estimation, specifically CAD-based novel object pose estimation from RGB images.

Technical level: Advanced. The paper assumes familiarity with multi-view geometry, attention mechanisms, cost volumes, and PnP-based pose solvers.

Scope: The paper introduces MixRI, a lightweight network that estimates the 6DoF pose of previously unseen objects by fusing features from a small set of reference images (12 by default) and matching them directly against a single query image, evaluated on the seven core datasets of the BOP challenge.

What This Paper Is About

Estimating the pose of an object that was never seen during training normally requires rendering and storing hundreds of reference images and running large neural networks, which is impractical on memory-constrained edge devices. MixRI tackles this by skipping the usual two-stage "retrieve the closest reference view, then match" pipeline and instead directly fusing features from all reference views to locate each sampled 3D object point on the query image. The goal is competitive pose accuracy with far fewer reference images, far fewer network parameters, and faster inference.

Key Contributions

1. MixRI, a lightweight RGB-based network for novel object pose estimation that requires only 12 reference images and does not require offline pre-extraction of image features. The network has 5.3 M parameters.
2. A View-Aggregated Point Matching module that finds correspondences on the query image simultaneously from multiple reference images and additionally predicts an occlusion flag for each point, rather than selecting a single closest reference view.
3. A Dual-Attention Based Feature Mixer composed of SAP (Self-Attention between Points), SAF (Self-Attention between Frames), and MARQ (Mix-Attention between Reference & Query), which operates on feature tokens gathered at the projections of the same 3D object points across all reference images.
4. An empirical demonstration that comparable performance to state-of-the-art methods can be achieved with significantly fewer reference images, significantly fewer network parameters, shorter reference-image preparation time, and faster inference.

Main Findings

• Seven-dataset accuracy: On the BOP core datasets with the CNOS detector and no refinement, MixRI reaches a mean AR of 31.4 using 12 reference images and 34.1 using 24 reference images.
• Per-dataset AR (12 references): LM-O 27.0, T-LESS 25.4, TUD-L 29.3, IC-BIN 29.7, ITODD 10.9, HomebrewedDB (HB) 44.9, YCB-V 52.8.
• Per-dataset AR (24 references): LM-O 30.4, T-LESS 27.4, TUD-L 33.6, IC-BIN 30.8, ITODD 11.6, HB 50.2, YCB-V 54.6.
• Efficiency: Inference time is 0.5 s with 12 reference images and 0.7 s with 24, measured on a single 4090 GPU and including the CNOS detection time.
• Comparison to FoundPose (37.2 mean AR): The paper reports similar performance while using 33× fewer reference images, fewer network parameters, and shorter inference time. FoundPose is listed with 798 reference images, 302.9 M parameters, and 1.6 s.
• Comparison to GigaPose (27.6 mean AR, 316.3 M parameters, 162 reference images, 0.8 s): MixRI improves accuracy by approximately 10% on the challenging IC-BIN and HB datasets and by around 25% on YCB-V, and raises the mean AR across the seven core datasets by 6.5%. MixRI has over 50× fewer parameters than GigaPose.
• Comparison to MegaPose (20.8 mean AR, 520 reference images, 21.6 M parameters, 15.5 s): MixRI achieves better results across all seven core datasets and improves by around 15% on average.
• ITODD weakness: ITODD performance (10.9 at 12 references, 11.6 at 24) is worse than GigaPose (18.8). The authors attribute this to ITODD's grayscale images, whereas training uses RGB, plus severe occlusions, reflections, and weak-texture surfaces.
• Limited reference images: Against GigaPose on YCB-V, LM-O, and TUD-L, MixRI's three-dataset mean is 17.9 at 4 references versus GigaPose's 9.1; 31.4 versus 11.0 at 6; 35.5 versus 14.4 at 8; 36.4 versus 14.7 at 12; and 39.5 versus 18.3 at 24. GigaPose degrades sharply when references are scarce because its retrieval stage cannot find a sufficiently similar view.
• Attention module ablation: With neither SAP nor MARQ, mean AR is 2.6. Adding SAP raises it to 11.9 (a 9.3% increase); adding MARQ alone gives 22.8 (a 20.2% increase); using both gives 39.5, a 36.9% increase over the baseline.
• Correspondence count: Performance is robust to the number of correspondences and improves until around 2400. With only 60 correspondences, YCB-V already reaches an AR of 45%, and at 240 correspondences the result is already comparable to other methods.
• Number of reference images at test time: Although trained with 12 reference images, the network accepts a different count at inference; performance degrades noticeably when the count exceeds 60, which the authors attribute to the lightweight network's limited capacity for fusing very many views.
• Occlusion flags: Disabling the mechanism (threshold 1.0) yields a mean AR of 30.7; the best setting, threshold 0.8, yields 34.1; threshold 0.5 yields 31.8.
• With refinement (MegaPose): Using 12 reference images, MixRI plus refinement reaches a mean AR of 45.6 in 1.5 s; with 24 references, 46.9 in 1.7 s. A variant called MixRI+ (same architecture, trained for more reference images and using 162 references) reaches 41.3 mean AR without refinement — surpassing FoundPose's 37.2 — and 53.0 with refinement in 2.1 s. For comparison, GigaPose with refinement is 54.7 in 2.3 s, FoundPose 54.7 in 4.4 s, and MegaPose 50.9 in 17.0 s.

Methodology in Plain English

The method reduces pose estimation to a matching problem, but a matching problem of an unusual shape: instead of matching the query image against one reference image, it matches one query image against the projections of the same 3D points across all reference images.

First, the pipeline samples 2D points inside the object mask of reference images, unprojects them into 3D object coordinates using the known ground-truth poses and depth, and then re-projects each 3D point into every other reference image. Points that fall outside the image or are occluded (detected by comparing projected depth against rendered depth with a 4 mm threshold) receive an occlusion flag. Because the projections of one 3D point across the reference images are known, the network can gather feature tokens exactly at those locations.

Training uses synthetic images only, from the GSO-Dataset. Each reference image contributes M = 10 sampled points, with S = 24 reference images, producing N = 240 points in total. Data augmentation includes Gaussian blur, contrast, brightness, sharpness, and color changes.

At the network level, the query image and all reference images pass through a shared ResNet-like encoder (feature maps at 1/8 resolution) that is made rotation-invariant. Feature tokens are retrieved at the projected locations by bilinear interpolation. The Dual-Attention Based Feature Mixer then iteratively mixes information: SAP performs self-attention across the N points within a frame, SAF performs self-attention across the S frames with learnable fused tokens, and MARQ applies two self-attention and two cross-attention layers between the fused tokens and the query feature. An occlusion mask is used in the attention to prevent occluded points from contributing erroneous features.

The fused features and the enhanced query feature form a cost volume processed by a Conv3D backbone with two heads: one predicts a heatmap for each 3D object point on the query image, the other predicts the point's occlusion flag. A "spatial soft argmax" turns the heatmap into 2D coordinates, and only points predicted as visible are kept. The final 6DoF pose is computed with a RANSAC-based SQ-PnP algorithm.

Training uses a BCE loss for occlusion supervision and a Huber loss for location supervision on unoccluded points, combined with a weight of λ = 100 on the location loss, averaged across the N sampled points.

Why This Matters

Impact on research: The paper pushes back on the assumption that high pose accuracy requires hundreds of reference images and large networks. By showing that a 5.3 M-parameter network with 12 reference images can reach a 31.4 mean AR across the BOP core datasets — and 41.3 with the MixRI+ variant using 162 references — it suggests that the two-stage retrieval-then-match paradigm, and the memory cost of caching pre-extracted features, may be avoidable.

Real-world applications:

• Robotic manipulation, where a robot must grasp an object it has never encountered and therefore needs its orientation and location in 3D space.
• Augmented reality, where virtual content must be placed consistently onto a physical object in the user's view.
• Warehouse and logistics picking, where the set of objects changes frequently and retraining per object is impractical.
• Edge and embedded deployment, such as on devices with limited memory where storing hundreds of reference images and their cached features is not feasible.

Industry relevance: The paper explicitly targets edge AI. Removing feature pre-extraction and using 12 reference images reduces both storage and preparation time relative to methods such as GigaPose and FoundPose, which need to pre-extract and cache reference features. The reported 0.5 s inference time including detection, and the 5.3 M parameter count, are framed as practical advantages for deployment on memory-constrained hardware.

Future Directions

• Improve matching accuracy further, which the authors state as a future goal.
• Extend the framework to scenarios where object models are unavailable, i.e., moving beyond CAD-based pose estimation.
• Address failure cases involving weak texture and heavy occlusion, such as the bowl example shown in the supplementary material, where the object is texture-less and nearly occluded.
• Handle grayscale imagery and weak-texture surfaces better, given the performance gap on ITODD that the authors attribute to training on RGB images and to severe reflections and occlusions in that dataset.
• Reconcile the lightweight design with larger reference sets, since the paper reports degradation when more than 60 reference images are used with the lightweight network.

Target Audience

Researchers and engineers working on 6DoF object pose estimation, novel/unseen object recognition, and robot perception. It is also relevant to practitioners deploying vision models on edge devices who care about memory footprint, reference-image storage cost, and inference latency, and to readers interested in attention-based multi-view feature fusion and feature matching. Readers without a background in multi-view geometry, PnP solvers, or transformer attention will find parts of the method section demanding.

Authors’ abstract

We present MixRI, a lightweight network that solves the CAD-based novel object pose estimation problem in RGB images. It can be instantly applied to a novel object at test time without finetuning. We design our network to meet the demands of real-world applications, emphasizing reduced memory requirements and fast inference time. Unlike existing works that utilize many reference images and have large network parameters, we directly match points based on the multi-view information between the query and reference images with a lightweight network. Thanks to our reference image fusion strategy, we significantly decrease the number of reference images, thus decreasing the time needed to process these images and the memory required to store them. Furthermore, with our lightweight network, our method requires less inference time. Though with fewer reference images, experiments on seven core datasets in the BOP challenge show that our method achieves comparable results with other methods that require more reference images and larger network parameters.