Research
Click3R: Interactive Stereo 3D Reconstruction with Sparse Correspondence Clicks
Feed-forward 3D reconstruction models such as DUSt3R predict dense 3D pointmaps from an image pair in a single forward pass, which makes them fast but also irrevocable: when the prediction is wrong…

- arXiv
- 2611.12875
- Published
- 2026-11-09
- Authors
- Hao Gu, Jin Yao, Zezhou Cheng
AI summary
Overview
- Research area: Computer vision — feed-forward stereo and multi-view 3D reconstruction, with a focus on human-in-the-loop (interactive) geometric correction.
- Technical level: Advanced. The method builds directly on the DUSt3R architecture (ViT-Large encoder, asymmetric cross-attention decoder) and assumes familiarity with pointmaps, camera pose metrics and standard 3D benchmarks.
- Scope (one sentence): The paper introduces Click3R, a lightweight correspondence-prompting adapter that lets a user correct a feed-forward stereo reconstruction with a handful of cross-view point clicks, plus Hard3DBench, a curated benchmark of scenes where such models typically fail.
What This Paper Is About
Feed-forward 3D reconstruction models such as DUSt3R predict dense 3D pointmaps from an image pair in a single forward pass, which makes them fast but also irrevocable: when the prediction is wrong — on repetitive textures, symmetric structures, textureless regions, low overlap or large viewpoint changes — the model fails silently and the user has no way to fix it. Click3R lets a person supply a small set of cross-view point correspondences ("clicks") that act as geometric constraints, and injects them into the network so the reconstruction can be corrected without test-time fine-tuning or optimisation. The authors also build Hard3DBench, a benchmark of deliberately hard image pairs and 5-view tuples, to measure whether such interaction actually helps.
Key Contributions
- Click3R framework: a lightweight point correspondence adapter that injects sparse human correspondence clicks into a feed-forward stereo 3D reconstruction network (DUSt3R) via cross-attention, between the encoder projection and the first decoder block. With no prompts the adapter is an identity mapping, so the model behaves exactly like the base network.
- Hard3DBench: a diagnostic benchmark with 2-view (404 in-distribution and 300 out-of-distribution pairs) and 5-view (400 in-distribution and 300 out-of-distribution tuples) subsets, curated by ranking pairs by pretrained DUSt3R rotation error and labelled with a five-category failure taxonomy computed from ground-truth poses and depth.
- Empirical evidence that sparse interaction works: improvements over a retrained DUSt3R baseline on standard benchmarks, on Hard3DBench, and with real human-labelled clicks (3.8 per pair on average), including gains from a single prompt.
- First-of-its-kind framing: the authors state this is the first work on interactive stereo feed-forward 3D reconstruction using sparse correspondence clicks, drawing the analogy to SAM's click-based prompting for 2D segmentation.
Main Findings
- One click already helps on standard benchmarks. With a single best-of-5 prompt, mAA@30° rises from 84.33 to 85.71 on CO3D, from 90.81 to 91.75 on ScanNet++, and from 82.71 to 85.90 on ARKitScenes, relative to the retrained DUSt3R baseline at 224×224 input.
- Prompt placement matters a lot. A grid-oracle strategy (partitioning view 1 into a 16×16 grid of 14-pixel cells and trying one correspondence per populated cell) reaches 88.21 on CO3D, 93.94 on ScanNet++, and 92.75 on ARKitScenes; the paper describes the ARKitScenes jump as +8.5% mAA@30° moving from 84.27% (no prompts) to 92.75%.
- Prompts mainly fix scene configuration, not local surface detail. The grid oracle gives the largest pose gains while pointmap error improves only modestly — for example ARKitScenes pointmap error goes from 0.157 (no prompts) to 0.146 (grid oracle).
- Large gains on hard scenes. On Hard3DBench 2-view in-distribution, the retrained DUSt3R baseline scores 56.1 mAA@30°; Click3R reaches 61.8 with 8 prompts (+5.7) and 61.3 with 4 prompts (+5.2), with pointmap error dropping from 0.216 to 0.186.
- Multi-view benefits are biggest. On the 5-view in-distribution split, the baseline scores 39.4 mAA@30° and Click3R reaches 49.3 with 8 prompts (+9.9), with pointmap error falling from 2.131 to 1.525.
- Out-of-distribution gains hold. On the 2-view out-of-distribution split, 8 prompts raise mAA@30° from 52.6 to 62.4 (+9.8); on the 5-view out-of-distribution split, from 54.8 to 60.0 (+5.2).
- Real human clicks capture most of the benefit. On 100 out-of-distribution pairs with human annotations (average 3.8 clicks, range 1–7), median rotation error drops from 17.30° (no prompts) to 8.78°, and mAA@30 rises from 36.2 to 56.2. The random ground-truth oracle with K=4 reaches 8.40° and 59.2, so human labels recover about 96% of the median rotation-error improvement gap.
- End-to-end training is necessary. Averaged over the three standard benchmarks and both Hard3DBench 2-view splits with one ground-truth click, training from CroCo v2 reaches 75.1% mAA@30°, fine-tuning DUSt3R reaches 72.3%, and freezing DUSt3R and training only the adapter gives 71.2%.
- The model is sensitive to misleading clicks only beyond a threshold. Shifting the matched point in view 2 lowers predicted confidence steadily, and reconstruction degrades noticeably once perturbations reach half a patch (8 pixels).
- Overhead is small. The adapter adds 34.1M parameters (6.4%) — 566.4M versus 532.3M for DUSt3R — with no additional compute when no prompts are given (20.4 ms on a single A100 at 224×224) and about 5% overhead at K=8 prompts (21.5 ms) and K=12 (21.4 ms).
- Hard3DBench is hard for newer models too. Compared with random pairs from the same datasets, VGGT and Pi3 at 518 resolution drop in RRA@5 from 82.6% and 85.1% to 54.7% and 57.9% on the in-distribution 2-view pairs, and from 98.2% and 97.0% to 88.7% and 81.3% out of distribution.
Methodology in Plain English
Start from DUSt3R, a network that takes two images, encodes each with a shared ViT encoder, decodes them with cross-attention between views, and outputs a dense 3D pointmap plus a confidence map per view.
Then insert a small "adapter" in the middle, right after the encoder features are projected into the decoder's dimension and before the first decoder block. This adapter takes two inputs: the dense image tokens and a set of correspondence tokens. Each click pair becomes two tokens, one per view. A token encodes the click's position with random Fourier features (so sub-pixel precision is kept), plus a learnable "which image am I in" embedding, plus a learnable "which correspondence pair am I part of" embedding — the pair embedding is what links the two halves of a correspondence across views.
The adapter is a two-layer, SAM-style two-way transformer: it does self-attention among the prompt tokens, then prompt-to-image cross-attention, then an MLP, then image-to-prompt cross-attention. So clicks push geometric constraints into the image features, while the click tokens also absorb local visual context. Each view gets its own adapter (matching DUSt3R's asymmetric decoder), but they share one tokenizer. If the user supplies zero clicks, the adapter is bypassed and the model is plain DUSt3R.
Training is fully end-to-end on the same nine datasets DUSt3R used (100,000 pairs each, at 224×224), using DUSt3R's confidence-weighted regression loss. Crucially, training prompts are simulated from ground-truth geometry: 3D points are sampled in view 1, projected into both views, and kept only if the projected point lands within 5 cm of the other view's ground-truth surface (which filters out occlusions). The number of clicks per sample follows a curriculum — 30% of samples get none, 40% get 1–4, and 30% get 5–12 — so the model learns to work at any click budget.
For evaluation, the authors compare against a retrained DUSt3R (same data as Click3R, to isolate the effect of prompting), original DUSt3R, MASt3R, and — as reference only, since they use bigger architectures and extra proprietary data — VGGT and Pi3/Pi3X. Metrics are scale-and-shift-invariant pointmap error and mAA@30° for pose, with poses recovered by PnP-RANSAC from the predicted pointmaps. Hard3DBench is built by running pretrained DUSt3R over tens of thousands of candidate pairs per dataset (roughly 69K in-distribution and 68K out-of-distribution), ranking them by rotation error, classifying them into failure categories (large viewpoint change above 60°, moderate viewpoint change between 30° and 60°, low overlap below 20%, large baseline where baseline over median depth exceeds 0.3, extreme depth range where max over min depth exceeds 50), and subsampling with a fixed seed.
Why This Matters
- Impact on research: The paper argues that silent, uncorrectable failure is a structural weakness of the feed-forward paradigm, and shows a cheap way to add a correction channel without giving up inference speed. It also reframes correspondences — long a classical geometry tool — as a prompt modality for learned 3D models, and supplies a diagnostic benchmark plus failure taxonomy that aggregate metrics tend to hide.
- Real-world applications:
- Augmented and virtual reality capture, where a user can tap a few matching points to repair a bad room scan.
- Robotics and embodied agents, where a human supervisor corrects a scene reconstruction before manipulation or navigation.
- 3D content creation and e-commerce, where product or scene models must be accurate and a small number of clicks is far cheaper than rescanning.
- Heritage, inspection and real-estate documentation, where textureless or symmetric structures routinely break automatic matching.
- Industry relevance: The overhead profile is attractive for deployment — identical cost to the base model when unused, roughly 5% when used, and no per-scene optimisation loop. It also suggests a product pattern: an automatic reconstruction that a user can nudge, rather than a one-shot black box.
Future Directions
- Automatic prompt suggestion. The grid-oracle results show large headroom depending on where clicks land; predicting good click locations (or auto-generating them and letting a human confirm) is a natural next step, and the paper already includes an automatic-prompt experiment in its appendix.
- Scaling beyond stereo pairs. Click3R is demonstrated on 2-view and, through DUSt3R-style global alignment over C(5,2)=10 pairwise predictions, 5-view settings; extending correspondence prompting natively to many-view and video models (Spann3R, MUSt3R, CUT3R, Fast3R, VGGT, Pi3) is open.
- Robustness to bad clicks. The perturbation study shows degradation past half a patch, and the authors flag limitations and failure cases in appendices; making the model actively detect and reject misleading or inconsistent clicks remains open.
- Combining with other conditioning signals. The paper positions correspondence prompting as orthogonal to conditioning on intrinsics, pose or depth (Pow3R, MapAnything, PromptDA); how these signals compose, and whether clicks can substitute for missing calibration, is unresolved.
Target Audience
Researchers and graduate students in 3D vision, multi-view geometry and generative/interactive perception who already know DUSt3R-style pointmap regression; practitioners building AR/VR capture, robotics perception or 3D asset pipelines who need correctable rather than one-shot reconstruction; and HCI or interactive-ML researchers interested in how sparse human input transfers from 2D prompting (SAM, drag-based editing) to 3D geometry. Beginners will find the problem framing accessible but will need background on pointmaps, camera pose metrics and transformer cross-attention to follow the method and results tables.
Authors’ abstract
Despite impressive progress, existing feed-forward large 3D reconstruction models produce a single, irrevocable prediction. When these models fail, there is no mechanism to correct their mistakes. To address this limitation, we introduce Click3R, a framework for interactive geometric correction of feed-forward models using sparse human correspondence clicks. Given an image pair and a small set of cross-view point correspondences provided by a user, Click3R injects geometric…
The full abstract on arXiv