Lite Any Stereo: Efficient Zero-Shot Stereo Matching

AI summary

Overview

• Research area: Computer vision, specifically stereo matching (estimating per-pixel depth/disparity from a rectified left-right image pair), with a focus on efficient architectures and zero-shot generalization to unseen real-world datasets.
• Technical level: Intermediate. The core problem and results are graspable without deep background, but the architectural details (cost volumes, 2D vs. 3D cost aggregation, distillation losses) assume some familiarity with stereo matching literature.
• Scope: The paper proposes Lite Any Stereo, an ultra-light stereo matching network with a hybrid 3D-2D cost aggregation module and a three-stage training pipeline on million-scale data, evaluated for zero-shot performance on KITTI 2012, KITTI 2015, ETH3D, Middlebury, and DrivingStereo weather.

What This Paper Is About

Recent stereo matching research has prioritized accuracy, producing large and computationally expensive models, while efficient models have been assumed to lack the capacity for zero-shot generalization to new domains. The gap matters because most efficient stereo methods still require domain-specific fine-tuning, so they are not usable as off-the-shelf tools. This paper asks whether an ultra-light model, trained correctly on large-scale data, can match accuracy-focused stereo models on unseen real-world benchmarks while using a small fraction of the computation.

Key Contributions

1. Lite Any Stereo, an efficient stereo matching model with zero-shot capability. The authors report it delivers the highest zero-shot accuracy among all efficient methods by a large margin, and is the first efficient model to outperform or match accuracy-oriented models that do not use foundational priors, while requiring less than 1% of their computational cost.
2. A hybrid cost aggregation module that jointly uses 3D and 2D blocks to capture complementary disparity (cross-disparity structure) and spatial information, enhancing representation at low cost.
3. A three-stage training strategy combining supervised synthetic training, self-distillation, and knowledge distillation on real-world unlabeled data, which the authors show improves zero-shot generalization and also transfers to other architectures (LightStereo-M and BANet-2D).
4. Compilation of a 0.5M real-world unlabeled stereo dataset across six sources (Flickr1024, InStereo2k, Holopix50K, DrivingStereo, SouthKenSV, UASOL) used with pseudo labels for training, alongside the finding that data quality matters more than scale.

Main Findings

• Zero-shot results (million-scale training block, Table 3): Lite Any Stereo reports KITTI 2012 D1 3.04 / EPE 0.79; KITTI 2015 D1 3.87 / EPE 0.99; ETH3D Bad 1.0 3.53 / EPE 0.32; Middlebury Bad 2.0 7.51 / EPE 0.94, at 33 G MACs.
• Efficient baselines trained on million-scale data: LightStereo-M* reports 4.10/0.99, 4.97/1.13, 5.33/0.41, 10.85/1.51 at 33 G MACs; BANet-2D* reports 3.90/0.93, 4.71/1.07, 5.92/0.38, 10.05/1.34 at 36 G MACs; StereoAnything-L† reports 4.00/0.92, 4.81/1.10, 3.81/0.31, 9.82/1.21 at 84 G MACs. The authors state their model outperforms all of these by a large margin.
• Comparison with accurate, non-prior-based methods: Selective-IGEV‡ is listed at 3.20 (KITTI 2012), 4.50 (KITTI 2015), 3.40 (ETH3D), 7.50 (Middlebury) with 3619 G MACs — about 110 times the MACs of the proposed 33 G model, which the authors say the proposed model matches or exceeds.
• Prior-based accurate reference: FoundationStereo is listed at 12824 G MACs with 2.51/0.67, 2.83/0.86, 0.49/0.14, and 1.12/0.37, and remains more accurate than the proposed method.
• SceneFlow-only fair comparison (Table 3, first block): With training restricted to stage 2 on SceneFlow, Lite Any Stereo reports KITTI 2012 D1 5.45 / EPE 1.18, KITTI 2015 D1 6.45 / EPE 1.32, ETH3D Bad 1.0 15.38 / EPE 0.77, Middlebury Bad 2.0 13.13 / EPE 1.60 at 33 G MACs — the lowest MACs among the listed efficient methods, which range from 33 G to 564 G (MobileStereoNet-3D).
• In-domain results on DrivingStereo weather (Table 4): The model surpasses its pseudo-label teacher, FoundationStereo, overall (D1 8.74 / EPE 1.80 at 33 G MACs versus 10.71 / 2.22 at 12824 G MACs), and wins on all four weather subsets: Rainy (20.69/2.61 vs. 27.01/3.96), Sunny (3.84/1.47 vs. 4.31/1.57), Foggy (6.78/1.64 vs. 7.67/1.82), Cloudy (3.65/1.47 vs. 3.85/1.53).
• KITTI leaderboard (Table 5): The finetuned version ranks 1st among all efficient methods on the leaderboards at the time of submission, with 3-noc 1.09, 3-all 1.49, 4-noc 0.76, 4-all 1.04, D1-bg 1.36, D1-fg 3.45, D1-all 1.71 at 33 G MACs.
• Ablation on aggregation design (Table 2, rows labelled a): The 3D-2D serial configuration reports the strongest balance, for example ETH3D 5.39 and Middlebury 10.89 at 33.9 G MACs, whereas the pure 2D baseline reports 6.48 and 11.29 at 32.9 G MACs, and the 2D-3D ordering fails badly on ETH3D at 28.85.
• Ablation on 3D kernel size (Table 2, rows labelled b): The standard (3, 3, 3) kernel performs best (4.38 / 4.84 / 5.75 / 9.50), while larger kernels such as (5, 3, 3), (7, 3, 3) and (11, 3, 3) do not improve results — contrary to findings in FoundationStereo.
• Ablation on 3D proportion (Table 2, rows labelled d): Increasing the proportion of 3D blocks degrades performance, particularly on Middlebury; the default configuration uses 4.8%.
• Ablation on distillation scheme (Table 2, rows labelled f): Keeping the teacher fixed gives 3.64 / 4.63 / 6.82 / 8.86, better than EMA (3.97 / 4.82 / 6.91 / 9.78) and hard copy (4.22 / 4.71 / 6.35 / 10.07).
• Three-stage training progression (Table 2, rows labelled g): Stage 1 gives 4.05 / 4.55 / 4.43 / 8.49, stage 2 gives 3.66 / 4.53 / 4.69 / 7.03, and stage 3 gives 3.04 / 3.87 / 3.53 / 7.51. The authors note a slight drop on Middlebury in stage 3, attributed to the limited scale of indoor real-world data.
• Generality of the training strategy (Table 6): Applied to LightStereo-M, results improve from stage 1 (4.34 / 5.27 / 6.68 / 10.29) to stage 3 (3.35 / 4.14 / 4.22 / 9.85); applied to BANet-2D, they improve from stage 1 (4.34 / 4.78 / 7.71 / 10.54) to stage 3 (3.28 / 4.08 / 4.05 / 10.30).
• Inference time (Table 7, measured locally, torch.compile, KITTI resolution): Lite Any Stereo records 4 ms on RTX 4090, 9 ms on A5000, 7 ms on A100, and 6 ms on H200, placing it among the fastest of the compared methods.
• Limitations stated: The model still trails depth-prior-based approaches, a major bottleneck being the limited availability of high-quality real-world stereo data; robustness under transparency and reflection can still be improved.

Methodology in Plain English

The system is a feed-forward network with four stages.

1. Feature extraction. A weight-sharing backbone processes the left and right images and produces multi-scale features at 1/4, 1/8, 1/16 and 1/32 resolution, which are upsampled to 1/4 resolution. The authors deliberately do not use DepthAnything depth features; they find that even the smallest DA-S variant adds prohibitive overhead for an efficiency-oriented model. They use a conventional backbone pretrained on ImageNet with no external priors, and report that MobileNetV2's channel configuration suits the task better than more recent CNN backbones.
2. Correlation. A cost volume is built by taking normalized inner products between left and right feature vectors across candidate disparities, at 1/4 resolution up to D_max/4.
3. Hybrid cost aggregation. A small 3D component aggregates jointly over disparity and spatial dimensions, followed by a 2D component using ConvNeXt layers for spatial refinement. The authors state that a naive half-and-half split is ineffective because 3D convolutions dominate the compute budget along the disparity dimension, so only a small proportion of 3D blocks is retained.
4. Disparity estimation. A soft-argmax over the aggregated cost volume regresses a 1/4-resolution disparity map, which is upsampled to full resolution with convex upsampling.

Training happens in three stages:

• Stage 1 – supervised on synthetic data. The model is trained end-to-end with a smooth L1 disparity loss on 1.8M annotated synthetic samples drawn from SceneFlow (35K), FallingThings (30K), FSD (1.1M), CREStereo (0.2M), VKITTI2 (21K), TartanAir (0.31M) and Dynamic Replica (0.14M). Datasets such as IRS, Sintel, Spring and InfinigenSV are excluded for annotation-quality or domain-mismatch reasons.
• Stage 2 – self-distillation. Two copies of the Stage 1 model are used: a teacher that sees clean inputs and a student that sees strongly perturbed inputs. Besides the disparity loss, a cosine-based feature alignment loss pulls the student's features toward the teacher's, encouraging domain-invariant representations. The teacher's weights stay fixed, which the ablation shows works better than EMA or copying student weights.
• Stage 3 – real-world knowledge distillation. The model is fine-tuned on 0.5M unlabeled real-world stereo pairs with pseudo labels generated by FoundationStereo. Even for datasets that have ground truth (for example DrivingStereo), dense pseudo labels are used instead of sparse LiDAR labels because they give stronger supervision. Low-quality or narrow-domain datasets such as Stereo4D, HRWSI and SCOD are deliberately excluded. Self-distillation is not applied in this stage because it gave no observable gains with pseudo labels.

Training details: 150K, 50K and 100K steps for stages 1, 2 and 3, total batch size 176 on NVIDIA A100 GPUs, AdamW optimizer with a one-cycle schedule and peak learning rate 2×10⁻⁴, random crops of 256×512 then finetuning at 320×736, and a maximum disparity D_max of 192.

Why This Matters

The work challenges the assumption that zero-shot generalization requires either a large model or a monocular depth prior, showing that careful architecture plus a well-designed data pipeline can close most of the gap at very low compute. It also introduces an architecture-agnostic training recipe (demonstrated on LightStereo-M and BANet-2D) and makes use of unlabeled real-world stereo data that the stereo community has largely left unused.

Real-world applications:

• Autonomous driving and driver assistance, where dense depth from a stereo pair is needed on embedded hardware with tight latency budgets; the paper's own evaluation includes DrivingStereo weather frames (500 frames per weather class) captured in driving environments.
• Robotics and mobile manipulation, where stereo cameras are common and on-board compute is limited.
• Mobile and embedded depth sensing in phones, AR/VR headsets and drones, where a 33 G MAC model is far more deployable than a 3619 G or 12824 G model.
• Off-the-shelf deployment on unseen scenes, since a model with zero-shot behavior avoids the cost of collecting and annotating data and fine-tuning per deployment domain.

Industry relevance: the reported inference times (4 ms on RTX 4090, 7 ms on A100, 6 ms on H200) and the roughly 1% MACs relative to accurate methods make this relevant to anyone deploying stereo depth under compute or power constraints, and the model's ability to beat its own pseudo-label teacher on DrivingStereo weather suggests the distillation pipeline is a practical way to convert cheap unlabeled data into usable accuracy.

Future Directions

• Scaling and improving real-world stereo data. The authors identify limited

Authors’ abstract

Recent advances in stereo matching have focused on accuracy, often at the cost of significantly increased model size. Traditionally, the community has regarded efficient models as incapable of zero-shot ability due to their limited capacity. In this paper, we introduce Lite Any Stereo, a stereo depth estimation framework that achieves strong zero-shot generalization while remaining highly efficient. To this end, we design a compact yet expressive backbone to ensure scalability, along with a carefully crafted hybrid cost aggregation module. We further propose a three-stage training strategy on million-scale data to effectively bridge the sim-to-real gap. Together, these components demonstrate that an ultra-light model can deliver strong generalization, ranking 1st across four widely used real-world benchmarks. Remarkably, our model attains accuracy comparable to or exceeding state-of-the-art non-prior-based accurate methods while requiring less than 1% computational cost, setting a new standard for efficient stereo matching.