PPMStereo: Pick-and-Play Memory Construction for Consistent Dynamic Stereo Matching

AI summary

Overview

Research area: Computer vision — dynamic (video) stereo matching and temporally consistent depth estimation. Technical level: Intermediate. Scope: This paper proposes a memory-buffer mechanism, called Pick-and-Play Memory (PPM), that lets a stereo matching network selectively store and reuse information from distant video frames to produce smooth, flicker-free depth maps at low computational cost.

What This Paper Is About

Most deep stereo matching models estimate depth for each image pair independently, so when applied to video they produce depth maps that jitter and flicker from frame to frame. Prior video-based methods try to fix this either by looking only at the immediately preceding frame (too little context) or by attending over a large sliding window of frames (expensive and treats every frame as equally reliable). PPMStereo instead builds a compact, self-curating memory buffer that keeps only the most useful past frames and weights them according to their reliability, achieving long-range temporal consistency without a large compute penalty.

Key Contributions

1. First memory-buffer framework for dynamic stereo matching. PPMStereo is the first work to introduce a memory buffer to the dynamic stereo matching task, enabling long-range spatio-temporal modeling at a computational cost far below naive all-frame attention.
2. A novel Pick-and-Play Memory (PPM) construction module. The "pick" stage chooses the top-K most informative reference frames from the full sequence; the "play" stage adaptively reweights those selected frames before aggregation.
3. A Quality Assessment Module (QAM). A scoring scheme that combines a confidence score (computed over memory values) with a redundancy-aware relevance score (combining query–key similarity and a usage-count penalty), so the memory stays both reliable and diverse.
4. State-of-the-art accuracy and temporal consistency across multiple benchmarks, at lower cost than the previous best method. The authors also present PPMStereo_VDA, a variant using a frozen Video Depth Anything (ViT-Small) feature extractor that further improves results.

Main Findings

• Strong temporal consistency on Sintel. PPMStereo reaches a Temporal End-Point Error (TEPE) of 0.62 pixels on the clean pass and 1.11 pixels on the final pass, with 3-pixel error rates of 5.19% and 7.64% respectively.
• Clear improvement over the previous state of the art. Versus BiDAStereo, this is a 17.3% (clean) and 9.02% (final) reduction in TEPE and a 9.74% / 10.32% improvement in 3-pixel error rate — achieved with lower compute and memory usage.
• Generalization to unseen data. Trained only on synthetic data, PPMStereo surpasses CREStereo on Sintel final pass temporal consistency, even though CREStereo was trained with Sintel in its data mixture.
• Memory capacity alone is not enough. Using the full 20-frame buffer causes out-of-memory errors, and random frame selection performs worse than simply keeping one adjacent frame on Sintel, showing that which frames are stored matters more than how many.
• Both pick and play contribute. Ablations show the pick stage alone brings large gains (e.g., 8.65 → 7.81 δ₃ₚₓ on Sintel final), the play stage alone brings smaller gains, and combining them is best (7.64).
• Optimal memory length is around K = 5. Performance improves up to K = 5 and then saturates (K = 7 gives only a marginal gain), which the authors attribute to feature redundancy.
• Individual QAM components all help. Confidence, similarity, and redundancy scores each improve results incrementally, with the full combination performing best; the learned confidence maps correlate strongly with actual disparity error maps.
• Better qualitative depth. PPMStereo produces sharper disparity maps in textureless regions (e.g., glass) and recovers thin structures such as fences on the real-world South Kensington dataset, with lower high-variance regions in 3D point-cloud renderings.

Methodology in Plain English

PPMStereo starts from an existing iterative stereo backbone (DynamicStereo) and adds a memory system that mimics how people make decisions in two steps: first pick the important items, then play with how much weight to give each.

For each new video frame, the model extracts features from the left and right images, builds a matching-cost representation, and builds a context representation. It also keeps a "vanilla memory" containing key/value features from every frame seen so far. From that large memory, the pick step scores each stored frame using two signals: (1) a confidence score, learned by a small network trained to predict where the disparity estimate is likely correct, and (2) a redundancy-aware relevance score that combines how similar the stored frame is to the current query with a decay factor that penalizes frames that have been reused too often. The top-K highest-scoring frames are kept in a small "dynamic memory."

The play step then normalizes those quality scores into soft weights, adds temporal position encodings (since the selected frames are no longer in order), and performs attention-based reading from the dynamic memory. The resulting aggregated cost feature is combined with the current cost and context features and fed to a GRU, which iteratively refines a residual disparity estimate. The whole system is trained end-to-end with a disparity loss plus a confidence loss, on 8 A100 GPUs for roughly 4.5 days.

Why This Matters

Research impact. This work reframes dynamic stereo matching as a memory management problem rather than an attention-window problem. It shows that a compact, quality-filtered memory can outperform much more expensive all-frame or optical-flow-based aggregation, and it adapts ideas from video segmentation and optical flow literature (XMem, RMem, MemFlow) in a way that is specifically tuned for disparity estimation.

Real-world applications:

• Augmented and virtual reality — flickering depth breaks immersion; smooth per-frame depth is essential for correct occlusion and rendering.
• Autonomous driving — temporally stable depth helps tracking, path planning, and obstacle detection in dynamic scenes.
• Robotics and manipulation — consistent 3D perception is needed when robots interact with moving objects or their own moving cameras.
• Video post-production and 3D content creation — stable depth enables reliable 2D-to-3D conversion, depth-based effects, and volumetric capture.

Industry relevance. The method is trained on synthetic data but generalizes to real footage, and it is cheaper than the previous best approach. That combination — accuracy, temporal stability, and efficiency — is exactly what deployment on embedded or automotive hardware demands.

Future Directions

• Scaling to much longer sequences. The current evaluation uses 20-frame clips; whether the memory buffer remains effective over minutes-long video is untested.
• Adaptive memory sizing. K is fixed at 5 empirically; a learned or content-adaptive K could capture more context when scenes change rapidly and save compute when they do not.
• Extending Pick-and-Play to other video tasks. The pick-and-play principle is task-agnostic and could plausibly be applied to optical flow, video segmentation, and multi-object tracking.
• Better pretrained representations. The authors' own PPMStereo_VDA variant, which swaps in a Video Depth Anything encoder, already shows gains; exploring stronger video foundation backbones is an obvious next step.

Target Audience

Graduate students and researchers in computer vision working on stereo matching, depth estimation, or video-based 3D perception will get the most value, followed by engineers building AR/VR, autonomous driving, or robotics systems who need temporally stable depth from stereo cameras. Readers should be comfortable with cost volumes, attention mechanisms, and GRU-based iterative refinement.

Authors’ abstract

Temporally consistent depth estimation from stereo video is critical for real-world applications such as augmented reality, where inconsistent depth estimation disrupts the immersion of users. Despite its importance, this task remains challenging due to the difficulty in modeling long-term temporal consistency in a computationally efficient manner. Previous methods attempt to address this by aggregating spatio-temporal information but face a fundamental trade-off: limited temporal modeling provides only modest gains, whereas capturing long-range dependencies significantly increases computational cost. To address this limitation, we introduce a memory buffer for modeling long-range spatio-temporal consistency while achieving efficient dynamic stereo matching. Inspired by the two-stage decision-making process in humans, we propose a \textbf{P}ick-and-\textbf{P}lay \textbf{M}emory (PPM) construction module for dynamic \textbf{Stereo} matching, dubbed as \textbf{PPMStereo}. PPM consists of a `pick' process that identifies the most relevant frames and a `play' process that weights the selected frames adaptively for spatio-temporal aggregation. This two-stage collaborative process maintains a compact yet highly informative memory buffer while achieving temporally consistent information aggregation. Extensive experiments validate the effectiveness of PPMStereo, demonstrating state-of-the-art performance in both accuracy and temporal consistency. % Notably, PPMStereo achieves 0.62/1.11 TEPE on the Sintel clean/final (17.3\% \& 9.02\% improvements over BiDAStereo) with fewer computational costs. Codes are available at \textcolor{blue}{https://github.com/cocowy1/PPMStereo}.