Robust Ego-Exo Correspondence with Long-Term Memory

AI summary

Overview

Research area: Computer vision — specifically object-level correspondence between egocentric (first-person) and exocentric (third-person) video, built on top of the Segment Anything Model 2 (SAM 2).

Technical level: Intermediate to Advanced. The paper assumes familiarity with video object segmentation, memory banks, transformer attention, and Mixture-of-Experts concepts, though the core ideas can be understood without implementing the math.

Scope: The paper proposes LM-EEC, a SAM 2-based framework with a Memory-View Mixture-of-Experts fusion module and a dual compressed long-term memory bank, evaluated on the EgoExo4D benchmark.

What This Paper Is About

The paper tackles ego-exo correspondence: given annotated object masks in one video view (say, the first-person view), find the same object's masks in a synchronized video from a different viewpoint (say, the third-person view). This is hard because the two views differ drastically in angle, objects are often occluded, many targets are small, and videos are long. The goal is a model that fuses information from both views effectively and remembers long-term context far beyond what SAM 2's short, first-in-first-out memory retains.

Key Contributions

1. Memory-View Mixture-of-Experts (MV-MoE) module: A dual-branch routing mechanism that assigns adaptive contribution weights to the memory-aware expert feature and the view-specific expert feature along both channel and spatial dimensions, replacing SAM 2's simple additive fusion of memory features and prompt embeddings.

2. Dual-memory bank system: Separate ego and exo memory banks, rather than a single unified bank, so that the two views' differing viewpoints, motion patterns, and visual appearance are stored and managed independently.

3. View-specific memory compression strategy: A feature-point-level compression that computes Euclidean distances between spatially corresponding feature points in temporally adjacent memory frames, merges the most redundant pair by averaging, and thereby reduces memory length by one while preserving long-term information.

4. State-of-the-art results on EgoExo4D: LM-EEC outperforms prior VOS models and SAM 2-based baselines on the EgoExo4D test set in both Ego2Exo and Exo2Ego settings, with the improvement over the base model most pronounced in Exo2Ego.

Main Findings

• Ego2Exo (exo query mask, Ego view target) test set: LM-EEC reaches IoU 54.98, LE 0.017, CA 0.778, BA 64.22, versus the base model's 52.13 / 0.024 / 0.734 / 61.56 and the strongest prior baseline among the compared models, SimVOS, at 38.26 / 0.090 / 0.481 / 57.21.

• Exo2Ego (ego query mask, Exo view target) test set: LM-EEC reaches IoU 65.77, LE 0.031, CA 0.774, BA 58.14, versus the base model's 57.27 / 0.047 / 0.677 / 57.11 and the strongest prior baseline, Cutie, at 47.52 / 0.070 / 0.579 / 70.71.

• Balanced accuracy is not the top score: XSegTx achieves the highest BA (66.31 in the Ego-query setting and 82.01 in the Exo-query setting), which the authors attribute to XSegTx being a co-segmentation model with specialized data augmentation.

• MV-MoE beats simple addition (validation, Ego2Exo): MV-MoE gives IoU 0.5925, LE 0.0198, CA 0.8006; the base model with simple addition gives 0.5673 / 0.0258 / 0.7643; and dropping the other view's prompt gives 0.5691 / 0.0276 / 0.7660.

• Both memory banks matter: Removing the ego memory gives 0.5748 / 0.0220 / 0.7837; removing the exo memory gives 0.5420 / 0.0293 / 0.7495; keeping all memory stores gives 0.5925 / 0.0198 / 0.8006.

• Compression strategy comparison: FIFO 0.5823 / 0.0214 / 0.7874, cluster selection 0.5867 / 0.0208 / 0.7942, IoU-based selection 0.5880 / 0.0204 / 0.7948, and the proposed feature-point-level compression 0.5925 / 0.0198 / 0.8006.

• Frame and memory-size ablations: Training with 4 frames gives 0.5857 / 0.0204 / 0.7931, 8 frames gives 0.5925 / 0.0198 / 0.8006, and 10 frames gives 0.5927 / 0.0201 / 0.8003; memory size 4 gives 0.5921 / 0.0205 / 0.8004, size 6 gives 0.5925 / 0.0198 / 0.8006, and size 8 gives 0.5963 / 0.0196 / 0.8025.

• Long-term association accuracy improves most on long videos: Association accuracy (fraction of frames with IoU above 0.5) is 0.5580 baseline vs 0.6054 for LM-EEC on videos under 200 frames, 0.6260 vs 0.6732 on 200–600 frames, and 0.4724 vs 0.6150 on videos over 600 frames — an improvement of approximately 14% on the longest videos.

• MV-MoE transfers to other backbones: Adding MV-MoE to STCN raises IoU from 27.39 to 30.51 (LE 0.109 to 0.083, CA 0.378 to 0.436, BA 61.97 to 62.81); adding it to QDMN raises IoU from 27.03 to 28.75 (LE 0.108 to 0.100, CA 0.346 to 0.376, BA 63.16 to 64.02).

• Scenario-dependent difficulty: Among EgoExo4D activity categories (Cooking, Basketball, Bike Repair, Health, Music), Basketball is generally easier due to limited variation in object shape and appearance, while Cooking and Bike Repair are harder because objects vary more across views — and in those harder scenarios, LM-EEC's advantage over baselines grows.

• Failure mode: The model may segment visually similar background objects when the target object has disappeared from the scene, because existing baselines do not explicitly model object disappearance during training and SAM 2 lacks an explicit mechanism for it.

Methodology in Plain English

LM-EEC starts from the official SAM 2 base model and changes two things.

Fusion. In SAM 2, the memory-derived feature and the prompt embedding are simply added together before decoding. The authors argue these two signals highlight different regions and have different distributions, so one can overwhelm the other. They instead treat them as two "experts" and build a router that first computes a channel-wise weight for each expert (concatenate the two features, global-average-pool, pass through two-layer MLPs with ReLU and sigmoid) and applies it residually, then computes a spatial weight for each expert (concatenate the channel-refined features, apply two parallel Conv-ReLU-Conv-Sigmoid branches) and applies that residually too. The two refined features are then summed to form the target feature fed to the mask decoder. The authors describe this as dense, feature-level expert routing rather than the sparse sub-network routing of conventional MoE.

Memory. Instead of one shared memory bank, ego and exo features go into separate banks. Each bank has a fixed capacity; when a new frame would exceed the threshold M, a compression step runs. For every spatial location, the algorithm computes Euclidean distances between corresponding feature points in temporally adjacent memory frames, picks the adjacent pair with the minimum distance (i.e., the most temporally redundant), and replaces that pair with its average. This shrinks the bank by one frame's worth of features while keeping long-term content, and it is applied per view.

Training and setup. The model uses SAM 2 base with these modifications, samples 8 consecutive frames per object from ego-exo pairs, sets memory bank size to 6, resizes frames to 480×480 (following prior EgoExo4D practice), trains all modules jointly from the pre-trained SAM 2 checkpoint without freezing anything, for 60 epochs on 8 NVIDIA A100 GPUs with batch size 32, and evaluates on a single V100 GPU at roughly 8.4 FPS. Memory compression is applied only during inference. Experiments use the EgoExo4D benchmark: approximately 5.5K annotated objects across 1.3K temporally synchronized video pairs, roughly 4 million annotated frames, and 742K ego-view plus 1.1M exo-view paired masks, split into 756 training, 202 validation, and 291 test videos. Metrics are IoU, Location Error (LE), Contour Accuracy (CA), and Existence Balance Accuracy (BA).

Why This Matters

Impact on research. The paper shows that a general-purpose video segmentation foundation model (SAM 2) can be adapted to a fundamentally cross-view task if its fusion and memory mechanisms are redesigned, rather than replaced. It provides evidence that separate memory banks for divergent viewpoints and feature-point-level memory compression are both individually valuable, and it demonstrates that the routing module is portable to other backbones (STCN, QDMN). It also names a concrete unsolved problem: explicitly modeling when a target object is absent from a view.

Real-world applications:

• Augmented reality coaching — smart glasses that guide a wearer through a physical task by relating what the wearer sees to what an instructor or third-person camera sees.
• Robotics skill acquisition — a robot observing a human to learn dexterous manipulation with less physical trial and error.
• Remote collaboration and assistance — an expert viewing a wide scene while helping a user who is wearing a first-person camera.
• Assistive technologies that need to reason about objects across differing viewpoints.

Industry relevance. The pipeline runs on top of SAM 2, trains in 60 epochs (12K iterations) on 8 A100s, and infers at approximately 8.4 FPS on a single V100 — a practical scale. The authors note privacy and surveillance risks, since associating first-person and third-person views could be misused for unauthorized tracking, and state the work must be applied only in ethical, privacy-compliant contexts.

Future Directions

• Explicit object-disappearance modeling: The authors suggest targeted data augmentation during training or an additional branch that predicts object presence or absence in the target view, to fix the failure case where visually similar background objects are segmented after the target leaves the scene.
• Resolving the metric trade-off: The paper observes an inherent trade-off among IoU, contour accuracy, location score, and balanced accuracy, and does not resolve it — a natural open problem.
• Extending MV-MoE to more backbones: Modularity was demonstrated on STCN and QDMN; the authors suggest it could be adapted to other segmentation architectures and future SAM 2-like frameworks.
• Scaling memory further: The memory-size ablation shows size 8 outperforming the default of 6, so extending long-term memory capacity while controlling cost remains an open direction.

Target Audience

Researchers and graduate students working on video object segmentation, cross-view or multi-view correspondence, and foundation-model adaptation will get the most from this paper. It is also relevant to practitioners building AR, robotics, or assistive systems that need to link first-person and third-person observations. Readers should already be comfortable with SAM-style segmentation architectures, memory-bank mechanisms, and attention-based feature fusion; the ablation tables make the paper accessible to those who want empirical guidance more than architectural derivation.

Authors’ abstract

Establishing object-level correspondence between egocentric and exocentric views is essential for intelligent assistants to deliver precise and intuitive visual guidance. However, this task faces numerous challenges, including extreme viewpoint variations, occlusions, and the presence of small objects. Existing approaches usually borrow solutions from video object segmentation models, but still suffer from the aforementioned challenges. Recently, the Segment Anything Model 2 (SAM 2) has shown strong generalization capabilities and excellent performance in video object segmentation. Yet, when simply applied to the ego-exo correspondence (EEC) task, SAM 2 encounters severe difficulties due to ineffective ego-exo feature fusion and limited long-term memory capacity, especially for long videos. Addressing these problems, we propose a novel EEC framework based on SAM 2 with long-term memories by presenting a dual-memory architecture and an adaptive feature routing module inspired by Mixture-of-Experts (MoE). Compared to SAM 2, our approach features (i) a Memory-View MoE module which consists of a dual-branch routing mechanism to adaptively assign contribution weights to each expert feature along both channel and spatial dimensions, and (ii) a dual-memory bank system with a simple yet effective compression strategy to retain critical long-term information while eliminating redundancy. In the extensive experiments on the challenging EgoExo4D benchmark, our method, dubbed LM-EEC, achieves new state-of-the-art results and significantly outperforms existing methods and the SAM 2 baseline, showcasing its strong generalization across diverse scenarios. Our code and model are available at https://github.com/juneyeeHu/LM-EEC.