Exploiting Textual Semantics for Robust Cross-View Object Correspondence

AI summary

Overview

Research area: Computer vision, specifically cross-view object correspondence (CVOC) — matching and segmenting the same object across videos filmed from different viewpoints (e.g., a first-person ego-centric camera and a third-person exo-centric camera).

Technical level: Intermediate. The paper assumes familiarity with segmentation backbones (SAM 2), attention mechanisms, and standard video segmentation metrics, but the core idea — using a written description of an object to help find it again from another camera — is intuitive.

Scope: The paper introduces TeSCo, a framework built on SAM 2 that generates a textual description of a target object from the query view and fuses that text into visual features to improve cross-view segmentation, evaluated on Ego-Exo4D and HANDAL-X.

What This Paper Is About

Given a video of an object from one camera angle along with masks marking that object, cross-view object correspondence asks a model to segment the same object in a synchronized video from a different angle. Existing methods rely almost entirely on visual features, which break down when viewpoint and appearance change drastically between the two cameras. This paper proposes supplementing those visual cues with a generated textual description of the object — capturing attributes like category, color, material, and texture — because such attributes stay relatively stable even when pixels do not.

Key Contributions

1. A text-guided CVOC framework (TeSCo). The paper introduces TeSCo, which exploits rich textual semantics of a target object as a complement to visual cues for cross-view object correspondence, built on top of SAM 2 and achieving state-of-the-art results on Ego-Exo4D and HANDAL-X.

2. Text-conditioned cross-modal fusion (TCF). A module that injects the textual feature into visual features through cross-attention followed by text-guided modulation, generating a channel-wise reweighting parameter γ and a shift parameter β from the text to emphasize target-relevant cues before a residual connection.

3. Iterative textual feature refinement (ITFR). A module that progressively adapts the query-view textual feature using search-view information across K cross-attention-plus-MLP iterations with residual connections, producing search-view-aware textual semantics before fusion.

4. A clear differentiation from prior text-based work (ObjectRelator). The paper argues TeSCo differs in three ways: it uses rich descriptions (category, color, texture) as feature-level guidance rather than only an object name as an LLM prompt; it uses a dedicated TCF module rather than plain cross-attention; and it refines text with search-view information via ITFR.

Main Findings

• State-of-the-art on Ego-Exo4D: TeSCo reaches 60.40% IoU (Cont.A 0.847, Loc.E 0.011) in the Ego→Exo setting and 70.30% IoU (Cont.A 0.808, Loc.E 0.023) in Exo→Ego, for an mIoU of 65.35%. The second-best method, LM-EEC, reaches 54.98%, 65.77%, and an mIoU of 60.38%.

• Gains over the visual-only baseline: The baseline (TeSCo with ITFR and TCF removed) scores 52.70% and 58.30% IoU with an mIoU of 55.50%, meaning TeSCo delivers absolute gains of 7.70%, 12.0%, and 9.85%.

• Large margin over the other text-using method: ObjectRelator achieves 35.30% and 40.30% IoU with an mIoU of 37.8%, so TeSCo improves by 25.10%, 23.24%, and 22.31% respectively.

• Strong cross-view generalization on HANDAL-X (zero-shot): Trained on Ego-Exo4D but tested on HANDAL-X, TeSCo reaches 85.10% IoU versus 78.80% for CCMP (a 6.30% margin) and 75.26% for the baseline (a 9.84% margin).

• Best result when retrained on HANDAL-X: TeSCo reaches 90.00% IoU, ahead of CCMP (85.00%), ObjectRelator (84.70%), PSALM (83.40%), and the baseline (83.94%).

• Both modules matter: Adding TCF alone lifts IoU from 52.70% to 58.74% (6.04% absolute); adding ITFR on top reaches 60.40%. The authors note ITFR cannot function alone because it is designed to adapt text for fusion.

• TCF beats simpler fusion: Addition-based fusion gives 56.43% IoU, standard cross-attention gives 58.26%, and TCF gives 60.40%.

• Both modulation parameters help: Removing γ and β degenerates TCF to standard cross-attention (58.26%). γ alone gives 59.73%, β alone gives 58.80%, and both together give 60.40%.

• Two refinement iterations are best: K=1 yields 59.90%, K=2 yields 60.40%, and K=3 drops to 59.29%, which the authors attribute to over-refinement.

• Inject text in both places: Injecting into the query view only gives 55.28% IoU, into temporal memory only gives 58.94%, and into both gives 60.40%.

• DAM outperforms Qwen2.5-VL for text generation: Qwen2.5-VL with box prompts gives 56.28% IoU, with masks 59.75%, while DAM with masks gives 60.40%. The paper states DAM performs very close to human-written manual text, though the manual-text score itself is not given a numeric value in the provided content.

Methodology in Plain English

Start with SAM 2, a strong existing video segmentation model, and keep its image encoder, memory encoder, memory attention, and mask decoder.

For each frame, use an off-the-shelf vision-language model (DAM) to write a detailed description of the target object given the query-view image and its mask. The paper generates a description per frame rather than one per video, reasoning that different moments reveal different visible attributes.

Encode that description with a text encoder (RoBERTa). Because some attributes visible in the query view — texture, for instance — may be ambiguous or invisible in the search view, run the text feature through ITFR: a few rounds of cross-attention where the text queries the search-view visual feature, each followed by a residual add and an MLP, so the text drifts toward what is actually discriminative in the current search frame.

Then run TCF: cross-attend the visual feature to the refined text, use MLPs on the text to produce a per-channel scale (γ) and shift (β), apply them to the fused feature, pass through an MLP, and add the original visual feature back as a residual. Do this twice — once for the query-view object-aware feature and once for the concatenated temporal memory feature.

Finally, feed both enhanced features into SAM 2's memory attention and mask decoder to predict the mask in the search-view frame, then update the query and search memory banks first-in-first-out for the next frame.

Training uses SAM 2's loss set (dice coefficient, focal mask, IoU, and binary cross-entropy losses summed over predicted search-view masks) for 60 epochs with batch size 12 (3 per GPU) on 4 Nvidia A6000 GPUs, AdamW, learning rates of 6×10⁻⁶ for the image encoder and 10⁻⁵ for the new modules and decoder, frames resized to 480×480, and K=2 in ITFR.

Why This Matters

The paper's core claim is that language supplies viewpoint- and appearance-invariant semantics that pixels cannot, which is a general recipe for any task where the same object must be recognized under severe transformation.

Real-world applications named in the paper:

• Robot learning — a robot observing a demonstration from one camera and reproducing it from another viewpoint.
• Augmented reality — aligning virtual content with a physical object seen from a different device's camera.
• Human-object interaction understanding — reasoning about hands and tools across ego and exo footage.
• Skill and activity analysis — the Ego-Exo4D scenarios the data covers include basketball, bike repair, cooking, health, and music.

Industry relevance: Multi-camera setups are routine in robotics, AR/VR headsets, sports analytics, and industrial monitoring. A method that maintains object identity across cameras without camera calibration or geometric constraints is easier to deploy than one relying on precise multi-view geometry, and the zero-shot result on HANDAL-X (85.10% IoU without training on that dataset) is evidence that the text signal transfers to new object domains.

Future Directions

• Temporal descriptions instead of frame-level ones. The authors explicitly flag that frame-level text degrades under occlusion or motion blur and propose using video temporal information to generate more stable descriptions.

• Efficiency. Text generation and text feature extraction add computational cost and resources; the authors list optimizing for resource-friendly, efficient CVOC as future work.

• Better text generation. DAM is chosen because it is trained for multi-granular region captioning, and it beats Qwen2.5-VL in the reported ablation — but the question of what description style best serves correspondence (per-frame vs. global, mask vs. box prompts) is still open.

• Beyond two viewpoints. The experiments cover ego↔exo pairs; whether the text-guided approach scales to many simultaneous cameras or to views with even less overlap is untested here.

Target Audience

Researchers and graduate students working on video object segmentation, multi-view video understanding, referring/vision-language segmentation, and vision-language model integration. It is also relevant to practitioners building multi-camera perception for robotics, AR, or sports and activity analytics who want a lightweight way to stabilize object identity across views. Readers should be comfortable with SAM-style architectures and attention fusion to get the most from the method details.

Authors’ abstract

Cross-view object correspondence (CVOC) aims to establish associations of the same target between query and search videos captured from different viewpoints. Existing methods often rely on visual cues for CVOC. Despite recent progress, these methods struggle in complicated scenarios with substantial viewpoint changes and appearance variations, due to lacking sufficient target information. Addressing this, we introduce TeSCo, a framework that exploits rich Textual Semantics of an object, in addition to its visual cues, for cross-view object Correspondence. Our key insight is, textual description of a target, by capturing diverse attributes such as category, texture, and color, provides viewpoint- and appearance-invariant semantics, which are complementary to visual cues and can hence enhance correspondence robustness in complicated scenes. Inspired by this, TeSCo first generates a textual expression of the target from the given query view using a vision-language model, and then extracts textual feature to enhance visual features for localization in the search view. To realize this, we introduce a text-conditioned cross-modal fusion (TCF) module that incorporates the textual feature into visual features with text-guided modulation, producing more robust multimodal target representation for object correspondence. Since not all textual cues are equally compatible with the target in current search-view frame, we propose an iterative textual feature refinement (ITFR) module that progressively adjusts textual feature using search-view information before the TCF module, enabling search-view-aware textual semantics to be fused into visual features and leading to better performance. In extensive experiments on Ego-Exo4D and HANDAL-X, TeSCo demonstrates state-of-the-art results and largely outperforms other models, validating its efficacy. Our code is released at https://github.com/VAIL-Fan/TeSCo.