Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency

AI summary

Overview

Research area: Computer vision, specifically egocentric (Ego) and exocentric (Exo) video understanding, action anticipation, and test-time adaptation (TTA).

Technical level: Advanced. The paper assumes familiarity with CLIP-style vision-language models, prototype and memory-bank methods, entropy-based test-time adaptation, and multi-label video classification.

Scope: The paper defines a new task (TE²A³) for adapting a source-view-trained action anticipation model online, at test time, to an unlabeled target view, and proposes a network (DCPGN) built from a multi-label prototype memory module and a visual-textual consistency module, evaluated on two benchmarks.

What This Paper Is About

Models trained to anticipate actions in one camera view (for example, watching someone cook from an Exo perspective) break down when applied to the other view, because the two views differ in object layout and in the timing of activity progress. Prior Ego-Exo adaptation methods solve this by requiring target-view data during training — either labeled data for fine-tuning or unlabeled data for unsupervised domain adaptation — which adds data collection and compute costs.

This paper asks whether the model can instead be adjusted online during testing, using only an unlabeled stream of target-view videos, and does so for action anticipation (predicting upcoming noun and verb classes) rather than for single-label image classification.

Key Contributions

1. First formulation of the TE²A³ task. The paper defines Test-time Ego-Exo Adaptation for Action Anticipation, in which a model trained on a labeled source view is adapted online at test time to anticipate actions in the unlabeled target view. The authors state this is the first exploration of this task.

2. The Dual-Clue enhanced Prototype Growing Network (DCPGN). A framework that progressively accumulates multi-label knowledge and integrates clues from different modalities for test-time Ego-Exo adaptation and action anticipation. Source-view training uses a BCE loss on the source-view labeled data, with a frozen CLIP (ViT-L/14) visual encoder and a trainable TA3N anticipation network.

3. A Multi-Label Prototype Growing Module (ML-PGM). It assigns Top-K pseudo labels to video-level representations, reweights them by confidence to compute per-class prototypes, and maintains class-wise memory banks with an entropy priority queue strategy. This is designed to prevent the model from collapsing onto the single most confident class.

4. A Dual-Clue Consistency Module (DCCM) and a new benchmark. A lightweight narrator (GRU units with an attention mechanism) generates textual descriptions as temporal clues to complement a final-frame visual clue; their logits are tied together with a bidirectional KL divergence loss. The authors also construct the EgoMe-anti benchmark from the EgoMe dataset to evaluate the task.

Main Findings

• State-of-the-art results on both benchmarks. Under the Exo2Ego setting on EgoMe-anti, DCPGN reaches 79.03 (Noun) and 43.84 (Verb), surpassing the second-place TCA by 1.80% in noun anticipation and outperforming ML-TTA by 6.92% in verb anticipation. Under Ego2Exo on EgoMe-anti, DCPGN reaches 72.01 (Noun) and 40.10 (Verb), gains of 2.49% and 5.71%.

• Large margins on EgoExoLearn. DCPGN reaches 46.26 (Noun) and 42.98 (Verb) under Exo2Ego, and 48.48 (Noun) and 46.51 (Verb) under Ego2Exo, outperforming ML-TTA by 9.91% and 5.31% under Exo2Ego, and by 5.52% and 6.08% under Ego2Exo.

• All TTA methods beat no adaptation, but by modest amounts. The source-view model without adaptation scores 71.94/32.46 (Exo2Ego) and 64.24/30.07 (Ego2Exo) on EgoMe-anti; competing TTA methods such as Tent, TPT, VITTA, TDA, ZERO, TCA and ML-TTA improve on this but remain well below DCPGN.

• The dual-clue consistency loss contributes. Removing it drops performance by 0.36%, 1.12%, 0.72%, 0.90% on EgoMe-anti and by 1.46%, 0.25%, 0.51%, 0.67% on EgoExoLearn, showing its role in correlating cross-modality clues.

• Visual and textual clues play complementary roles. Removing visual clues hurts noun anticipation more, while removing textual clues causes a larger drop in verb anticipation, consistent with visual clues carrying spatial object information and textual clues carrying temporal activity progress.

• Confidence-based reweighting matters. Removing it decreases performance by 1.48%, 0.97%, 0.45%, 0.71% on EgoMe-anti and by 0.67%, 1.21%, 1.06%, 1.24% on EgoExoLearn, indicating it mitigates interference from potential negative classes.

• Multi-label assignment is essential. Assigning only the Top-1 class instead of Top-K causes drops of 1.89%, 4.40%, 3.25%, 4.50% on EgoMe-anti and 3.06%, 4.36%, 4.98%, 4.30% on EgoExoLearn.

• Optimal K differs by benchmark. The best K is 3 on EgoMe-anti and 5 on EgoExoLearn, which the authors attribute to denser action packing in EgoExoLearn (average actions per time segment: 2.46 versus 1.70). With K=1, performance deteriorates notably in all settings.

• Modest added complexity. The baseline model has 251.18 M parameters and 367.55 GFLOPs. ML-PGM adds 8.54 M parameters and 0.00 GFLOPs; narration generation adds 2.38 M parameters and 0.03 GFLOPs; textual clue feature extraction adds 54.04 M parameters and 4.06 GFLOPs.

• Qualitative evidence of representation quality. t-SNE visualizations of memory-bank representations and prototypes for five dominant EgoExoLearn classes show a single-label variant sidelining most representations (few assigned to class 16), an un-reweighted variant producing prototypes that are extremely close, and DCPGN producing clearly distinguishable classes and prototypes.

Methodology in Plain English

The setup is deliberately asymmetric: one view has labels, the other does not. The authors pick one annotated view as the "source view" and train a model there using standard supervised binary cross-entropy loss. The other view becomes the "target view," and its videos arrive only as an unlabeled online stream during testing. Two settings are studied: Exo2Ego and Ego2Exo.

The source model uses a frozen CLIP (ViT-L/14) encoder for frame features and a trainable TA3N network for anticipation. Each observation clip is 2 seconds long, taken 1 second before the action starts, with 5 frames sampled uniformly. The visual feature dimension is 768 and the anticipation network dimension is 512.

At test time, the first module (ML-PGM) takes each clip's video-level representation, ranks the predicted classes, keeps the Top-K as pseudo labels, and computes the entropy of the prediction as a reliability signal. Each class has a memory bank of capacity 500. New representations and confidence scores enter the bank; when a bank is full, only the entries with the lowest N entropies are kept, so the stored knowledge becomes more reliable over time. Prototypes are then computed as confidence-weighted averages of the stored representations, and a prototype classifier scores each incoming sample by similarity to these prototypes. This design counters the tendency of standard entropy-based TTA to obsess over one confident class.

The second module (DCCM) attacks the view gap directly. The final frame of the observation serves as a visual clue, carrying object information but no sense of temporal progression. A lightweight narrator — trained on video-text pairs collected from existing datasets such as EgoExo-4D, EgoMe and EgoExoLearn, and implemented with GRU units plus attention — produces a short description that acts as a temporal clue. Both clues are encoded with CLIP's visual and textual encoders, compared against learnable prompt-augmented class embeddings, and turned into logits. A bidirectional KL divergence forces the visual and textual logit distributions to agree, explicitly linking spatial and temporal evidence across views. The final prediction is the prototype logits plus a weighted sum of the visual and textual logits.

Hyperparameters: K is 5 on EgoExoLearn and 3 on EgoMe-anti, memory capacity N is 500, μ₁ is 1.0, μ₂ is 0.5, α is 0.5, prompt length is 4. Testing uses batch size 64 with no data augmentation; prompts are optimized online with SGD at a learning rate of 1e-4 on EgoExoLearn and 5e-4 on EgoMe-anti.

For evaluation, the authors build EgoMe-anti from the EgoMe dataset (7902 video pairs of Exo observer and Ego follower, 82 hours total). They filter for correctly following pairs, extract verbs and nouns with the Spacy library, convert them to base form, fuse synonyms, and remove classes below 0.5% frequency, constraining both views to share the same label set. EgoExoLearn, containing 120 hours of daily and professional videos, is used with its official action anticipation benchmark. Both use class-mean Top-5 recall on noun and verb classes.

Why This Matters

Impact on research. The paper reframes Ego-Exo adaptation as an online test-time problem, removing the requirement that target-view data be available for training or fine-tuning. It also shows that standard TTA methods built for single-label, image-level classification transfer poorly to multi-label video anticipation, where each event typically involves several atomic actions. The EgoMe-anti benchmark adds a paired observer/follower resource with noun and verb labels derived from descriptive annotations. Code is released at https://github.com/ZhaofengSHI/DCPGN.

Real-world applications (from the paper's framing):

• Human-robot cooperation, where a robot must understand and take over a task after observing a human from a different viewpoint.
• Embodied AI, where agents need to follow instructions across perspectives.
• Multi-robot collaboration, cited specifically for the Ego2Exo direction.
• AI assistance systems that forecast upcoming fine-level actions to support a user.

Industry relevance. The "no retraining" property matters wherever collecting and labeling target-view video is expensive or impossible, such as deployment in new camera setups or new environments. The reported overhead is small relative to the baseline, supporting practical online deployment.

Future Directions

• Where does the narrator's domain coverage stop? The narrator is trained on video-text pairs from existing datasets and is frozen during testing. How it behaves on target domains or action vocabularies not represented in those datasets is not reported.
• Can prototype instability be reduced further? The paper notes that ML-PGM suffers from inevitable prototype instability in the initial few adaptation steps and introduces DCCM partly to compensate. A more direct solution to cold-start instability remains open.
• Adaptive K instead of benchmark-specific K. The optimal K differs between EgoMe-anti (3) and EgoExoLearn (5), and the authors set it dynamically per benchmark based on action density. Choosing K automatically per sample or per stream was not explored.
• Streaming and ordering robustness. Adaptation is driven by an online stream of unlabeled target data with a fixed batch size of 64. Sensitivity to stream ordering, batch size, or non-stationary distributions is not evaluated in the reported content.

Target Audience

This paper suits researchers and graduate students working on test-time adaptation, egocentric and exocentric video understanding, video-language models, and action anticipation or planning. It is also relevant to practitioners building robot-assistance or embodied-AI systems who need view-robust anticipation without target-domain retraining, and to those who need a multi-label, video-level counterpart to the largely image-level, single-label TTA literature. Readers should already be comfortable with CLIP, memory banks and prototype classifiers, entropy-based adaptation, and Top-K recall evaluation; the paper is not an introductory treatment.

Authors’ abstract

Efficient adaptation between Egocentric (Ego) and Exocentric (Exo) views is crucial for applications such as human-robot cooperation. However, the success of most existing Ego-Exo adaptation methods relies heavily on target-view data for training, thereby increasing computational and data collection costs. In this paper, we make the first exploration of a Test-time Ego-Exo Adaptation for Action Anticipation (TE$^{2}$A$^{3}$) task, which aims to adjust the source-view-trained model online during test time to anticipate target-view actions. It is challenging for existing Test-Time Adaptation (TTA) methods to address this task due to the multi-action candidates and significant temporal-spatial inter-view gap. Hence, we propose a novel Dual-Clue enhanced Prototype Growing Network (DCPGN), which accumulates multi-label knowledge and integrates cross-modality clues for effective test-time Ego-Exo adaptation and action anticipation. Specifically, we propose a Multi-Label Prototype Growing Module (ML-PGM) to balance multiple positive classes via multi-label assignment and confidence-based reweighting for class-wise memory banks, which are updated by an entropy priority queue strategy. Then, the Dual-Clue Consistency Module (DCCM) introduces a lightweight narrator to generate textual clues indicating action progressions, which complement the visual clues containing various objects. Moreover, we constrain the inferred textual and visual logits to construct dual-clue consistency for temporally and spatially bridging Ego and Exo views. Extensive experiments on the newly proposed EgoMe-anti and the existing EgoExoLearn benchmarks show the effectiveness of our method, which outperforms related state-of-the-art methods by a large margin. Code is available at \href{https://github.com/ZhaofengSHI/DCPGN}{https://github.com/ZhaofengSHI/DCPGN}.