RefAtomNet++: Advancing Referring Atomic Video Action Recognition using Semantic Retrieval based Multi-Trajectory Mamba

AI summary

Overview

• Research area: Computer vision — specifically language-guided video understanding, cross-modal grounding, and fine-grained human action recognition.
• Technical level: Intermediate. Readers should be comfortable with vision-language models (CLIP/BLIP-style architectures), attention mechanisms, and basic video recognition concepts. The Mamba/state-space modeling component is explained at a level accessible to readers familiar with transformers.
• Scope: This paper extends a prior ECCV 2024 work by introducing a larger benchmark dataset (RefAVA++) and a new model (RefAtomNet++) for recognizing fine-grained actions of a specific person identified through a natural language description.

What This Paper Is About

Most action recognition systems in multi-person videos require manually cropped windows or pre-detected regions of interest for every person, then post-processing to isolate the individual someone actually cares about. This paper defines and advances the task of Referring Atomic Video Action Recognition (RAVAR), where a video and a text description (e.g., "the woman in the red jacket on the left") are given together, and the model must both locate that person and classify their fine-grained, atomic-level actions. The goal of this work is to build a larger benchmark for this task and a model that aligns text and video more precisely than previous approaches.

Key Contributions

1. A doubled benchmark dataset (RefAVA++). The authors extend their earlier RefAVA dataset from roughly 36,600 annotated persons to 75,111 annotated persons across 2,950,830 frames, drawn from 33,495 video clips spanning 231 movies from the AVA dataset. The test set uses movie-level partitioning so that test movies never appear in training or validation, forcing genuine generalization rather than memorization of actors or scenes.

2. A systematic benchmark of 15 baselines. Baselines are reformulated from four related domains — atomic action localization (I3D, X3D, MViTv2, Hiera, VideoMAE), video question answering (a GPT-based AskAnything model and a transformer VQA architecture), video-text retrieval (XCLIP, CLIP4CLIP, BLIPv2, MeVTR), and single-frame/VOS foundation models (SAM, DETR, REFCLIP, and a segmentation encoder) — all adapted to output both a bounding box and action labels.

3. A new model, RefAtomNet++, built on BLIPv2. It introduces (a) a multi-trajectory semantic-retrieval Mamba that builds per-keyword and per-attribute visual trajectories across time and aggregates them with a state-space model, and (b) a multi-hierarchical semantic-aligned cross-attention module that fuses these trajectories back with the original video tokens.

4. State-of-the-art accuracy with lower parameter count. RefAtomNet++ outperforms its predecessor RefAtomNet on RefAVA and sets new records on RefAVA++, while using roughly 93M fewer parameters than RefAtomNet.

Main Findings

• RefAVA results: RefAtomNet++ reaches 43.71% / 42.52% mIoU, 56.83% / 59.81% mAP, and 71.27% / 75.72% AUROC on the validation and test sets.

• RefAVA++ results: On the new, larger dataset it achieves 39.12% / 38.58% mIoU, 58.24% / 58.84% mAP, and 72.58% / 73.28% AUROC on validation and test, again leading prior methods.

• Clear gains over RefAtomNet on RefAVA test: +6.10% mIoU, +2.29% mAP, and +1.77% AUROC, while reducing parameters by 93M — a rare combination of better accuracy and greater efficiency.

• Backbone-agnostic improvements: When swapped onto an XCLIP backbone, RefAtomNet++ still lifts test performance over RefAtomNet from 36.61% to 42.24% mIoU, 48.59% to 53.03% mAP, and 66.47% to 70.50% AUROC, and also surpasses XCLIP used alone.

• Earlier model remains a strong baseline but is limited: RefAtomNet continues to beat the 15 domain baselines on RefAVA++, but its agent-attention mechanism highlights salient features without achieving precise semantic-to-visual alignment, which is the specific gap RefAtomNet++ targets.

• Held-out movies are hard: The strict movie-level train/test split means reported numbers reflect generalization to unseen films, casts, and environments rather than to familiar visual contexts.

• Text descriptions deliberately exclude actions: Annotations cover age, gender, clothing, appearance, and relative position, but never mention the action itself, preventing the model from shortcutting to the label through textual hints.

Methodology in Plain English

The model starts from BLIPv2, a pretrained vision-language system that produces a set of visual tokens from the video and a set of text tokens from the reference sentence. The key idea is that a single global sentence embedding is too coarse to pin down which visual evidence matters, so the authors build three complementary levels of semantic guidance.

At the holistic-sentence level, the entire reference sentence is encoded as one semantic vector. At the partial-keyword level, common stop words like "the" and "is" are stripped out, and the remaining content words are encoded individually, giving fine-grained anchors such as "red," "jacket," or "left." At the scene-attribute level, an object detector (DETR) is run on the middle frame of the clip; each detected object's category embedding is concatenated with its bounding box coordinates, producing tokens that describe what objects exist and where they sit in the scene.

For each of these keyword and attribute tokens, the model then walks through the video timestep by timestep and, at each step, picks the single visual token that is most similar in embedding space. This cheap nearest-neighbor selection produces a "trajectory" — a sequence of visual tokens followed over time by one semantic anchor. Multiple anchors produce multiple trajectories. A Mamba layer, which is a state-space sequence model that maintains a running hidden state with linear cost, then summarizes each trajectory. Mamba is chosen over attention because it treats the sequence as a continuous dynamical system, which matches the smooth evolution of real human motion better than discrete pairwise weighting.

Finally, these trajectory-derived tokens and the original video tokens are fused by a multi-hierarchical semantic-aligned cross-attention module. The three semantic levels act as queries, the original spatio-temporal visual tokens supply keys and values, and a set of learnable prompt queries is added at each level for flexibility. Spatial and temporal aggregation run independently, and separate bounding-box regression and action classification heads produce predictions from each branch; the two branch outputs are averaged into the final answer.

Why This Matters

• Research impact: The paper shifts referring understanding from static objects toward human-centric, temporally extended reasoning. It supplies a large, carefully partitioned benchmark that makes comparisons between action recognition, VQA, retrieval, and segmentation methods meaningful under one task definition, and it demonstrates that state-space models are a viable alternative to attention for multi-trajectory token aggregation in video.

• Assistive technology: A system that can locate "the man with the cane on the right" and describe what he is doing could give visually impaired users narration tailored to the person they actually care about, rather than an undifferentiated description of the whole scene.

• Rehabilitation and elder care: Monitoring a specific patient's exercises or daily movements requires identifying the right individual in a shared room and judging the fine-grained quality of their motion, which is exactly the RAVAR formulation.

• Human-robot collaboration: Robots following instructions such as "hand the tool to the woman in the blue shirt" need to ground a person by description and then anticipate their next action — the two outputs RAVAR produces jointly.

• Skill assessment and sports or workplace analytics: Coaches and safety reviewers often care about one athlete or worker at a time; language-guided targeting removes the need to manually crop or track each person separately.

• Industry relevance: Video platforms, robotics companies, healthcare monitoring vendors, and AR/VR developers all consume video-language models. A method that improves accuracy while cutting 93M parameters is attractive for on-device or latency-sensitive deployment, and a benchmark with held-out movies gives industry teams a fair way to compare competing models.

Future Directions

• Further scaling and domain diversity: RefAVA++ is built entirely from movie footage. Extending to surveillance, egocentric, sports, and clinical video would test whether the multi-trajectory retrieval mechanism generalizes beyond cinematic content.

• Multimodal references: The current task uses text only. Adding speech, gestures, or user clicks as reference signals could make the formulation more practical for interactive systems.

• Longer videos and complex references: References with relational or negational structure ("the person who is not holding the phone") and clips far longer than the current ones remain largely untested.

• Real-time efficiency: The paper reports parameter savings but not latency or throughput. Measuring and optimizing inference speed would matter for robotics and assistive deployment.

• Rethinking evaluation: mIoU, mAP, and AUROC are adapted from detection and action localization. Whether they capture the quality of language grounding remains an open methodological question the field has not settled.

Target Audience

Researchers and graduate students working on video-language understanding, referring expression grounding, and fine-grained action recognition will find the benchmark and baseline suite immediately useful. Practitioners building person-centric video analytics, assistive systems, or human-robot interaction pipelines will benefit from the efficiency results and the practical framing of the task. Readers with a general computer vision background can follow the high-level argument, but the methodology sections assume familiarity with vision-language transformers and state-space sequence models.

Authors’ abstract

Referring Atomic Video Action Recognition (RAVAR) aims to recognize fine-grained, atomic-level actions of a specific person of interest conditioned on natural language descriptions. Distinct from conventional action recognition and detection tasks, RAVAR emphasizes precise language-guided action understanding, which is particularly critical for interactive human action analysis in complex multi-person scenarios. In this work, we extend our previously introduced RefAVA dataset to RefAVA++, which comprises >2.9 million frames and >75.1k annotated persons in total. We benchmark this dataset using baselines from multiple related domains, including atomic action localization, video question answering, and text-video retrieval, as well as our earlier model, RefAtomNet. Although RefAtomNet surpasses other baselines by incorporating agent attention to highlight salient features, its ability to align and retrieve cross-modal information remains limited, leading to suboptimal performance in localizing the target person and predicting fine-grained actions. To overcome the aforementioned limitations, we introduce RefAtomNet++, a novel framework that advances cross-modal token aggregation through a multi-hierarchical semantic-aligned cross-attention mechanism combined with multi-trajectory Mamba modeling at the partial-keyword, scene-attribute, and holistic-sentence levels. In particular, scanning trajectories are constructed by dynamically selecting the nearest visual spatial tokens at each timestep for both partial-keyword and scene-attribute levels. Moreover, we design a multi-hierarchical semantic-aligned cross-attention strategy, enabling more effective aggregation of spatial and temporal tokens across different semantic hierarchies. Experiments show that RefAtomNet++ establishes new state-of-the-art results. The dataset and code are released at https://github.com/KPeng9510/refAVA2.