Otter: Mitigating Background Distractions of Wide-Angle Few-Shot Action Recognition with Enhanced RWKV

AI summary

Overview

Research area: Computer vision, specifically few-shot action recognition (FSAR) and video understanding, with an architecture built on the Receptance Weighted Key Value (RWKV) recurrent model family.

Technical level: Advanced. The paper defines new modules by composing RWKV-5/6 style "mixing" units, defines its prototypes and loss terms mathematically, and evaluates under standard 5-way 1-shot / 5-shot episodic protocols.

Scope: The paper introduces a method called Otter (CompOund SegmenTation and Temporal Reconstructing RWKV) that targets background distractions in wide-angle videos by highlighting subjects and reconstructing degraded temporal relations, and validates it on SSv2, Kinetics, UCF101, HMDB51, and the wide-angle VideoBadminton dataset.

Paper metadata as given: arXiv:2511.06741v6 [cs.CV], dated 19 Mar 2026, published 2025-11-10, license CC BY 4.0. Code is listed at https://github.com/wenbohuang1002/Otter. Affiliations listed are Southeast University, Institute of Science Tokyo, The University of Newcastle, Hokkaido University, and Nanjing Normal University.

What This Paper Is About

Few-shot action recognition has to classify unseen actions from only a handful of labeled videos, which is hard when different actions look alike. The authors argue that wide-angle videos (defined in the paper as a field of view, FoV, exceeding 80 degrees) carry useful scene context, but that in these videos the subject occupies a small proportion of the frame and a lot of near-identical background obscures how the subject evolves over time. Their goal is a model that can operate on wide-angle samples without being dominated by background, while still recovering the temporal relations that are lost when many frames look alike.

Key Contributions

1. Compound Segmentation Module (CSM). Splits each frame into patches, learns adaptive per-patch weights that highlight the primary subject, and reassembles the patches in their original positions before feature extraction. This is intended to keep RWKV from latching onto secondary background content.

2. Temporal Reconstruction Module (TRM). Performs bidirectional scanning of frame features and reconstructs temporal relations through a weighted average of the scanning results, producing a temporal-enhanced prototype. This is intended to counteract temporal relation degradation in wide-angle videos.

3. Dual-prototype design. A regular prototype is combined with the temporal-enhanced prototype so that subject emphasis and temporal modeling are addressed at the same time, with the distances from the two prototypes combined by weighted averaging (with weights ω₁ = ω₂ = 0.5).

4. First use of RWKV for wide-angle FSAR, with state-of-the-art validation. The authors state Otter is the first attempt to use RWKV for wide-angle FSAR and report state-of-the-art accuracy on SSv2, Kinetics, UCF101, and HMDB51, plus extra evaluation on the wide-angle VideoBadminton dataset.

Main Findings

• Wide-angle samples are common but previously unexplored. FoV estimation on popular FSAR benchmarks reveals that approximately 35% of samples per dataset fall into the wide-angle category (FoV exceeding 80 degrees), yet this category "remains unexplored" per the authors.

• Both modules contribute, and together they contribute most. On SSv2 and Kinetics with the default backbone, the baseline with neither module reaches 54.6 / 69.2 (SSv2 1-shot / 5-shot) and 78.1 / 85.3 (Kinetics 1-shot / 5-shot). CSM alone raises this to 61.3 / 85.6 and 89.4 / 94.8; TRM alone gives 59.5 / 83.4 and 87.8 / 92.7; both together give 64.7 / 88.5 and 90.5 / 96.4.

• Patch size p = 56 is the best setting in CSM. Finer segmentation initially helps but then declines: p = 224 gives 62.7 / 86.4 on SSv2 and 87.7 / 94.6 on Kinetics; p = 112 gives 63.6 / 87.1 and 89.5 / 95.2; p = 56 gives 64.7 / 88.5 and 90.5 / 96.4; p = 28 drops to 64.1 / 87.9 and 90.2 / 95.8. Multi-scale configurations were worse (p ∈ {28, 56}: 64.2 / 88.1 and 90.1 / 96.1; p ∈ {56, 112}: 63.7 / 87.9 and 89.6 / 95.8). The authors attribute the multi-scale drop to redundant features and adopt p = 56, corresponding to 4 × 4 segmentation.

• Bidirectional scanning in TRM is necessary. Ordered scanning alone reaches 63.2 / 87.3 on SSv2 and 89.7 / 95.7 on Kinetics; reserved (reversed) scanning alone reaches 60.6 / 85.2 and 89.1 / 94.2 — the authors note reversed scanning alone even harms performance relative to ordered scanning, which they explain by confusion of direction-related actions. Combining both gives 64.7 / 88.5 and 90.5 / 96.4.

• State-of-the-art accuracy across backbones. With ImageNet-RN50, Otter reports SSv2 64.7 / 88.5, Kinetics 90.5 / 96.4, UCF101 96.8 / 99.2, HMDB51 88.1 / 89.8 (1-shot / 5-shot). With ImageNet-ViT: 67.2 / 89.9, 91.8 / 97.3, 97.7 / 99.4, 89.9 / 90.6. With ImageNet-ViR: 67.1 / 89.8, 91.7 / 96.8, 97.5 / 99.3, 89.5 / 90.5. For reference, the paper's own re-implementation of Manta (AAAI'25) with ImageNet-RN50 reports SSv2 63.4 / 87.4, and the abstract/introduction describes Otter improving over Manta on SSv2 1-shot "from 63.4% to 64.7%."

• Large gains on the wide-angle VideoBadminton dataset. In the VB → VB setting (train and test on VideoBadminton), Otter reports 71.2 / 75.8 versus MoLo 60.2 / 64.5, SOAP 63.5 / 66.9, and Manta 64.1 / 67.1. In the KI → VB setting (trained on Kinetics, tested on VideoBadminton), Otter reports 69.5 / 72.6 versus MoLo 58.9 / 61.7, SOAP 60.1 / 63.1, and Manta 62.1 / 65.3.

• More stable as FoV widens. Using PQDiff for outpainting magnification and an introduced distortion factor to simulate different CMOS sizes and focal lengths, the authors construct five distinct FoV levels. They report that recent methods decline sharply as FoV level increases, while Otter is negatively affected but with a "much more stable" downward trend. Exact per-level accuracy values are shown in a figure and are not given as numbers in the text provided.

• CAM visualizations show attention shifting to subjects. For a "smash" clip from VideoBadminton, models without Otter focus mostly on background and ignore the distant subject, whereas with Otter most focus moves to the subjects. The authors note Otter captures both badminton players rather than only the nearby subject.

• RWKV-5/6 is preferred over RWKV-4. In the supplementary study, the RWKV-4 configuration of the three mixing units reports 64.0 / 87.5 on SSv2 and 89.2 / 94.3 on Kinetics (1-shot / 5-shot); the authors state RWKV-5/6-based components perform better, so they use RWKV-5/6 as the basis. The remaining rows of that table are cut off in the provided content.

Methodology in Plain English

• Problem setup. Data are split into disjoint training, validation, and testing sets, and each split is further divided into a support set (at least one labeled sample per class) and a query set (unlabeled). The paper studies standard N-way K-shot episodes; experiments use 5-way 1-shot and 5-shot. Videos are represented as F successive frames (F = 8 in the experiments).

• Stage 1 — Segment and highlight before feature extraction. CSM divides each frame into H·W/p² patches. Three RWKV-derived units — Spatial Mixing (aggregates information across spatial locations), Time Mixing (combines current and past states), and Channel Mixing (mixes channels via receptance and value) — are applied to the patches. Their outputs are added back to the patch inputs as residuals, a convolution plus residual connection produces learnable per-patch weights, and those weights multiply the patch features. The weighted patches are placed back in their original frame positions, and a residual connection to the original frame yields the highlighted version.

• Stage 2 — Backbone feature extraction. The highlighted support and query frames go through a backbone (ResNet-50, ViT-B, VRWKV-B are reported in the comparison table; the implementation section also lists VMamba-B as one of the backbones initialized with ImageNet pre-trained weights). Feature dimension D is 2048.

• Stage 3 — Build two prototypes. The temporal-enhanced prototype is built by passing frame features through TRM: the sequence is scanned in one direction and in the reverse direction, each direction produced by Time/Channel Mixing plus a learned weight, and the two directional outputs are averaged and added to the original input as a residual. The regular prototype is built from the same frame features without TRM. Class prototypes are the average of support features over the K samples per class; distances from a query to each prototype are computed per class.

• Stage 4 — Score and train. The final distance is a weighted mean of the two prototype distances with ω₁ = ω₂ = 0.5, and the predicted class is the arg-min distance. The training objective combines a cross-entropy loss with cosine-similarity terms that push class prototypes apart, weighted by λ₀ = 0.8, λ₁ = λ₂ = 0.1. Optimization uses SGD with initial learning rate 10⁻³. Training uses 75,000 tasks for the larger SSv2 and 10,000 tasks for the other datasets; hyper-parameters were determined on the validation set. Testing records the average accuracy over 10,000 random tasks. Input frames are sized 3 × 256 × 256, with 3 × 224 × 224 random crops and horizontal flipping during training and only a center crop at test time; horizontal flipping is omitted for SSv2 because many of its actions have a horizontal direction. Most experiments ran on two 32GB NVIDIA Tesla V100 PCIe GPUs.

Why This Matters

• Research impact. The paper introduces RWKV to wide-angle FSAR and names one concrete, measurable property of FSAR benchmarks (approximately 35% of samples per dataset have FoV above 80 degrees) that prior work has not addressed. It provides a reusable design pattern — highlight subjects before feature extraction, reconstruct temporal relations during prototype construction — and releases code.

• Fewer labels needed. The framing of the work is that collecting and labeling video is expensive, so a method that classifies unseen actions from extremely few samples reduces labeling dependency.

• Real-world applications (as cited or directly supported by the paper):

  • Health monitoring and motion analysis, which the paper names as real-world applications of FSAR.
  • Sports analysis and coaching — the extra evaluation uses the wide-angle VideoBadminton dataset and the qualitative example is a badminton "smash," an action requiring interaction between two subjects.
  • Scenario-specific action understanding, which is the paper's stated motivation for wide-angle video: context such as a climbing wall or a construction site helps separate actions that look similar against a wall, such as "indoor climbing" versus "construction working."
  • Deployment on wide-angle camera setups, since the method is explicitly designed for and tested under varying field of view.

• Industry relevance. RWKV is described as combining the parallel processing of Transformers with the linear complexity of RNNs, giving efficient global modeling with reduced memory usage and accelerated inference. A method that stays accurate as FoV widens is relevant to any pipeline built on wide-angle or action-camera footage, where background dominance is the norm rather than the exception. The supplementary material includes a computational complexity study, which speaks to practical deployment considerations.

Future Directions

• Backbone coverage. The implementation section lists ResNet-50, ViT-B, VMamba-B, and VRWKV-B as backbones, but the main comparison table reports results only for ImageNet-RN50, ImageNet-ViT, and ImageNet-ViR; results for VMamba-B are not reported in the content provided.

• Combining with multimodal cues. The related-work section notes that prior FSAR methods augment themselves with depth, optical flow, and motion cues, whereas Otter's reported comparisons are organized under RGB-based and multimodal groups without combining those cues into Otter itself. Whether Otter plus those modalities helps is an open question.

• Patch design. The authors report that finer segmentation eventually hurts (p = 28 is worse than p = 56) and that multi-scale patch designs introduce redundant features, so better ways of choosing or fusing patch granularity remain open.

• Extending wide-angle treatment beyond the current evaluation. The paper estimates roughly 35% of samples in popular FSAR benchmarks are wide-angle but "remain unexplored," and its wide-angle evaluation relies on VideoBadminton plus synthetic FoV variation (PQDiff outpainting and a distortion factor). Applying and validating the approach across more genuine wide-angle datasets and real lens/CMOS configurations is a natural next step.

Target Audience

Researchers and practitioners working on few-shot learning and video action recognition will benefit most, especially those dealing with wide-angle or action-camera footage, small subjects, and heavy background clutter. It is also relevant to readers interested in applying RWKV-family sequence models to vision tasks, and to engineers who need efficient video models with reduced memory and faster inference. The paper assumes familiarity with metric-based meta-learning, episodic N-way K-shot evaluation, and attention-style

Authors’ abstract

Wide-angle videos in few-shot action recognition (FSAR) effectively express actions within specific scenarios. However, without a global understanding of both subjects and background, recognizing actions in such samples remains challenging because of the background distractions. Receptance Weighted Key Value (RWKV), which learns interaction between various dimensions, shows promise for global modeling. While directly applying RWKV to wide-angle FSAR may fail to highlight subjects due to excessive background information. Additionally, temporal relation degraded by frames with similar backgrounds is difficult to reconstruct, further impacting performance. Therefore, we design the CompOund SegmenTation and Temporal REconstructing RWKV (Otter). Specifically, the Compound Segmentation Module~(CSM) is devised to segment and emphasize key patches in each frame, effectively highlighting subjects against background information. The Temporal Reconstruction Module (TRM) is incorporated into the temporal-enhanced prototype construction to enable bidirectional scanning, allowing better reconstruct temporal relation. Furthermore, a regular prototype is combined with the temporal-enhanced prototype to simultaneously enhance subject emphasis and temporal modeling, improving wide-angle FSAR performance. Extensive experiments on benchmarks such as SSv2, Kinetics, UCF101, and HMDB51 demonstrate that Otter achieves state-of-the-art performance. Extra evaluation on the VideoBadminton dataset further validates the superiority of Otter in wide-angle FSAR.