DiA: Directional Adapter

AI summary

Overview

• Research area: Computer vision, specifically fine-grained video action recognition and parameter-efficient transfer learning on top of a frozen CLIP visual backbone.
• Technical level: Intermediate. Readers should be comfortable with vision transformers, adapters, and standard action-recognition benchmarking, but the core idea (asymmetric temporal filtering) is conceptually simple.
• Scope: The paper proposes DiA (Directional Adapter), a small adapter that splits temporal modeling into past-to-present (causal) and future-to-present (anti-causal) pathways with learnable fusion, and evaluates it on seven public action-recognition benchmarks against the accuracy–compute efficiency–unimodality objective.

What This Paper Is About

Fine-grained actions such as "switch leap with 0.5 turn" versus "switch leap with 1 turn", or "Forward 25som 2Twis PIKE" versus "Forward 25som 3Twis PIKE", look nearly identical in appearance and differ mainly in how motion unfolds over time. Most efficient video adapters aggregate temporal context uniformly, which can blur the phase-specific and direction-specific cues that separate these classes. The paper's goal is to recover that asymmetric temporal evidence using only the video modality, while adding few tunable parameters and keeping inference cheap.

Key Contributions

1. Problem framing. The authors formulate fine-grained action recognition as a temporal specialization problem on top of CLIP, explicitly targeting a three-part objective: accuracy, compute efficiency, and unimodality (video only).
2. Directional temporal adaptation. Temporal evidence is decomposed into a causal (past-to-present) and an anti-causal (future-to-present) pathway, so the two directions are modeled differently rather than with one symmetric temporal operator.
3. Learnable directional fusion. A pair of trainable direction logits, normalized with softmax, sets how much causal versus anti-causal evidence is used, instead of fixing the balance by hand.
4. Efficient implementation. Both directions use depth-wise temporal convolutions in a compact bottleneck space and are applied as a token-local residual initialized near identity, giving roughly 3.6M tunable parameters on the reported Diving48 setting (the introduction cites approximately 3.0M for the same result; Table 1 reports 3.6M).

Main Findings

• State-of-the-art among single-stream, video-only methods: On Diving48, DiA reaches 92.8% ± 0.15 top-1 with 3.6M tunable parameters at 32 frames, versus AIM at 88.9% with 11.0M.
• Consistent gains on FineGym: 96.2% ± 0.13 on FineGym99 and 91.7% ± 0.10 on FineGym288 with 3.0M parameters, compared with AIM at 94.9% and 89.1% under the same 32-frame setting.
• Gains transfer across domains: 82.4% ± 0.08 on HAA500 (2.3M parameters, versus AIM at 80.2%), 86.8% ± 0.12 on ToyotaSmartHome under the cross-subject protocol (3.7M, versus AIM 84.3%), 78.7% ± 0.15 on MPII Cooking 2 (3.7M, versus AIM 76.1%), and 85.6% ± 0.10 on HMDB51 (3.6M, versus AIM 82.4%).
• SSv2 stays in the small-parameter regime: 69.2% top-1 with 3.0M tunable parameters, close to ST-Adapter at 69.3% (7.2M) and AIM at 68.1% (14M), while large models such as MViTv2-L (213M, 73.3%) remain more accurate.
• Accuracy and efficiency improve together: Against AIM on Diving48, accuracy rises from 88.9% to 92.8% while per-clip cost drops from 812.06 to 558.37 GFLOPs (31.24% reduction), batch-level cost from 6496.49 to 4466.99 GFLOPs, throughput rises from 28.15 to 61.20 clips/s (2.17×), effective compute from 22.86 to 34.17 TFLOP/s (1.49×), and p50 batch latency falls from 284.14 ms to 130.73 ms (53.98% lower).
• Both directions are needed: Causal-only reaches 91.3% and anti-causal-only 91.5% on Diving48, while combining them reaches 92.8%, indicating complementary evidence.
• No directional collapse: On a held-out Diving48 subset of N = 120 clips, both activity ratios are 1.0000, mean corr(E_c, E_a) is -0.1884 and mean corr(E_c, reverse(E_a)) is -0.1705, mean dominance is 0.0278 with standard deviation 0.0193, and 88.33% of clips satisfy the balanced-clip criterion |D̄| ≤ 0.05.
• Learned fusion stays balanced: Across 10 layers, mean fusion weights are 0.5030 and 0.4970, a mean gap of 0.0342.
• Beats a multimodal method with video only: With ViT-L/14, DiA scores 93.8 vs. PeVL's 92.5 on Diving48 and 92.9 vs. 91.8 on FineGym288 (DiA's FineGym99 result at that backbone is not reported in the table), using 12.7M vs. 109M tunable parameters. Video-only PeVL at ViT-B/16 scores 84.5% on Diving48 against DiA's 92.8%.
• Efficiency holds across depths and frame counts: With 6 adapted blocks DiA gets 90.6% at 1.8M parameters versus AIM's 87.3% at 5.3M; with 10 blocks 92.7% at 3.0M versus 87.7% at 8.9M; at 8/16/32 frames DiA scores 83.0/90.5/92.7 versus AIM's 81.8/84.7/88.9.

Methodology in Plain English

DiA is a small module bolted onto a frozen CLIP ViT-B/16 image encoder. For each transformer block, the patch tokens are reshaped into a video tensor, squeezed down to a narrower "bottleneck" width (α = 0.25, so 768 channels become 192), and then filtered along the time axis at each spatial location.

Two filters run in parallel. One looks only backward in time (past-to-present, using left replication padding); the other looks only forward (present-and-future, using right replication padding). Both are depth-wise temporal convolutions with kernel size k = 5, so they are cheap. A pair of trainable logits, passed through softmax, decides how heavily to weight each direction; the logits start at [0.5, 0.5]. The fused response passes through GELU, is projected back to the original channel width, and is added residually to the input tokens. Because the output projection is zero-initialized, the adapter begins as an identity mapping and learns temporal corrections gradually.

Only patch tokens are updated; the class token is left alone and later picks up the refined information through normal self-attention. After the final LayerNorm, frame-wise class features are averaged and fed to a linear classifier. Training follows the AIM protocol for 50 epochs with 32 frames, AdamW at learning rate 0.0025, weight decay 0.05, cosine decay, and 2–3 warm-up epochs. Results are averaged over three random seeds.

Why This Matters

Research impact. The paper argues that the direction of temporal aggregation, not just its presence, is a useful inductive bias for phase-sensitive action classes, and it backs this with an ablation, activation statistics, and a compute analysis in one package. It also offers a clean baseline showing that a video-only adapter can match or exceed a video+pose+text method at matched backbones, which reframes where multimodal gains actually come from.

Real-world applications.

• Sports analytics and judging support, where rotation counts and landing mechanics define the class (Diving48-style events).
• Fitness and rehabilitation coaching, where correct execution order of preparation, execution, and completion phases matters.
• Procedural and cooking-assistance systems that must distinguish ordered steps (MPII Cooking 2, ToyotaSmartHome).
• Always-on or edge video monitoring, where 2.17× higher throughput and 130.73 ms batch latency versus 284.14 ms make deployment more practical.

Industry relevance. The design keeps the backbone frozen and trains only about 3.6M parameters against an 85.8M CLIP ViT-B/16 backbone (89.4M total at inference), which fits low-cost fine-tuning pipelines and single-GPU serving. The measured 31.24% GFLOP reduction and 1.49× realized compute improvement point to direct savings on existing A100-class hardware without a new data pipeline for pose or text.

Future Directions

1. Longer-range temporal structure. The authors note that existing fine-grained benchmarks contain relatively short clips and that capturing extended temporal dependencies would require curating long-duration fine-grained action datasets.
2. Online recognition. DiA uses both causal and anti-causal directions within the observed clip, which suits offline recognition; streaming settings where future frames are unavailable remain unaddressed.
3. Beyond the closed-set, video-only setting. The paper deliberately excludes pose and language cues, leaving open how directional adapters would compose with auxiliary streams when those are reliable.
4. Scaling and depth/frame trade-offs. Performance saturates at 12 adapted blocks and the authors select 10 as the best accuracy–parameter trade-off; whether the saturation point shifts with larger backbones or longer clips is not established.

Target Audience

Graduate students and applied researchers in video understanding who work on parameter-efficient adaptation of vision-language models; engineers deploying action recognition under latency and memory constraints; and practitioners in sports, fitness, or procedural-activity analytics who need to separate visually similar classes that differ mainly in motion order. Readers without background in transformers or adapter methods will find the empirical tables useful but the formulation section heavy.

Authors’ abstract

Fine-grained action recognition often depends on how an action unfolds over time rather than on appearance alone. Visually similar classes may differ in preparation, execution, or completion phases, making uniformly aggregated temporal context suboptimal. We study this problem under a practical objective: i) improving recognition performance with few tunable parameters, ii) being compute efficient, and iii) using only the readily available video modality. Motivated by this missing temporal observation and the accuracy--compute efficiency--unimodality objective, we propose Directional Adapter (DiA), a parameter-efficient adapter on top of CLIP. DiA formulates temporal specialization into causal and anti-causal directions, allowing past-to-present and future-to-present cues to be modeled differently. To keep this directional modeling parameter- and compute-efficient, both directions use depth-wise temporal convolutions in a compact bottleneck space and are combined through a learnable fusion. DiA achieves state-of-the-art performance among methods using a single video input modality across seven public benchmarks, with only $\sim$3.6M tunable parameters, while also delivering higher throughput and lower inference latency than prior methods. Code is available at https://github.com/prakashchhipa/DiA-Directional-Adapter.