SV-TAD: Native Sparse Convs for Efficient Temporal Action Detection

AI summary

Overview

Research area: Computer vision — efficient video understanding, specifically temporal action detection (TAD) with large frozen Vision Transformer (ViT) backbones.

Technical level: Intermediate. Familiarity with Vision Transformers, token pruning, convolutional adapters, and GPU kernel concepts helps, but the core idea is explained in accessible terms.

Scope: The paper introduces a sparse 2D convolution primitive (SparseConv2D) and a sparse adapter framework (SV-TAD) that let convolutional adapters operate directly on pruned token sequences, cutting compute and latency for temporal action detection while matching or improving accuracy.

What This Paper Is About

Adapting billion-parameter video ViTs to temporal action detection is normally done by freezing the backbone and training small convolutional adapters. Those adapters still process every token on a dense spatial grid, so inference cost stays proportional to full video length, and pruning uninformative tokens breaks the grid that convolutions require, forcing an expensive dense reconstruction step that erases most of the savings. SV-TAD closes that gap by defining a sparse 2D convolution that reads a precomputed neighbor table over the retained tokens, so pruning translates directly into faster inference and lower memory.

Key Contributions

1. Native sparse 2D convolution for Vision Transformers. A computational primitive that processes dynamically sparse token sequences using a precomputed neighbor index table and custom CUDA kernels, with no dense grid reconstruction. The table maps each kept token to its 8 spatial neighbors' positions in the sparse list, with −1 marking pruned or out-of-bounds entries, and is built in O(N_kept) with less than roughly 3 ms of CPU overhead.
2. Sparse adapter architectures. SparseConv2D is integrated into a bottleneck adapter inserted after each Transformer block's attention layer, enabling convolutional adapters to operate on token-selected sequences for the first time. Early layers (before token selection) still use a standard 2D convolution.
3. Strong efficiency–accuracy trade-offs on TAD benchmarks. The framework reports 64% fewer TFLOPs, 2.2× faster inference, and 1.7× training acceleration over the state of the art on THUMOS-14 while maintaining mAP, plus a new state of the art when scaled to InternVideoNext-L on both THUMOS-14 and ActivityNet-1.3.
4. Multitask support and a released library. Auxiliary task tokens are supported through a cross-attention module that updates only the auxiliary tokens (down-projected to D/8) at no cost to the visual path; on ATTACH, landmark supervision improves average mAP by +2.41%. The CUDA kernels are released as a standalone library for sparse token processing in PyTorch, with code and models at https://github.com/pcr-upm/eccv26_tad.

Main Findings

• Headline efficiency on THUMOS-14 (VideoMAEv2-L): SV-TAD reduces TFLOPs by 64% (21.25 vs. AdaTAD's 59.4) while matching accuracy (73.47% vs. 73.50% average mAP).
• Throughput and training speed: SV-TAD (ViT-L) processes 1.57 videos/second versus the baseline's 0.73, a 2.2× improvement. Training takes 3.01 s per step (forward 0.70 s, backward 2.31 s) versus the baseline's 5.35 s (forward 1.49 s, backward 3.86 s), a 1.7× speedup.
• Memory: Peak allocated memory per training step is 9.23 GB versus AdaTAD's 13.30 GB, a reduction of about 31%, measured on a single RTX 3090 with batch size 1. SV-TAD has more parameters than AdaTAD at this scale (361M vs. 346M).
• Scaling to larger backbones: With InternVideoNext-L on THUMOS-14, SV-TAD needs 87.05 TFLOPs versus 176.87 for AdaTAD's VideoMAEv2-G, exceeding its 73.87% average mAP with 74.13% (+0.26%) at roughly half the compute. On ActivityNet-1.3, SV-TAD with InternVideoNext-L uses 34 TFLOPs versus 83.59 (59% fewer) and reaches 40.06% average mAP versus 39.20% (+0.86%).
• Controlled backbone comparison: Running AdaTAD with the same InternVideoNext-L backbone yields 39.72% average mAP at 95.3 TFLOPs, so at matched backbone SV-TAD still cuts compute substantially (34 vs. 95.3 TFLOPs) while improving accuracy (+0.34%).
• Backbone-matched THUMOS-14 result (VideoMAEv2-B): SV-TAD requires 10.29 TFLOPs versus AdaTAD's 17.9 (43% fewer) while reaching 72.44% versus 72.15%.
• Kernel crossover behavior: The sparse kernel matches dense convolution at roughly a 55% keep rate; below that threshold it is strictly faster, reaching 8× speedup at a 10% keep rate.
• Kernel-level speed and memory: At k_r = 0.3, the sparse kernel is 2× faster at 768 frames (4 ms vs. 8 ms) and 1.75× faster at 6,144 frames (40 ms vs. 70 ms). Peak memory stays near-constant at about 800 MB while dense grows to about 6 GB at 6,144 frames, an 87% reduction (the figure caption states a 7.5× reduction).
• Dense vs. sparse convolution (THUMOS-14): SparseConv2D reaches 69.35% average mAP versus 68.83% for DenseConv2D (+0.52%), while DenseConv1D reaches 69.60%. In FP32 with no token selection (k_r = 1), the dense-versus-sparse gap shrinks to 0.10%.
• Cross-attention recovers temporal context: Replacing a temporal 1D convolution with a spatial 2D convolution costs +0.77% in favor of DenseConv1D over DenseConv2D; the CLS-query cross-attention closes the gap at a cost of +0.11 TFLOPs (SV-TAD (No CrossAttn) at 10.18 TFLOPs and 71.80% versus full SV-TAD at 10.29 TFLOPs and 72.44%).
• Multitask supervision on ATTACH (VideoMAEv2-B): Adding auxiliary landmark supervision raises average mAP from 16.28% to 18.69% (+2.41%), with per-threshold mAP from 0.1 to 0.5 rising from 20.17/18.69/16.90/14.39/11.25 to 22.58/21.10/19.21/16.80/13.76, at a cost of 10.29 → 11.17 TFLOPs. The authors state this is the first formulation of ATTACH as a full TAD problem.
• Keep-rate control: On THUMOS-14 with InternVideoNext-L, k_r = 0.7 gives 87.05 TFLOPs, 0.42 videos/s, 74.13% average mAP; k_r = 0.6 gives 74.09 TFLOPs, 0.50 videos/s, 73.64%; k_r = 0.5 gives 60.13 TFLOPs, 0.60 videos/s, 73.00% — a graceful degradation of 1.13% average mAP for raising throughput from 0.42 to 0.60 videos/s.
• Compounding sparsity: With k_r = 0.6 per stage, the effective rate after two selection stages is 0.36 and after four stages is 0.13, placing most layers below the per-kernel crossover point; token selection is applied at blocks 4, 8, 12, and 16 for ViT-L.
• FlashAttention compatibility: Selection needs only the K attention rows from auxiliary tokens to visual tokens rather than the full N×N matrix, so the entire pipeline runs with FlashAttention enabled — a distinction from merging-based methods that require the pairwise similarity matrix.

Methodology in Plain English

The authors start from the observation that a convolution only needs local neighborhoods, not the full grid, so the grid itself can be discarded if adjacency is encoded explicitly. They keep the original (y, x) grid indices of every retained token, then build a lookup table from grid coordinates to sparse indices and a neighbor table of shape N_kept × 9 that points each kept token at its 3×3 spatial neighbors in the sparse list, with −1 for pruned or out-of-bounds entries. Both construction steps run in O(N_kept) on CPU with negligible overhead, and the lookup map is reused across adapter layers until the next pruning stage.

The convolution itself is implemented as CUDA kernels: a forward pass that gathers neighbor features through the table and applies per-position weight slices, with two strategies chosen automatically — an implicit GEMM path via cuBLAS for large channel counts (C_out ≥ 256) that uses Tensor Cores, and custom tiled kernels that fuse gather and GEMM for smaller channels (the supplementary describes collapsing 36 kernel launches into 3). Memory is decoupled from sequence length through chunked processing.

On top of this primitive, the adapter is a bottleneck: down-projection to D/4, GELU, SparseConv2D, a cross-attention module where auxiliary tokens query the sparse visual features, and an up-projection modulated by a learnable scalar γ initialized to 1. Token selection follows EViT-style attention scoring, extended so that when multiple auxiliary tokens are present, importance scores are a weighted sum over their attention rows using learnable weights. Everything is built on frozen backbones (VideoMAEv2, InternVideoNext) inside the OpenTAD framework with ActionFormer as the detection head.

Why This Matters

Impact on research. The paper separates three lines of work that previously could not be combined: parameter-efficient adapters (which save training cost but not inference cost), token selection (which saves attention cost but not convolution cost), and 3D sparse convolution engines (which require hash-table coordinate management and COO-style formats that are architecturally mismatched to ViT packed tensors). The claim is that native sparse 2D convolution is the missing primitive, and that the resulting savings compound across layers as model capacity grows — with ViT-L the authors match top accuracy at 64% fewer TFLOPs, and with InternVideoNext-L they report a new state of the art on THUMOS-14 and ActivityNet-1.3 at 51% and 59% fewer TFLOPs respectively.

Real-world applications:

• Long-form video surveillance and security review, where sequences exceeding tens of thousands of tokens make dense processing a bottleneck.
• Sports analytics and highlight generation, matching the sports classes evaluated on THUMOS-14.
• Industrial assembly monitoring and quality control, the fine-grained setting evaluated on ATTACH with 51 classes and 68% temporal overlap.
• Large media-archive indexing and search, where inference cost per hour of footage governs feasibility.

Industry relevance. Serving billion-parameter video foundation models is dominated by inference cost, and the paper reports an efficiency-accuracy frontier rather than just a kernel micro-optimization: keep rate k_r becomes a tunable deployment dial, with throughput moving from 0.42 to 0.60 videos/s for a 1.13% average mAP cost on InternVideoNext-L. The kernels are released as a standalone PyTorch library, and the primitive is presented as applicable wherever convolutional modules follow token selection — spatial adapters, feature pyramid networks, and task-specific heads — not only TAD.

Future Directions

• Non-NVIDIA portability. The implementation targets NVIDIA GPUs; porting requires replacing the backend even though the core algorithm is described as architecture-agnostic.
• Learned, content-adaptive keep-rate schedules. The current schedule uses fixed per-stage keep rates; learned schedules are named as future work.
• Extending the primitive to 3D spatio-temporal neighborhoods. The paper addresses 2D spatial convolution and argues a native sparse 1D temporal kernel is impractical because temporal neighbors sit in different frame blocks with roughly 8× larger memory access distances.
• Deployment regimes below the crossover point. The sparse kernel only beats dense below approximately a 55% keep rate, so the primitive is most useful when progressive pruning through the layers is acceptable — how to handle densely processed early layers efficiently remains open.

Target Audience

Researchers and engineers working on efficient video understanding, temporal action detection, and parameter-efficient fine-tuning of large vision transformers; GPU/systems engineers interested in custom sparse kernels for dynamically pruned token sets; and practitioners who need to deploy billion-parameter video foundation models under real compute and memory budgets. Readers focused on the ATTACH benchmark or on multitask supervision via auxiliary tokens will also find relevant ablations.

Authors’ abstract

To adapt billion-parameter Vision Transformers for long-video understanding, recent methods freeze the backbone and train lightweight convolutional modules. While effective for parameter-efficient training, existing adapters do not reduce inference-time computation, leaving scalability with respect to video length largely unaddressed. Token selection can reduce attention cost by pruning redundant tokens, but it breaks the spatial grid structure required by convolutional adapters. This forces an expensive dense reconstruction, nullifying much of the potential speedup. We address this by introducing native sparse 2D convolutions, a primitive that allows these adapters, for the first time, to operate directly and efficiently on dynamically pruned token sets. We integrate this primitive into SV-TAD, an adapter framework for temporal action detection, reducing VideoMAEv2-L computation by up to 64% and achieving 2.2x faster inference, while maintaining state-of-the-art accuracy on THUMOS-14 and ActivityNet-1.3. When scaled to InternVideoNext-L, our approach surpasses the previous state of the art at roughly half its computational cost. Moreover, the sparse formulation naturally supports auxiliary task tokens, which improves fine-grained assembly detection on ATTACH.