APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention

AI summary

Overview

• Research area: Efficient inference for Large Multimodal Models (LMMs), specifically long-video understanding — sitting at the intersection of computer vision, sequence parallelism, and approximate/sparse attention.
• Technical level: Advanced. The paper assumes familiarity with transformer attention, KV caches, FlashAttention, sequence parallelism, and multi-GPU communication primitives such as AllGather.
• One-sentence scope: The paper introduces APB-V, a multi-GPU sequence-parallel framework that combines distributed approximate attention with system-level optimizations to speed up long-video inference in LMMs without discarding visual embeddings.

What This Paper Is About

Long-video inference in LMMs is slow because the prefill stage computes dense attention over a very long sequence of visual embeddings, and existing fixes on a single GPU either compress the visual input (hurting accuracy) or apply sparse attention (giving limited speedup). APB-V instead spreads approximate attention across multiple GPUs, reducing computation and communication while keeping the full set of visual embeddings intact. The goal is to break the usual efficiency-versus-performance trade-off for ultra-long videos.

Key Contributions

1. A sequence-parallelism-aware approximate attention framework (APB-V). Rather than compressing visual embeddings or optimizing attention on one GPU, APB-V uses local KV cache compression plus a distributed approximate attention mechanism to balance efficiency and task performance.
2. System-level optimizations. The authors design a load-balancing strategy across hosts (a ZigZag-style virtual-host arrangement), a fused context-and-query forward pass, and a two-stage attention scheme in which communication is overlapped with computation.
3. Extensive evaluation across benchmarks and models. APB-V is tested on LongVideoBench and VNBench with InternVL3-2B and Qwen2.5VL-3B/7B, reporting speedups of 12.72x over FlashAttn, 1.70x over ZigZagRing, and 1.18x over APB without notable performance loss.
4. Ablations covering both algorithm and system design. The paper isolates the anchor block, the passing block, each of the four system optimizations, host-count scalability, and the sensitivity of the anchor length and passing length hyperparameters.

Main Findings

• Best overall accuracy among approximate-attention methods on VNBench. On VNBench (retrieval, ordering, counting; 64 frames), APB-V reaches an overall accuracy of 43.26 with InternVL3-2B (vs. FullAttn 44.89, APB 41.11, XAttn 42.74, Sparge 38.52, StarAttn 33.85, SlowFast 24.81); 50.67 with Qwen2.5VL-3B (vs. FullAttn 52.81, APB 43.93); and 56.22 with Qwen2.5VL-7B (vs. FullAttn 58.44, APB 49.93).
• Near-lossless results on LongVideoBench. With InternVL3-2B, APB-V scores 55.42 overall versus FullAttn's 55.35; with Qwen2.5VL-7B it scores 59.76 versus FullAttn's 58.38; with Qwen2.5VL-3B it scores 54.30 versus FullAttn's 53.47 (APB reaches 54.97 there).
• Token pruning is the weakest family. SlowFast achieves less than 3x speedup but suffers roughly 25% accuracy degradation, illustrating the cost of discarding fine-grained video detail.
• Consistent speed advantage. APB-V outperforms all baselines across base models, video resolutions, and video lengths, reaching 12.72x, 1.70x, and 1.18x speedups over FlashAttn, ZigZagRing, and APB respectively when processing 64-frame 1440p videos on Qwen2.5-VL-3B.
• Both attention blocks matter. On the VNBench counting subsets, removing the anchor block drops average accuracy to 25.33 and removing the passing block drops it to 17.33, versus 30.00 for full APB-V; the passing block is the more critical component.
• All four system optimizations help. On 1440p video with Qwen2.5VL-3B, APB-V achieves 1.846 req/s at 16 frames and 0.471 req/s at 56 frames, versus 0.381 and 0.107 req/s when all optimizations are disabled — about 4x slower — and versus 0.226 and 0.042 req/s for FlashAttn. The fused context-query forward pass and frame parallelism give the largest gains.
• Good host scalability. Across H ∈ {2, 4, 6, 8} hosts, APB-V is fastest for most frame counts on 720p video; at H=8 and 16 frames it reaches 6.171 req/s versus 5.595 (ZigZagRing), 4.891 (APB), and 4.485 (StarAttn). LongVideoBench accuracy with Qwen2.5VL-3B stays stable across host counts (53.18, 53.63, 54.38, 54.30 for H = 2, 4, 6, 8).
• Robust to weaker interconnect. Moving from 8 NVLink-connected GPUs to two groups of 4 GPUs linked by InfiniBand, APB-V with Qwen2.5-VL-7B drops only 0.75% (1.461 to 1.450 req/s), versus 3.44% for ZigZagRing, 2.09% for StarAttn, and 1.38% for APB.
• Hyperparameter sensitivity. For passing length l_p on VNBench's Ordering-E subset with InternVL3-2B, accuracy rises from 12.67 (l_p=0) to 34.00 (128), 37.33 (256), and 40.00 (512), with 40.67 in the no-compression setting; throughput falls from 5.67 to 4.26 req/s across the same sweep. Compressing the block with l_p = n/128 gives a 1.12x speedup. The anchor length l_a is less sensitive: accuracy is 36.67, 35.33, 38.00, 37.33, and 39.33 for l_a of 0, 128, 256, 512, and no compression, with throughput of 4.73, 4.60, 4.77, 4.78, and 4.36 req/s.
• Case study confirms the mechanism. In a 256-frame first-person road-trip video from VNBench's Retrieval-E subset, where the subtitle "The secret word is Nick" appears in frames 63 to 67, the region containing the answer shows noticeably higher yellow intensity (selection frequency into passing blocks) than other regions.

Methodology in Plain English

APB-V treats a group of GPUs as a "host" that holds a full copy of the model, then splits the work in stages:

1. Frame parallelism for the visual encoder. Because each video frame is encoded independently, frames are distributed across hosts, and the resulting embeddings are gathered with an AllGather.
2. Context splitting. The gathered video sequence is divided into an anchor block (the first l_a embeddings), a query block (the text query embeddings at the end of the sequence), and context blocks split evenly across hosts.
3. Approximate attention. Each host uses query-to-context attention scores to identify only the most important l_p key-value pairs inside its own context block. These essential KV pairs are shared with the relevant hosts as "passing blocks," while the rest stay local. Attention is then computed over the anchor block, passing blocks, and the host's own context block. Query-block attention is computed separately and merged across hosts using FlashAttn's log-sum-exp (lse) output.
4. Load balancing via virtual hosts. Because later context blocks naturally attend to more preceding blocks, the paper introduces 2H virtual hosts mapped in a ZigZag fashion onto H physical hosts, so each physical host carries two complementary virtual hosts and the total passing-block length stays balanced.
5. Overlapping and fusing. Communication of passing blocks and partial query results is overlapped with attention computation, and the context and query blocks are run in a single fused forward pass to avoid a memory-bound query-only pass.

Experiments use 8 A800-40GB GPUs with 3rd-generation NVLink interconnects, host count 8 with 1 GPU per host. APB-V uses l_a = n/64 and l_p = n/128, while APB uses l_a = n/64 and l_p = n/64; with APB-V's 2H virtual hosts the two methods maintain the same amount of compute. Sparge uses similarity and cumulative distribution thresholds of 0.3 and 0.96 respectively.

Why This Matters

• Impact on research: The paper argues that a third scaling axis — more GPUs running approximate attention — is needed when single-GPU methods can no longer buy speed without sacrificing accuracy. It also shows that approximate attention, not just exact sequence parallelism, is what makes scaling to larger host counts worthwhile.
• Real-world applications (the paper's own framing of time-to-first-token-constrained scenarios):
  • Surveillance camera video processing
  • Autonomous driving
  • Any setting where long videos must be understood under an end-to-end latency budget
  • Synthetic-video-style retrieval, ordering, and counting tasks such as those in VNBench
• Industry relevance: The ablations on host count and on interconnect media (NVLink versus InfiniBand) speak directly to deployment on real multi-GPU clusters, where communication bandwidth is often the bottleneck. APB-V's reduced communication design loses the least speed under weaker interconnects (-0.75%), which matters for cost-conscious serving infrastructure.

Future Directions

• Beyond decoder-only transformer LMMs. The limitations section states that APB-V focuses on decoder-only transformer-based LMMs, and that other architectures such as convolutional networks are incompatible with the proposed methods.
• Single-GPU behavior. APB-V is restricted to multi-GPU inference and degenerates to FlashAttn on a single GPU, so how to retain gains in single-device or heterogeneous deployments is left open.
• Further tuning of block lengths. The paper's hyperparameter analysis notes implementation constraints (l_p must be a multiple of 128 in the modified FlashAttn kernel) and shows accuracy still climbing toward the no-compression setting, suggesting room for refinement.
• Broadening the evaluation. The evaluation covers two benchmarks (LongVideoBench and VNBench) with 64 frames on three LMMs; whether the approach transfers to other long-video benchmarks, frame counts, and model families is not reported.

Target Audience

Systems and efficiency researchers working on multi-GPU LLM/LMM inference; practitioners deploying long-video understanding on GPU clusters; and graduate students or engineers already comfortable with attention kernels, KV caching, and distributed sequence parallelism who want a concrete blueprint for combining approximate attention with parallel execution. Readers without a background in transformer internals or distributed training will find the framework design sections demanding.

Authors’ abstract

The efficiency of long-video inference remains a critical bottleneck, mainly due to the dense computation in the prefill stage of Large Multimodal Models (LMMs). Existing methods either compress visual embeddings or apply sparse attention on a single GPU, yielding limited acceleration or degraded performance and restricting LMMs from handling longer, more complex videos. To overcome these issues, we propose APB-V, a sequence-parallel framework with optimized attention that accelerates long-video inference across multiple GPUs. By distributing approximate attention, APB-V reduces computation and increases parallelism, enabling efficient processing of more visual embeddings without compression and thereby improving task performance. System-level optimizations, such as load balancing and fused forward passes, further unleash the potential of APB-V, delivering speedups of 12.72x, 1.70x, and 1.18x over FlashAttn, ZigZagRing, and APB, without notable performance loss. Code available at https://github.com/thunlp/APB