Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

AI summary

Overview

Research area: Inference efficiency for video and audiovisual large language models (VideoLLMs), spanning computer vision, multimodal machine learning, and ML systems.

Technical level: Intermediate. The organizing ideas are accessible to anyone familiar with transformer-based vision-language models, while the cost model in Section III-B requires comfort with attention/FLOP scaling and KV-cache accounting.

Scope: A survey of 125 papers covering mechanisms that measurably reduce parameter count, FLOPs per input, latency, memory, or visual/audio token count in VideoLLM inference, organized by the pipeline stage at which each mechanism acts.

What This Paper Is About

VideoLLMs — systems that pair video (and sometimes audio) encoders with a pretrained language model — perform well on captioning, question answering, retrieval, and temporal grounding, but their cost grows with frame count and context length, which blocks deployment in real-time, mobile, and resource-constrained settings. The core difficulty the survey addresses is that efficiency methods are proposed in isolation, tied to different tasks, and measured with incompatible protocols, so it is hard to tell where computation is actually spent or which strategy wins under a given constraint. The authors' goal is a video-specific synthesis that connects reported efficiency gains to pipeline stage, temporal coverage, and evaluation conditions, while separating genuinely comparable results from heterogeneous cross-paper claims.

Key Contributions

1. A pipeline-grounded taxonomy. The survey synthesizes efficiency mechanisms across four stages — input construction and selection, encoder computation, encoded representations and connector, and LLM execution and state — linking upstream temporal coverage and encoding cost to downstream token and memory budgets. It also folds in pre-VideoLLM frame-sampling and vision-encoder methods that remain direct components or antecedents of current pipelines (marked separately in the taxonomy figure).

2. A disciplined comparison protocol. The authors assemble literature-reported accuracy–cost comparisons under shared host models, input settings, and token budgets wherever such values exist, and explicitly flag when comparisons cross heterogeneous systems. Differences in input protocols and FLOP-accounting boundaries are made explicit rather than smoothed over.

3. A dedicated audiovisual efficiency analysis. The survey examines audio-token compression, audio-guided visual selection, and joint audiovisual token budgets — a dimension largely absent from prior efficiency surveys — and analyzes how the joint workload differs from visual-only inference.

4. Identification of evaluation gaps plus a maintained inventory. The authors catalog shortfalls in audiovisual efficiency coverage, benchmarking standardization, and energy reporting, and maintain a public repository of the reviewed literature at momentslab/awesome-efficient-videollm.

Main Findings

• Cost decomposes cleanly into four controllable levers. Frame count and resolution set encoder cost and the raw modality token count N_v^enc = T · (H/P) · (W/P); the connector decides how many of those tokens (Ñ_v, Ñ_a) actually reach the LLM; the resulting context L = N_t + Ñ_v + Ñ_a sets LLM prefill cost and KV-cache memory, which scales as 2B · n_layers · L · d_KV · b. Each stage therefore offers a distinct, non-substitutable place to cut cost.

• The encoder alone can dominate total cost. Long-video VideoLLMs often process hundreds of frames at high resolution via sliding windows or dense sampling, so encoder compute is linear in frame count at fixed resolution and can outweigh downstream LLM cost unless frames are subsampled or pooled.

• Frame selection is the least reversible lever. Frames discarded before encoding cannot be recovered downstream, so temporal selectors trade coverage against cost in a way that later token pruning cannot undo. The survey traces this from fixed-coverage schemes (TSN-style segment sampling) through content-based coverage (KTS, MaxInfo, MGSampler), learned query-free policies (AdaFrame, PEEK), and query-conditioned relevance-plus-diversity methods (AKS, Q-Frame, FOCUS, AdaRD-Key, BOLT, F2C, T*, QCA, EFS, GIFT).

• Simple sampling remains a strong baseline. The survey reports that frame-sampling choices alone can shift video-QA accuracy, and that uniform or strided sampling is frequently competitive — sometimes the strongest strategy for small VLMs on Video-MME — despite the proliferation of learned selectors.

• Token reduction and KV compression target different bottlenecks. Reducing input tokens benefits both prefilling (where attention-score computation is quadratic in L) and decoding, whereas KV-cache compaction primarily addresses decoding memory. Their relative payoff therefore varies across interactive, batched, and offline workloads.

• Connector cost depends on source length, not only on query count. For Q-Former-style cross-attention, per-layer cost scales as (N_q + N_s)·d² + N_q·N_s·d. Because N_q is small but N_s (the concatenated visual plus audio source tokens) can be large, many systems must pool, downsample, or selectively fuse before cross-attention to enforce a fixed joint token budget.

• Audio cannot be ranked against video by sampling rate. Audio starts from a denser temporal signal, but each video frame yields many spatial patch tokens, so the relative weight of audio versus visual tokens depends on connector downsampling and architecture — audio may be negligible or substantial in long-form audiovisual inputs.

• Heterogeneous reporting limits what can be compared. Even among methods reporting "LLM prefilling FLOPs," accounting boundaries differ: HoliTom and HieraVid report prefilling FLOPs, while EarlyTom includes vision-encoder FLOPs. Input conditions (frame counts, resolutions, modality coverage) also vary between systems claiming similar GFLOPs per video.

• Energy is a blind spot. Energy is a relevant system-level metric for the deployment scenarios the survey targets, yet none of the surveyed methods reports it.

• Audiovisual efficiency is comparatively under-explored. The survey finds that audio-token reduction, audio-guided visual selection, and joint token budgeting receive substantially less attention than visual-only token compression.

Methodology in Plain English

The authors ran keyword searches on arXiv and Google Scholar combining VideoLLM terms with efficiency terms (token pruning, token merging, frame selection, KV-cache compression), then expanded the set through backward and forward citation snowballing from existing surveys and from every retained paper. This surfaced several hundred candidates, which they screened by title and abstract and then read in full, keeping 125.

A method entered the taxonomy only if it contributes or evaluates a targeted efficiency mechanism and reports a concrete effect on parameter count, FLOPs, retained tokens, latency, or memory. VideoLLMs developed since late 2022 form the core; earlier frame-sampling and vision-encoder methods are included when they remain components of current pipelines. Training-only methods, generic LLM optimizations, and image-only techniques are cited only as adjacent context. Papers that reduce cost at several stages appear in multiple families, so family sizes exceed the paper count.

For comparison, the authors classify each reduction by the pipeline stage whose computation it removes — pooling after an encoder's final block counts as a connector-stage method even when implemented inside the encoder, and whole-frame selection can occur after encoding. They use only values explicitly reported by each paper, record the corresponding model variant and input setting, refuse to infer FLOPs or latency from architecture alone, and decline to convert token-retention budgets into FLOPs. Quantitative comparisons emphasize 7B–8B language backbones; mechanisms demonstrated only on larger hosts stay in the taxonomy but out of the shared-host tables.

Why This Matters

Impact on research. The survey supplies a shared vocabulary and a stage-based taxonomy for a fast-moving subfield, which makes it possible to state precisely which bottleneck a new method targets and to avoid double-counting gains that actually come from several stages. Its comparison protocol is arguably as valuable as its inventory: by separating controlled shared-host comparisons from indicative cross-paper ones, and by insisting that input protocols and FLOP boundaries be reported, it exposes a measurement-hygiene problem that currently makes headline efficiency numbers hard to trust.

Real-world applications.

• Real-time and streaming video assistants, where response latency and bounded memory must hold as a stream grows indefinitely.
• On-device and mobile video understanding — search, accessibility, and egocentric assistants operating under strict power and memory limits.
• Large-scale media archives: searching, summarizing, and answering questions over movies, broadcast footage, and long-form egocentric recordings.
• Audiovisual content pipelines such as meeting understanding or multimodal moderation, where both the audio and visual token streams must fit a shared context window.

Industry relevance. The author affiliations (Télécom SudParis/SAMOVAR and Moments Lab, a media-tech company) reflect the practical framing: efficiency determines whether video AI is economically deployable at scale, since cost scales with frame count, resolution, and context length. The survey gives practitioners a way to budget cloud or edge compute deliberately, choose levers appropriate to their latency regime (interactive versus batch versus offline), and evaluate vendor claims that were measured under incompatible conditions. For hardware and infrastructure planning, the distinction between prefill-bound and decode-memory-bound workloads is directly actionable.

Future Directions

• Standardized efficiency evaluation. The survey identifies the absence of unified frame/time budgets, modality conditions, and accounting boundaries as the main obstacle to fair comparison, and calls for protocols that make these explicit across methods.
• Audiovisual efficiency as a first-class problem. Joint audio-visual token budgeting, audio-guided visual selection, and principled accounting that separates encoder, connector, and LLM cost for two modality streams remain largely open.
• Energy and power reporting. No surveyed method reports energy, despite its direct relevance to mobile and edge deployment; establishing it as a standard reported metric is an obvious next step.
• Bridging analytical indicators to runtime. Since FLOPs, parameter counts, and token-retention ratios do not reliably predict wall-clock speed — operator mix, parallelism, memory access, and implementation all intervene — the field needs measurements that connect the two, especially for streaming systems with bounded memory.

Target Audience

Researchers and graduate students working on VideoLLMs, multimodal LLMs, or efficient inference who need a structured map of the field and a defensible basis for choosing and benchmarking efficiency mechanisms. It is also directly useful to ML systems and infrastructure engineers responsible for deploying video understanding under latency, memory, or cost constraints, and to authors of related surveys seeking the video- and audiovisual-specific complement to existing efficiency taxonomies. Prior familiarity with transformer pipelines and vision-language architectures is helpful but not required for the taxonomy and comparison protocol.

Authors’ abstract

Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained large language models and condition generation on a textual prompt. Their strong performance on captioning, question answering, retrieval and temporal grounding comes at a computation and memory cost that grows with frame count and context length, limiting deployment in real-time, mobile and resource-constrained settings. This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count. We analyze bottlenecks across frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding. We organize methods by the pipeline stage at which they act, covering VideoLLMs developed since late 2022 together with earlier frame-sampling and vision-encoder mechanisms that remain components of current pipelines. We assemble literature-reported accuracy--cost comparisons under shared host models and input protocols wherever available, distinguish them from heterogeneous cross-paper evidence, and identify gaps in audiovisual efficiency and standardized evaluation. We maintain a repository at https://github.com/momentslab/awesome-efficient-videollm.