Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically efficient long-form and high-resolution video understanding.

Technical level: Advanced. The paper assumes familiarity with vision transformers, autoregressive decoding, reinforcement learning (GRPO), masked auto-encoding, and multimodal benchmarking.

Scope in one sentence: The paper proposes AutoGaze, a 3M-parameter module that autoregressively selects a minimal set of multi-scale video patches before a ViT ever sees them, then introduces the HLVid benchmark to measure high-resolution long-form video QA.

What This Paper Is About

Video MLLMs spend compute uniformly across every pixel of every frame, even though video is highly redundant in space and time. Existing token-reduction methods prune tokens inside the LLM (or between the ViT and the LLM), so the ViT remains an efficiency bottleneck that blocks scaling to long, high-resolution video. The paper's goal is to remove redundant patches before the ViT, so that a model can handle videos up to 1K frames at 4K resolution under realistic compute budgets.

Key Contributions

1. AutoGaze, a 3M-parameter lightweight module that autoregressively selects a minimal set of multi-scale patches which reconstruct a video within a user-specified reconstruction loss threshold, decoding only patch indices rather than being a fixed grid tokenizer.

2. A two-stage training recipe: next-token-prediction pre-training on 250K videos with gazing sequences collected by greedy search, followed by on-policy GRPO reinforcement learning with reconstruction loss as reward.

3. A ViT integration method that lets standard image ViTs accept multi-scale patch inputs and process all 16 frames in one sequence, plus an any-resolution/any-duration tiling scheme (16x224x224 spatiotemporal tiles) requiring no additional training.

4. HLVid, described as the first high-resolution, long-form video QA benchmark: 268 QA pairs over up to 5-minute, 4K-resolution videos, each requiring visual perception at 1K-2K resolution.

Main Findings

• Large token and latency reductions. AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x and 10x respectively. For 30-FPS, 4K-resolution videos, roughly 1% of patches suffice to reach a reconstruction loss of 0.7.

• Threshold of 0.7 is near-lossless for downstream tasks. A reconstruction loss threshold of 0.7 usually leads to less than 0.5% performance degradation across benchmarks.

• Benchmark results. NVILA-8B-Video with AutoGaze, scaled to 1024 frames and 3584 resolution, reaches 67.0% on VideoMME (w/o sub), 71.8% (w/ sub), 69.7% on MVBench, 82.8% on NExT-QA, 61.0% on LongVideoBench (val), 66.9% on EgoSchema, 71.6% on MLVU, and 52.6% on HLVid.

• HLVid improvement. On HLVid, scaling with AutoGaze improves the base NVILA-8B-Video from 42.5% to 52.6% (a 10.1% gain), and the paper reports it outperforms the previous best MLLM by 4.5%.

• Baseline runs out of memory. The ViT baseline runs out of memory around 30 FPS and 896 resolution; the MLLM baseline can only encode 30 FPS at 224 resolution, and the baseline runs out of memory beyond 256 frames while AutoGaze scales to 1K frames and 4K resolution.

• Vs. token-reduction baselines. At a 6.25% selection ratio on 128-frame videos, prior methods improve LLM latency by 3.7x-13.4x but leave ViT latency unchanged (around 2.2s), whereas AutoGaze cuts ViT latency to 0.55s (about 4x) and LLM latency to 0.10s while keeping performance comparable to no reduction (52.3 vs 53.4 on VideoMME w/o sub; 50.3 vs 51.1 on LongVideoBench).

• Vs. heuristic gazing. AutoGaze reaches reconstruction loss 1.0 with 5% of patches versus 15% for Random Gaze. RGB-difference gazing and optical-flow gazing (SEA-RAFT) perform worse than Random Gaze because they fixate on the first frame.

• Behavioral evidence. Across all scales, AutoGaze more frequently selects patches with higher optical flow (measured on FlyingChairs); it uses finer scales for more detailed patches, measured by Laplacian variance over 2,000 ImageNet images (rho = .12, p < 0.001).

• Out-of-distribution generalization. AutoGaze continues to track changing regions on CCTV footage, a robot demo, an object-swapping video, and style-transferred video (TokenFlow), without retraining.

• Ablations. Gazing ratio at reconstruction loss 0.7: 0.263 with no training, 0.102 with pre-training only, 0.209 with RL only, 0.094 with both (about 10% better than pre-training alone). Multi-token prediction shows a trade-off: 1 token gives ratio 0.074 at 0.949s latency, 5 tokens 0.078 at 0.246s, 10 tokens 0.094 at 0.193s, 20 tokens 0.109 at 0.156s. Removing multi-scale gazing raises the ratio to 0.220 and latency to 0.467s, so multi-scale gazing improves efficiency by 2.3x.

Methodology in Plain English

The researchers reframe video understanding as a reconstruction problem. AutoGaze looks at each frame and asks: what is the smallest set of patches that, combined with what I already chose in earlier frames, lets me rebuild this frame within an error budget?

The module is a convolutional encoder plus a four-layer transformer decoder (LLaMA 3-style, hidden dimension 192). Frames are encoded one at a time and the decoder emits patch indices from a vocabulary of 265 possibilities, drawn from four scales (32x32, 64x64, 112x112, 224x224 with patch size 16, giving 4+16+49+196 = 265). A separate head predicts the reconstruction loss after each selection, and decoding stops for a frame once the predicted loss falls below the user-set threshold, so the model decides its own gazing length. Multi-token prediction lets it emit several indices at once.

Training has two phases. First, a greedy search over 250K videos produces approximately optimal gazing sequences, and AutoGaze is pre-trained on them with next-token-prediction cross-entropy (150 epochs, batch size 256, learning rate 5e-4) plus L2 supervision of the loss predictor. Second, because the greedy sequences are only approximately optimal, on-policy GRPO post-training (group size 12, discount factor 0.995, temperature annealed from 1 to 0.01, 3 epochs, batch size 256, learning rate 5e-4) uses reconstruction loss as reward to discover better gazing sequences.

The reconstruction model is an image-MAE fine-tuned on video with block-causal attention so it never depends on future frames. The distance function is a weighted sum of pixel L1 loss and L2 losses on DINOv2 and SigLIP2 embeddings, weighted 1, 0.3, and 0.3. Training data came from Ego4D, 100DoH, InternVid, SA-1B, and IDL, totaling about 800K videos, with videos sampled at 16 frames and 224 resolution.

To run on arbitrary inputs, videos are split into 16x224x224 tiles, AutoGaze runs on each tile independently, and results are merged. To plug into an existing stack, image ViTs are adapted to interpolate frames to multiple scales and to consume all 16 frames in one sequence, after which the tokens feed into the downstream MLLM normally. The paper uses SigLIP2-SO400M as the ViT and NVILA-8B-Video as the MLLM.

Why This Matters

The work relocates token reduction from the LLM to before the ViT, which is where the paper argues the real bottleneck sits. It also pairs that efficiency claim with a new benchmark specifically designed to test whether high resolution (not just long duration) is being used, addressing what the authors describe as a gap where existing long-video benchmarks remain low-resolution.

Real-world applications:

• Autonomous driving, where HLVid's videos come from: a vehicle must parse high-resolution detail over minutes of footage in real time.
• Surveillance and CCTV analysis, where long recordings at high FPS contain mostly static background with sparse events.
• Robotics, where the paper demonstrates tracking on a robot grasping demo, useful for manipulation policies that must react to moving objects.
• Video search, editing, and archival, where indexing five-minute 4K clips at scale depends directly on per-frame token cost.

Industry relevance: The 19x ViT and 10x MLLM speedups and the 4x-100x token reduction translate into fewer GPUs needed per video hour, and the any-resolution tiling scheme means an existing image-ViT-based MLLM can adopt AutoGaze without retraining the whole stack.

Future Directions

• Breaking the reconstruction bottleneck. The method's quality ceiling depends on a VideoMAE-style reconstruction model and a hand-tuned loss threshold; improving or replacing the reconstruction objective is a natural lever the paper does not fully explore.
• Beyond the reconstruction threshold metaphor. The paper fixes the downstream threshold at 0.7 and links it to less than 0.5% degradation; whether a task-aware or prompt-aware threshold could go lower remains open.
• Pushing past 1K frames and 4K. The paper's ceiling is set by the experiments, not by an architectural limit; how far the tiling scheme scales before merging gazed positions across tiles breaks down is not reported.
• Extending to other modalities and settings. The paper demonstrates OOD generalization across semantics and visual styles but does not test streaming, interactive, or audio-visual video settings, and does not report comparisons against adaptive-tokenization methods on the same benchmark.

Target Audience

Researchers and engineers working on efficient multimodal models, long-context video understanding, or vision transformer inference optimization will get the most from this paper. It also suits practitioners designing benchmarks for high-resolution video QA, and team leads evaluating whether to place token reduction before or after a ViT in a production video pipeline. Readers need a working background in ViT architectures, autoregressive modeling, and reinforcement learning to follow the training section in detail.

Authors’ abstract

Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: https://autogaze.github.io/.