ReaSon: Reinforced Causal Search with Information Bottleneck for Video Understanding

AI summary

Overview

• Research area: Computer vision, specifically keyframe selection for video question answering with vision-language models (VLMs).
• Technical level: Intermediate. The paper combines mutual-information theory, causal inference (structural causal models, interventions), and reinforcement learning, so it assumes some familiarity with VLMs and reinforcement learning objectives.
• Scope: The paper proposes ReaSon, a framework that redefines which video frames count as "keyframes" using a Causal Information Bottleneck and trains a selection policy with reinforcement learning, evaluated on NExT-QA, EgoSchema, and Video-MME.

What This Paper Is About

VLMs can only accept a limited number of input tokens, and relevant evidence in a video is sparsely distributed across frames, so systems must select a small subset of keyframes. Existing selection methods pick frames that are visually or semantically correlated with the question, but correlation does not guarantee that a frame is actually decisive for producing the correct answer. ReaSon reframes keyframe selection as an optimization problem over two properties — predictive sufficiency (the subset is enough to answer correctly) and causal necessity (removing any selected frame would change the output) — and trains a policy network to find frames that satisfy both.

Key Contributions

1. A formal problem statement. Keyframe selection is formulated as an information-theoretic optimization under a new Causal Information Bottleneck (CIB), which extends the classic Information Bottleneck with an interventional term: max I(S;O) + I_c(O; do(S)) subject to I(V,Q;S) ≤ β. A new bottleneck variable S (target keyframes) replaces the original F so that selection becomes an intervenable decision.
2. The ReaSon framework. A dual-module architecture: a predictive sufficiency module that builds a question-relevant candidate pool and distills a compact subset from it, and a causal necessity module that uses counterfactual interventions to test whether each selected frame is causally indispensable.
3. Three CIB-aligned rewards. An answer reward R_ans (binary correctness), a cycle consistency reward R_cycle (IoU between question-derived visual elements E_q and answer-derived elements E_a), and a counterfactual reward R_cf (KL divergence between VLM logits under the original and counterfactual subsets), combined as R = R_ans + λ₁R_cycle + λ₂R_cf to guide policy learning via reinforcement learning.
4. Empirical validation. Extensive experiments showing ReaSon outperforms non-selection and frame-selection baselines under limited-frame settings, with consistent gains across multiple VLMs.

Main Findings

• NExT-QA and EgoSchema at 8 frames. ReaSon reaches 81.4% overall accuracy on the NExT-QA validation set with LLaVA-Video-7B (temporal 77.3, causal 82.1, descriptive 87.4) and 72.2% on the EgoSchema subset with GPT-4o. With Qwen2.5-VL-7B it reaches 80.4% on NExT-QA and 68.0% on EgoSchema; with GPT-4o on NExT-QA it reaches 77.6%.
• Comparison to non-selection methods. Against VideoINSTA, which uses 90 frames, ReaSon improves by 9.1% and 7.2% while using 8 frames.
• Comparison to prior frame-selection state of the art. Against AKEYS, ReaSon improves accuracy by 3.3% on NExT-QA and 3.6% on EgoSchema. The largest gains appear on temporal and causal questions, with improvements of 4.4% and 3.1% over AKEYS. Under the same frame setting, ReaSon outperforms T* by 5.6% on EgoSchema.
• Video-MME at 8 and 32 frames. ReaSon with GPT-4o achieves the best overall accuracy among comparable configurations at 59.1% (8 frames) and 66.4% (32 frames). The paper reports improvements of 2.6% (8 frames) and 2.3% (32 frames) over state-of-the-art methods. Gains are largest on short videos: at 8 frames ReaSon raises the short-video score from GPT-4o's 55.7% to 65.9% and beats T* by 9.5%; at 32 frames it exceeds GPT-4o and T* by 8.5% and 7.3% on short videos. For medium and long videos at 8 frames it improves GPT-4o by 2.8% and 3.0%; at 32 frames the gains over vanilla GPT-4o are 3.5% for medium and 1.9% for long videos. It surpasses T* by 0.7% on medium videos but falls short on long videos.
• Ablation of reward components. With LLaVA-Video-7B and 8 frames: R_ans alone gives 80.1% on NExT-QA and 66.0% on EgoSchema; adding R_cycle gives 80.5% and 68.2%; adding R_cf as well gives 81.4% and 69.0%. Each reward contributes.
• Generalization across VLMs. Comparing each VLM with and without ReaSon: LLaVA-Video-7B improves from 80.2% to 81.4% on NExT-QA and from 65.2% to 69.0% on EgoSchema; Qwen2.5-VL-7B from 79.9% to 80.4% and from 65.8% to 68.0%; GPT-4o from 72.0% to 77.6% and from 70.0% to 72.2%. The largest overall gains are on Qwen2.5-VL-7B (+3.8% on EgoSchema) and GPT-4o (+5.6% on NExT-QA).
• Qualitative behavior. A visualization on an EgoSchema video shows ReaSon's selection probability forming three prominent peaks, anchoring question-relevant regions plus preceding and succeeding context, whereas T* produces broader and less precise peaks with unnecessary frames.
• Not reported. The paper does not report training wall-clock time, total training compute, the number of training samples used from NExT-QA, or parameter counts for the policy network.

Methodology in Plain English

The starting point is a structural causal model in which the selected frames sit between the video/question and the model's output. The authors argue that the standard Information Bottleneck is insufficient because its bottleneck variable is not something you can intervene on, so they introduce S, the target keyframe subset, and add a term that measures how much the output changes when S is deliberately altered.

Because these mutual-information terms cannot be computed directly, the authors derive tractable surrogates and turn them into rewards:

1. Building a candidate pool. The video is sampled at 1 fps. Uniformly sampled frames are fed into a frozen VLM to extract the visual elements the question refers to. Those elements are matched against every frame using an open-vocabulary detector (YOLO-World), producing a pool of visually relevant candidate frames. This pre-selection follows a heuristic inspired by T*.
2. Selecting a subset. A policy network scores every frame in the pool, and a subset is drawn by multinomial sampling with a cardinality limit |s| ≤ K. This limit acts as the practical version of the compression constraint in the CIB.
3. Scoring sufficiency. The selected frames plus the question go to a frozen VLM. A binary answer reward checks whether the output matches the ground truth. A second reward recycles the predicted answer: the answer is concatenated with the question and sent back to the VLM (without the video) to infer visual elements again, and the IoU between those and the original question-derived elements measures whether the reasoning loop closes.
4. Scoring causal necessity. A counterfactual policy is built by inverting the selection probabilities of the original policy, so frames the policy favored become unlikely and vice versa. A counterfactual subset is sampled and run through the VLM, and the KL divergence between the two output distributions becomes the counterfactual reward. Large divergence means the removed frames mattered.
5. Training. The policy is trained with a group-wise policy gradient: G = 4 subsets are sampled per instance, rewards are mean-centered into advantages (Â_i = R_i − mean of group rewards), and the policy is updated with these advantages. Training uses NExT-QA's training set, with pool size M = 32 and K = 8 frames, λ₁ = λ₂ = 0.5, BLIP embeddings for frames and questions, and a three-layer LSTM plus MLP as the policy network. For Video-MME the pool grows to M = 64 with K = 32 frames.

Evaluation covers NExT-QA (5,440 videos; temporal and causal reasoning on short videos), EgoSchema (5,000 egocentric three-minute videos with public labels for only 500 questions), and Video-MME (average duration 44 minutes, split into short under 2 minutes, medium 4–15 minutes, and long 30–60 minutes). Eight-frame experiments run on a single RTX 3090 (24GB); 32-frame inference needs at least 48GB and uses an A100.

Why This Matters

Impact on research. The paper challenges a widely used assumption in frame selection: that visual or semantic relevance is a good proxy for informativeness. By grounding selection in an interventional causal criterion and showing that the necessity term adds measurable accuracy (+0.9% on NExT-QA and +0.8% on EgoSchema over sufficiency rewards alone with LLaVA-Video-7B), it gives the frame-selection literature a theoretical framing where previously most methods relied on heuristics or architectural intuition. It also demonstrates a small, model-agnostic plug-in that improves three different VLMs, which is useful for researchers who cannot retrain large base models.

Real-world applications implied by the paper's scope (video question answering, captioning, and reasoning; egocentric and long-form video):

• Egocentric assistants and wearables. EgoSchema-style first-person video in three-minute clips is close to what always-on cameras and AR glasses capture; selecting causally decisive frames lowers the compute needed for on-device assistants.
• Long-video search and review. Video-MME's average 44-minute videos resemble surveillance archives, lecture recordings, or recorded meetings where a user asks a targeted question and needs a few frames that actually answer it.
• Accessibility tooling. Accurate question answering over video supports captioning and descriptive systems for users who cannot watch the footage directly.
• Video analytics and moderation. Reducing a full video to 8–32 causally necessary frames cuts token cost and inference time for downstream review pipelines.

Industry relevance. The method is designed for limited-frame budgets (8 frames, or 32 for long video) and runs 8-frame inference on a single 24GB RTX 3090, which matters for deployment cost. Its plug-in nature means a product team can keep its existing VLM and add the selection policy rather than fine-tuning a large model.

Future Directions

• Long videos remain the weak point. ReaSon surpasses T* on medium-duration Video-MME videos by 0.7% but falls short on long

Authors’ abstract

Keyframe selection has become essential for video understanding with vision-language models (VLMs) due to limited input tokens and the temporal sparsity of relevant information across video frames. Video understanding often relies on effective keyframes that are not only informative but also causally decisive. To this end, we propose Reinforced Causal Search with Information Bottleneck (ReaSon), a framework that formulates keyframe selection as an optimization problem with the help of a novel Causal Information Bottleneck (CIB), which explicitly defines keyframes as those satisfying both predictive sufficiency and causal necessity. Specifically, ReaSon employs a learnable policy network to select keyframes from a visually relevant pool of candidate frames to capture predictive sufficiency, and then assesses causal necessity via counterfactual interventions. Finally, a composite reward aligned with the CIB principle is designed to guide the selection policy through reinforcement learning. Extensive experiments on NExT-QA, EgoSchema, and Video-MME demonstrate that ReaSon consistently outperforms existing state-of-the-art methods under limited-frame settings, validating its effectiveness and generalization ability.