VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction

AI summary

Overview

Research area: Computer vision, specifically video event prediction (VEP), multimodal large language models (MLLMs), and tool-augmented reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with MLLM post-training, chain-of-thought supervised fine-tuning, and GRPO-style reinforcement learning with composite reward shaping.

One-sentence scope: The paper introduces VepAgent, a 4B-parameter agentic framework that combines causal-transition chain-of-thought data, a diagnostic visual tool library, and a composite anti-prior reward to predict future video events from partially observed footage.

What This Paper Is About

Video Event Prediction asks a model to anticipate what will happen next after watching only part of a video. Existing multimodal LLMs tend to summarize what they have already seen rather than reason forward, and they often pick answer options that merely sound like the question instead of matching the actual visual state. VepAgent targets these failures by explicitly modeling the transition from the last observed state, through an unfinished ongoing action, to a plausible future event, and by letting the model call diagnostic tools when the visual evidence is ambiguous or incomplete.

Key Contributions

1. Tool-augmented agentic RL framework. The authors unify causal-transition reasoning with dynamic evidence acquisition, using structured supervised fine-tuning plus a Causal Gap Reward so that prediction is future-oriented rather than retrospective.
2. Composite Anti-Prior Reward. A reward term penalizes language-shortcut behavior, giving credit when the model predicts a correct option that is less textually similar to the query, which pushes reliance toward visual evidence rather than surface lexical overlap.
3. Spatiotemporal multi-tool integration. A multimodal tool library combines a State Transition Tracker (STT), Focused Evidence Retriever (FRE), Region Focus Magnifier (RFM), and Visual Detail Extractor (VDE), all operating only on the observed portion of the video.
4. Empirical effectiveness. On FutureBench, VepAgent (4B) raises average accuracy by 14.58 percentage points over the prior best reported MLLM baseline, from 66.86% to 81.44%; on NEPBench, it raises Overall accuracy by 22.70 percentage points, from 47.50% to 70.20%.

Main Findings

• FutureBench results: VepAgent-4B scores 82.08 (1-Hop), 88.60 (2-Hop), 80.10 (3-Hop), 78.94 (Interp.), and 81.44 AVG with 32 frames. The previous strongest reported baseline, Qwen3-VL-30B-A3B, reaches 66.86 AVG; the same-size Qwen3-VL-4B backbone reaches 59.09 AVG, so the gain over the same-size backbone is 22.35 percentage points.

• NEPBench results: VepAgent-4B scores 70.20 Overall, with Charades 83.46%, VidSitu 54.78%, and YouCook2 83.57%. The prior strongest reported baseline is Qwen2.5-VL-72B at 47.50 Overall; the same-size Qwen3-VL-4B backbone reaches 42.17 Overall, a gain of 28.03 percentage points. On Charades the next best reported is GLM-4.1V-9B at 45.98%; on VidSitu the next best is Kimi-VL-A3B at 53.06%; on YouCook2 the next best reported is 64.48% for both GLM-4.1V-9B and Qwen2.5-VL-72B.

• Training stage progression (G=4 ablations): Zero-shot Qwen3-VL-4B gives 59.09 AVG; SFT only raises this to 71.59; SFT plus RL reaches 75.09; the full tool-augmented model reaches 78.79. The authors report that a direct RL attempt without SFT caused tool invocation frequency to decline progressively, motivating the two-stage design.

• Individual tools contribute unequally: Removing STT drops AVG from 78.79 to 75.66 (3.13 points), removing FRE drops it to 76.99 (1.80 points), removing VDE drops it to 77.46 (1.33 points), and removing RFM drops it to 78.31 (0.48 points). STT removal causes the largest single drop.

• Reward components matter: With only the accuracy reward, AVG is 74.62. Adding the causal gap reward raises it to 76.42, and adding the anti-prior reward instead raises it to 76.33. The full combination reaches 78.79.

• More generations help overall: Increasing rollout count in GRPO from 2 to 8 raises AVG from 78.60 to 81.44, though metrics fluctuate; Num = 2 gives the highest 1-Hop score (84.39%) while Num = 8 gives the highest 3-Hop (80.10%) and AVG (81.44%).

• Dynamic tool selection beats static and random: No tools yields 75.09 AVG at 0.00 average calls; invoking all four tools statically yields 74.91 AVG at 4.00 calls (slightly below the tool-free baseline); random selection yields 76.99 AVG at 1.85 calls; dynamic selection yields 78.79 AVG at 1.13 calls.

• Temporal and spatial tools are complementary: Temporal-only tools (STT + FRE) reach 77.84 AVG, spatial-only tools (RFM + VDE) reach 75.38 AVG, and the combination reaches 78.79 AVG.

• Qualitative behavior: In the reported case study, Qwen3-VL-4B predicts a meat-addition event that had already occurred, while VepAgent uses the State Transition Tracker to reconstruct the observed progression and identify the unfinished cooking process, leading to the correct future event.

Methodology in Plain English

The authors start from a simple observation: a model that only summarizes the past cannot reason about an unobserved gap between the final observed state and the next event. Their pipeline has four moving parts.

First, they build training data. Using Qwen3-VL-32B, they run 8 independent policy rollouts per partially observed video, keeping instances where at least one rollout matches the ground truth. Qwen-VL-Max then acts as an expert validator, scoring reasoning chains and discarding ones with factual errors about observed states, logical inconsistencies, or language-prior shortcut behavior. This produces roughly 4,000 validated reasoning chains for SFT, called futurebench-4K. The remaining validated trajectories supply roughly 5,000 samples for RL. All data comes only from the benchmarks' training splits to avoid leakage.

Second, they do supervised fine-tuning on Qwen3-VL-4B-Instruct for one epoch with an effective batch size of 16 and a learning rate of 5e-6, jointly optimizing the reasoning tokens and the final answer. This "cold-start" phase teaches the model to produce structured reasoning and to use tools.

Third, they run GRPO reinforcement learning for 250 steps with 8 rollouts per sample and a learning rate of 1e-6, with KL coefficient 0.04 and clipping ratios 0.2 and 0.3. Each trajectory gets a composite reward gated by a format check: 0.65 times accuracy reward, plus 0.20 times causal gap reward, plus 0.15 times anti-prior reward. The causal gap reward compares the model's terminal-state description and causal-transition description against references using MiniLM sentence embeddings, weighting the transition more heavily (0.55) than the terminal state (0.45), and is then conditioned on whether the prediction was correct so wrong answers cannot collect large process credit. The anti-prior reward computes the rank of the correct option by textual similarity to the query and gives (k-1)/(K-1) when the prediction is correct and there is more than one option, so the most textually obvious answers earn 0 and the least obvious earn 1.

Fourth, at inference the agent follows a two-stage workflow. In stage one it estimates the terminal observed state and ongoing unfinished action, forms a causal hypothesis, and selects a subset of tools from the library. Stage two executes those tools once and folds the returned evidence into the reasoning context before predicting the answer. Training uses eight NVIDIA RTX A5000 GPUs with 24 GB memory each.

Why This Matters

Impact on research: The paper reframes video event prediction from passive future guessing into an active loop of causal analysis, visual verification, and temporal grounding. It also offers a concrete recipe for diagnosing and penalizing language-prior shortcuts via reward shaping, which is a failure mode relevant well beyond VEP. The finding that static invocation of all tools performs slightly worse than using no tools at all (74.91 vs. 75.09 AVG) is a useful caution for the broader tool-augmentation literature.

Real-world applications:

• Proactive robotics, where a system must anticipate the next state of a scene before acting.
• Autonomous driving, where anticipating plausible future events from partial observation is safety-critical.
• Intelligent surveillance, where operators need forward-looking alerts rather than retrospective summaries.
• Instructional and cooking video assistants, an application domain explicitly represented by the YouCook2 subset of NEPBench.

Industry relevance: The result that a 4B model outperforms much larger reported baselines such as Qwen3-VL-30B-A3B and Qwen2.5-VL-72B under the same evaluation protocols is directly relevant to deployment, since smaller models reduce serving cost. The tool-augmented agent design also maps naturally onto production systems that already have frame retrieval and region cropping infrastructure.

Future Directions

• Extending beyond the two benchmarks. The conclusion states explicitly that evaluation covers FutureBench and NEPBench and that broader video understanding settings remain untested, so generalization is an open question.
• Expanding the diagnostic tool library. The authors name this as future work, which raises questions about which additional tools would help and how dynamic selection would scale to a larger library.
• Scaling video understanding to other causal-transition tasks. The paper frames applying causal-transition reasoning to broader settings as a next step, but does not report results there.
• Understanding tool-selection policy more deeply. Dynamic selection averages 1.13 tool calls versus 1.85 for random selection, and the authors attribute this to selective retrieval when confidence in the terminal state and ongoing action is low; the precise conditions under which the agent chooses to call tools are not fully characterized.

Target Audience

Researchers and engineers working on multimodal LLMs, video understanding, and agentic tool-use systems will gain the most, particularly those interested in reinforcement learning post-training and reward design for visual grounding. Practitioners building forward-looking perception for robotics, autonomous driving, or surveillance will also find the tool library and reward formulation directly transferable. Readers should have prior exposure to MLLM fine-tuning and policy-gradient methods to follow the methodology sections in full.

Authors’ abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in video understanding, yet their reliance on retrospective summarization and text-centric priors often limits their ability to bridge unobserved causal transitions when applied to Video Event Prediction (VEP). To address this, we propose VepAgent, an agentic framework that integrates causal-transition reasoning with tool-augmented reinforcement learning (RL) for robust VEP. Unlike prior methods that passively project future trajectories from historical dependencies, our approach explicitly models the logical progression from terminal observed states to future events. Specifically, we first construct futurebench-4K, a high-quality chain-of-thought dataset for supervised fine-tuning (SFT) that effectively bridges the causal-logic gap by structuring the deduction of unobserved intermediate states. Subsequently, we develop a diagnostic tool library integrating state tracking, frame retrieval, and region magnification, enabling the agent to dynamically augment reasoning with external tools to recover missing spatio-temporal evidence and resolve visual ambiguities during inference. Moreover, we propose a composite reward mechanism that jointly optimizes prediction accuracy, causal coherence, and reliable prior, compelling the agent to rely on genuine visual grounding rather than superficial textual similarities. Extensive evaluations on FutureBench and NEPBench datasets demonstrate that our method achieves state-of-the-art performance, significantly outperforming larger MLLMs and validating the empirical effectiveness of our agentic, future-oriented reasoning paradigm.