From Cause to Effect: Generating Intermediate Event Chains in Videos

AI summary

Overview

Research area: Computer vision and video-language modeling, specifically video causal/narrative reasoning and evaluation benchmark design.

Technical level: Intermediate (the task and metrics are intuitive; the training recipe assumes familiarity with supervised fine-tuning and GRPO-style reinforcement learning).

Scope: This paper introduces Causal Chain Generation, a task where a model must produce the ordered list of intermediate events linking a given start event to a given end event in a video, and provides two evaluation metrics, two datasets, an analysis of current model failures, and a fine-tuning baseline.

What This Paper Is About

Video-Language Models can recognize what happens in a video, but they struggle to explain why events unfold, i.e. to trace the causal steps from an inciting incident to its outcome. Existing evaluations are weak in two opposite ways: multiple-choice QA is easy to score but easy to game with superficial cues, while open-ended dense video captioning is open-ended but scored with metrics (tIoU, F1, CIDEr, METEOR, SODA) that either ignore semantic content or only measure word overlap. The paper's goal is a task that is complex enough to demand genuine causal reasoning yet structured enough to be scored automatically, plus the datasets and metrics needed to measure progress on it.

Key Contributions

1. A new task, Causal Chain Generation (CCG). Given a video, a start event and an end event, a model must generate the sequence of intermediate events that causally bridges the two, each event a natural-language description of one atomic action or state change. The task is built on three principles: minimality (smallest causally sufficient set of events), granularity (one action per event), and context-dependence (descriptions reflect narrative roles, e.g. "the detective retrieves the evidence" rather than "a man picks up an object").

2. Two complementary metrics. SODA-Embedding extends the SODA metric by dropping the temporal-IoU constraint and replacing n-gram (METEOR) matching with cosine similarity between text embeddings (Qwen3-Embedding), computing an order-preserving alignment by dynamic programming and reporting its F1. Causal-QA builds a bank of five-way multiple-choice questions per reference chain whose answers are recoverable only from the intermediate events, then measures the accuracy of a fixed reader LLM given only the model's generated chain.

3. Two datasets. SF20K-CC is a manually annotated diagnostic set of 20 full silent animation short films from SF20K (205 chains, mean length 7.67 events, 537 MCQs, mean chain duration 96.9 s). CMD-CC is derived from audio descriptions of Condensed Movies: a test split of 471 videos from 98 movies (1,031 chains, mean length 6.17, 2,001 MCQs, 69.1 s) and a train split of 6,497 videos from 1,320 movies (14,175 chains, mean length 6.12, 26,726 MCQs, 68.6 s). Chains are labelled with four causality types drawn from an existing taxonomy: Physical, Motivational, Psychological, and Enabling.

4. A baseline training recipe and failure analysis. Supervised fine-tuning plus GRPO with a composite verifiable reward (format reward + SODA-Embedding + Causal-QA accuracy), with an ablation of each reward term, plus a systematic study of where state-of-the-art VLMs fail.

Main Findings

• Strong VLMs struggle on the task. On CMD-CC, base Qwen3.5-9B scores 54.6 SODA-Embedding and 21.5 Causal-QA; on SF20K-CC, 52.7 and 16.5. The best baseline cited is a 27B Qwen3.5 model at 59.9 SODA-Embedding on CMD-CC, which the authors describe as leaving substantial headroom.
• Proprietary models lead on causal coherence. GPT-5.4-mini reaches the highest Causal-QA overall (29.5 on CMD-CC, 23.4 on SF20K-CC); Gemini-3.1-Flash-lite reaches 61.6 SODA-Embedding and 26.4 Causal-QA on CMD-CC, and 56.0 / 22.6 on SF20K-CC.
• Chain-of-thought can hurt chain quality. Qwen3.5-9B-Thinking obtains the best Causal-QA among non-fine-tuned open models (24.9 on CMD-CC, 21.7 on SF20K-CC) but much lower SODA-Embedding (40.3 and 40.0), which the authors attribute to verbose reasoning diverging from the reference.
• Decomposing the task loses information. Two-stage pipelines (caption-to-chain and graph-to-chain) underperform the corresponding single-pass VLM on CMD-CC (45.9 and 47.4 SODA-Embedding, 8.5 and 11.2 Causal-QA).
• SODA-Embedding tracks human judgment better than n-gram metrics. Over 256 stratified A/B pairs (128 CMD-CC, 128 SF20K-CC) with three annotators (Fleiss κ = 0.881), SODA-E agrees with humans 79.2% of the time versus 61.8–72.0% for METEOR, CIDEr and ROUGE-L, and 66.1% for its own n-gram ablation SODA-C. It improves with embedder quality, from 76.2% with MiniLM to 79.2% with Qwen3-Embedding-4B.
• Causal-QA questions are answerable by humans. Two annotators answering 150 sampled questions given the full ground-truth chain reached 94.2% average accuracy on five-way MCQs (chance 20%), with Fleiss κ = 0.853.
• Rankings are robust to auxiliary-model choice. Swapping the embedder (Qwen3-Embedding 0.6B/8B or all-MiniLM-L6-v2) preserves system ordering (≥ 91% rank agreement, Spearman ρ ≥ 0.91); a 7B Qwen judge and cross-family Gemma-3-12B and Llama-3.1-8B judges reproduce the default Qwen2.5-3B ranking (≥ 85% agreement). A 0.5B judge is too weak and collapses to near-chance agreement on SF20K-CC.
• SFT and GRPO trade off different strengths. On CMD-CC, SFT lifts SODA-Embedding to 62.3 (+7.6) but only moves Causal-QA to 24.6; GRPO from the base model reaches 32.2 Causal-QA at 58.9 SODA-Embedding; SFT + GRPO gives the best balance (63.3 SODA-Embedding, 29.4 Causal-QA). On SF20K-CC the three settings give 56.6/19.3, 57.2/24.9 and 57.9/22.1 respectively.
• The reward you optimize is the metric you win. Format-only GRPO regresses below the base model (55.4/17.7 on CMD-CC, 53.9/12.9 on SF20K-CC). Format + SODA-Embedding maximizes SODA-Embedding (60.1 and 58.8). Format + Causal-QA maximizes Causal-QA (37.8 on CMD-CC, the highest of any open-weights configuration trained, and 26.0 on SF20K-CC). The full reward is best balanced (58.9/32.2 and 57.2/24.9).
• Dialogue, not chain length, drives difficulty. Models score +11 to +20 Causal-QA points higher on with-dialogue clips than on silent ones, and on the former a text-only control rivals the video model; stripping subtitles from Qwen3.5-9B costs −4.2 on CMD-CC. The silent subset is therefore the harder test of visual causal reasoning.
• Scaling saturates early. Performance rises steeply with model size up to 4B and then flattens; returns from more input frames are roughly logarithmic, with Causal-QA the more frame-hungry axis.
• Common failure modes are omission and hallucination. Models frequently drop critical events, invent non-existent ones, and produce causally incoherent sequences.

Methodology in Plain English

The authors first define the output format: a model sees a video plus a start and an end event, and must write out the ordered chain of events in between. Because the answer is a structured list rather than a free-form paragraph, it can be scored automatically.

For scoring, they use two checks that catch different mistakes. The first, SODA-Embedding, compares the generated chain to a reference chain by embedding each event as a vector, computing similarity between every predicted and reference event, then finding the best alignment that preserves order. That gives a soft count of correct matches, from which precision, recall and their F1 are computed. Because it uses embeddings rather than word overlap, paraphrases are not penalized, but order errors are.

The second, Causal-QA, tests whether the chain actually contains the causal information needed to explain the story. For each reference chain, a generator LLM (Qwen3.6-27B) proposes 6–10 five-option questions where option E is always "Cannot be determined from the chain" and distractors are built from fixed confusion types (wrong agent, wrong instrument, swapped order, inverted polarity). A fixed judge (Qwen2.5-3B-Instruct) keeps only questions that pass three filters: answerable from the full chain, not answerable with no context, and not answerable from the start and end events alone. At evaluation time the same fixed judge answers those questions using only the model's generated chain.

Data is built with a three-step protocol: obtain dense audio descriptions (written manually for the silent films, reused from CMD-AD for movie clips), prompt a large language model to build a causal graph over the video's events, then extract start-to-end chains from that graph and attach a Causal-QA bank. The CMD-CC test split was additionally manually curated.

For training, the authors fine-tune Qwen3.5-9B with LoRA (r = 16, α = 32, on q/k/v/o projections) for one epoch of supervised fine-tuning on the reference chains, then one epoch of GRPO. GRPO samples a group of candidate chains per query and updates the policy using group-relative advantages, with a reward that sums three non-differentiable signals: a binary JSON-format reward, the SODA-Embedding score, and Causal-QA accuracy from the fixed judge.

Why This Matters

Causal reasoning about video is a capability that current benchmarks largely fail to measure: multiple-choice formats reward guessing and captioning metrics reward word overlap. This paper supplies both a task and metrics designed so that a system can only score well by producing a coherent causal explanation, and it shows that today's best models, including proprietary ones, still fall well short.

Real-world applications:

• Video summarization and editing that condenses a long recording into the events that actually explain the outcome, rather than every salient moment.
• Accessibility and audio description, where a system must narrate the causally relevant events of a scene for blind or low-vision viewers; the paper explicitly builds on audio-description data.
• Surveillance and incident review, where the useful output is the chain of events leading to an accident or failure, not just detection of individual actions.
• Instructional and training video analysis, where the value lies in explaining how a step leads to a result, for example in safety or procedural training.

Industry relevance: the work gives teams a scoring harness for video reasoning that resists shortcut answers, a training recipe using reinforcement learning with verifiable rewards (an approach already common in language-model post-training but still open for video), and two released datasets with train and test splits for building and comparing systems.

Future Directions

• Closing the gap between chain quality and causal accuracy. The authors note that SFT and GRPO each improve one metric at the expense of the other, and that dense captioning and causal understanding appear to be distinct capabilities; a reward or training scheme that improves both simultaneously is an open problem.
• Scaling and enriching the diagnostic benchmark. SF20K-CC has only 20 films and 205 chains, limiting statistical power per split and per causality type, while CMD-CC clips have truncated narrative arcs; larger manually annotated sets with complete arcs would strengthen both.
• Reducing dependence on auxiliary models. Absolute scores depend on the frozen embedder and judge, and a judge of 0.5B parameters cannot discriminate between systems, so metrics that are less model-dependent, or better calibrated across judges, remain to be developed.
• Establishing human performance on the task itself. The paper validates metric–human agreement and question answerability, but does not report human performance at generating causal chains, which would give a ceiling to compare models against.

Target Audience

Researchers and practitioners in video-language modeling, video understanding benchmark design, and multimodal reinforcement learning. It is most useful to those building or evaluating video reasoning systems, to teams designing automatic metrics for open-ended generation, and to anyone applying verifiable-reward RL to multimodal models. Readers mainly interested in applications, such as video summarization or accessibility narration, will find the task framing and failure analysis accessible, while the metric implementation details and the reward ablation require some familiarity with evaluation methodology.

Authors’ abstract

Video-Language Models (VLMs) excel at visual recognition but struggle with causal reasoning. Current evaluations are flawed: multiple-choice QA allows shortcuts, and open-ended QA lacks robust automatic evaluation. We introduce Causal Chain Generation (CCG), a novel task where models have to generate the sequence of intermediate events connecting a cause and an effect. This structured setup demands genuine reasoning and enables reliable automatic scoring. We accompany this task with two complementary metrics: SODA-Embedding, which measures the semantic matching of a generated chain against a reference, and Causal-QA, which probes the chain's causal coherence by asking whether it contains the specific causal facts needed to reconstruct the narrative. We provide two test sets: SF20K-CC, a manually annotated diagnostic on 20 short films, and CMD-CC, a movie clip benchmark derived from audio descriptions. Our analysis reveals that state-of-the-art VLMs struggle with narrative causality understanding, frequently hallucinating or omitting key events. To address this, we fine-tune VLMs with GRPO using a composite reward that combines a format reward, SODA-Embedding, and Causal-QA accuracy, demonstrating substantial improvements over strong baselines.