REFRAMED: Towards Realistic Audio Description Generation for Movies

AI summary

Overview

• Research area: Natural Language Processing, multimodal video understanding, accessibility (audio description), and benchmark/dataset construction.
• Technical level: Advanced. The paper assumes familiarity with ASR pipelines, temporal alignment, multimodal LLMs, and captioning metrics such as CIDEr, METEOR, and SODA.
• Scope: The paper reformulates audio description (AD) generation as a joint "when and what to describe" decision problem and introduces the Re d dataset, challenge benchmark, and adapted evaluation metrics for it (arXiv:2608.09765v1, 10 Aug 2026, CC BY 4.0).

What This Paper Is About

Audio Description is the spoken narration of key visual content inserted into the gaps between dialogue, allowing visually impaired audiences to follow a film. Existing systems treat this as clip-level video captioning: they are told in advance which clip to describe and when to place the description, so they never have to make the editorial decisions that real describers make. This paper reframes the task so a model must itself decide which moments need a description and what to say in them, and provides the data and metrics to measure that.

Key Contributions

1. A realistic task formulation. AD generation is posed as a joint decision problem: given video, dialogue subtitles, and the spans of dialogue gaps, the model must decide both when to describe and what to describe. Evaluation metrics that exploit multiple references accompany this formulation.
2. The Re d dataset. 2,023 video excerpts spanning 3,302 scenes from 206 movies, with dual AD versions (US and UK), professional dialogue subtitles, SDH subtitles, aligned screenplays, and video.
3. The Re d challenge set. 10 fully manually annotated movies with 2–3 professionally transcribed AD versions each, scene boundaries, AD-to-scene labels, screenplay alignment, and professional subtitles.
4. A suite of benchmarks. Evaluations of ASR quality, temporal alignment, screenplay alignment, scene segmentation, AD splitting, and AD generation on the above data.

Main Findings

• Human agreement is high despite AD's subjectivity. A third professional AD transcript scored QEval = 69.6% and QEval-T = 61.2% against two other references, with CIDEr = 51.4 and METEOR = 20.1.
• Random descriptions receive no undeserved credit. The random baseline scored CIDEr = 1.3, METEOR = 4.4, SODA-M = 4.0, and QEval-T = 2.3%. The LLM evaluator still answered 34.1% of questions correctly from random text (chance-level = 20%), and an empty-context baseline reached 41.4% QEval with all other metrics at zero, showing parametric knowledge and common-sense reasoning.
• Existing AD systems beat random but remain far below human performance. DistinctAD scored CIDEr = 15.6 and Shot-by-Shot 16.3 (vs. random 1.3); Shot-by-Shot outperformed DistinctAD on both QA-based metrics (both p < 0.001). These results are an upper bound, because these models are given gold-standard AD placement and content selection.
• LLMs work best with ten-minute chunks, not full movies. Qwen 3.5 scored CIDEr = 13.2 and QEval-T = 17.3% with chunked input versus 10.3 and 9.4% with full-movie input; Gemini 3.1 scored 19.0 and 16.4% with chunked input versus 12.2 and 10.6% with full-movie input. Gemini's chunked CIDEr of 19.0 was the highest among tested systems.
• Temporal grounding degrades with full-movie input. SODA-T was 26.0% for Qwen and 27.1% for Gemini with full-movie input, versus 38.4% and 36.0% with chunks.
• ASR quality in prior corpora is poor. WhisperX achieved the best overall word error rate at WER = 12.8; the MAD-v2-eval transcriptions achieved 22.1; a Speechmatics-based system with character-name adaptation achieved WER = 2.9 and was used to transcribe all training data.
• Pipeline components were validated. Temporal alignment reached 100% accuracy to within one frame for scenes of 32 seconds or longer (over 99% of video extracts); AD splitting achieved F1 = 64.8 versus a comma-splitting baseline of 36.9; scene-boundary detection achieved F1 = 59.2; screenplay alignment achieved F1 = 80.9 versus a dialogue-only baseline of 63.8.
• Descriptions lag behind the events they describe. AD narration occurs within 10 seconds of the described element but not at the same instant, reflecting the need to fit narration into dialogue gaps; this motivated τ = 10 seconds in SODA-T and QEval-T.
• Gap-based assignment covers most references. 90.2% of reference description elements in the challenge set fall in dialogue gaps of at least 1 second: 87.8% (N = 10,644) for American AD and 93.1% (N = 8,734) for British AD.
• Models miss narrative essence and run long. In a Harry Potter and the Goblet of Fire (2005) example, both LLMs retrieved the correct moment but failed to convey the narratively salient detail both references included; Qwen hallucinated an action and Gemini named the wrong character. Gemini's generations implied 110–118 words per minute versus roughly 200 words per minute for professional AD in the challenge set; generations ran 21–26 words against references of 34–35 words, while pre-existing systems produced 30–47 words that were less coherent and faithful.
• Test-set scores are higher, and depend on character information. Gemini reached CIDEr 22.5 and QEval-T 23.1%; Qwen reached CIDEr 18.0 and QEval-T 21.7%. Removing the provided character names and face crops dropped CIDEr to 14.5 (Gemini) and 12.8 (Qwen), both p < 0.001.
• Safety guardrails interfere with legitimate AD. Gemini's non-configurable guardrails triggered on the full movie Charlie St. Cloud (2010), on two 10-minute chunks of the same movie, and on two excerpts from Speak No Evil (2024), likely due to detection of depictions of harm involving minors.

Methodology in Plain English

The researchers start by rebuilding the task definition. Instead of handing a model a pre-selected clip and asking for a caption, they give it the movie video, the timed dialogue subtitles, and a list of the gaps between dialogue that are longer than one second, and ask it to produce zero or more timed descriptions per gap.

To make that possible, they assemble data from movie excerpts licensed by Fandango and available on rottentomatoes.com, choosing movies with at least five excerpts and with both American and British AD versions listed on audiovault.net. They exclude animated movies, following prior work. They then run a pipeline: transcribe the AD tracks, align the excerpts to the full AD track using cross-correlation while correcting PAL–NTSC frame-rate differences, split AD transcripts into single-element segments, predict scene boundaries, align dialogue-only and SDH subtitles from opensubtitles.org using ASR-based timestamp matching, and align screenplay scenes to movie scenes with Vecalign over joint dialogue and AD representations. All evaluation data uses professional human transcription rather than ASR output.

For measurement, they combine three families of metrics. Dialogue-gap metrics assign references and generations to the gaps that contain them and compute CIDEr (restricted to gaps under 20 seconds because of its length penalty) and METEOR (maximum across references). Alignment-based metrics adapt SODA: an optimal one-to-one monotonic alignment between reference and generated elements is found by dynamic programming using only METEOR similarity, giving SODA-M (F1 over METEOR scores) and SODA-T (proportion of matched pairs whose time midpoints differ by less than 10 seconds). QA-based metrics use OLMo 3 32B Think to generate multiple-choice questions from reference AD, filter them so they are answerable from references but not when the supporting segment is removed, and score QEval (answer correctness) and QEval-T (correctness plus temporal alignment), with round-trip consistency of 92.6%.

They then benchmark random and greedy baselines, two pre-existing AD systems given gold-standard placements, and two LLM families (Qwen 3.5, a 27B dense open-weight model run via vLLM on two H200 GPUs, and Gemini 3.1 Flash-Lite with thinking=high) on one frame per second of video and no audio. Because generation degenerates after roughly ten minutes of script (about 1k tokens), models are instructed to work in ten-minute chunks, with prior generations fed back in, and chunked input is compared against supplying the full movie as context. Significance is tested with two-tailed Monte Carlo permutation tests (R = 10,000, α = 0.05).

Why This Matters

Impact on research. The paper argues that prior AD datasets share three unvalidated weaknesses: automatically produced AD transcripts whose quality was never evaluated, unvalidated alignment of AD tracks to scenes, and an evaluation set inheriting LSMDC timecoding, transcription errors, and character-name replacement. Re d supplies gold-standard transcription, multiple references for both training and evaluation, and richer parallel streams (screenplays, dialogue subtitles, SDH) than LSMDC, MAD, MAD-v2-eval, MAD-v3-eval, or CMD-AD. It is presented as the first benchmark requiring placement and content to be decided jointly.

Real-world applications:

• Automating AD production for streaming catalogues, where the UK's Media Act sets a streaming quota of 10% by 2030 and Title II of the Americans with Disabilities Act will require public entities to provide AD for pre-recorded video.
• Producing both American and British AD versions, which the dataset supports directly since it contains both.
• Supporting AD for hard-of-hearing audiences and other accessibility workflows, through the included SDH subtitles.
• Evaluating and diagnosing multimodal video-understanding systems on long-form, narratively structured content rather than short clips.

Industry relevance. Professional AD is tightly constrained: it must fit dialogue gaps and stay near roughly 200 words per minute, and the paper shows current LLM output would need to be spoken at 110–118 words per minute. That gap, plus the fact that LLMs handle a ten-minute chunk better than a full movie, identifies concrete engineering problems for anyone building AD tooling, and the guardrail failures on Charlie St. Cloud and Speak No Evil flag a deployment obstacle for proprietary models.

Future Directions

• Teach models to use full-movie narrative context. LLMs currently perform worse with the full movie as input than with ten-minute chunks, so retrieving and tracking narrative context across hours remains unsolved.
• Improve temporal placement. SODA-T and QEval-T both fall when full-movie input is used, showing precise grounding is not yet achievable at that input length.
• Close the content-selection and realization gaps. The conclusion names three coupled challenges: content selection (finding narratively salient characters, objects, actions, and details), temporal placement (fitting descriptions into dialogue or salient audio), and AD realization (being informative while matching available narration time).
• Extend cross-lingually. The dataset is English-only; the authors expect the formulation and evaluation setup to transfer, but note stylistic conventions vary by region, and independent AD tracks in other languages exist for some of their movies.
• Address safety-guardrail interference. Gemini's non-configurable guardrails blocked legitimate AD content, an open problem for applying hosted models to accessibility tasks.

Target Audience

Researchers in multimodal video understanding, video captioning, and long-form narrative modeling; accessibility and AD researchers and practitioners; and engineers building AD automation for studios and streaming platforms. It is also relevant to anyone interested in dataset and benchmark design for subjective, multi-reference generation tasks, since the paper's metric adaptations (dialogue-gap CIDEr/METEOR, SODA-M, SODA-T, QEval, QEval-T) are as much a contribution as the data itself.

Authors’ abstract

Audio Description (AD) is a verbal narration of key visual content in videos, enabling access for visually impaired audiences. Unlike standard video captioning, AD is a structured editorial task: descriptions must be inserted into gaps in dialogue and must convey only what is needed to understand the narrative being told. However, existing approaches formulate AD generation in an artificial setting where both the content and timing of descriptions are pre-specified, reducing the task to clip-level captioning. They further rely on noisy transcription and alignment pipelines, and lack the rich parallel data required for modeling narrative context. We introduce a new formulation of AD generation in which models must jointly decide what to describe and when to do it. To support this, we present REFRAMED, a high-quality dataset of 2,023 videos that span 3,302 scenes from 206 movies, with professional AD transcripts (both American and British versions), professional subtitles and aligned screenplays. We also provide a manually curated challenge set that pairs full movies with multiple AD references, together with evaluation protocols that leverage dialogue gaps and multi-reference comparisons. Experiments with state-of-the-art AD systems and multimodal LLMs show that they outperform trivial baselines but fall far short of expert human performance. Our dataset and benchmark establish a new foundation for research on video understanding.