Mind The Gap (MTG-AD): Dataset and Fine-grained Evaluation for End-to-End Inline Audio Descriptions

AI summary

Overview

Research area: Computer vision and accessibility — specifically multimodal large language models (MLLMs) generating inline audio descriptions (AD) for user-generated video, plus a benchmark dataset and evaluation framework.

Technical level: Intermediate. Comfortable for readers who know basic multimodal model concepts (video-in, text-out, TTS rendering) and standard evaluation metrics; no deep mathematical background required.

Scope (one sentence): The paper introduces MTG-AD, a 36-hour professionally co-authored dataset of inline audio descriptions for 433 YouTube videos, and MTG-AD-Eval, a reference-free, end-to-end evaluation suite that renders generated scripts to speech to measure timing, overlap, and quality against expert guidelines — then validates both against sighted raters and blind and low-vision (BLV) participants.

What This Paper Is About

Blind and low-vision viewers rely on audio descriptions to follow video, but professional AD is rarely available for user-generated content. Inline AD is harder than ordinary captioning because descriptions must be squeezed into the natural silences of the existing soundtrack without talking over dialogue or leaving long dead air. The paper asks whether frontier multimodal LLMs can do this end-to-end — generate a timestamped script, synthesize it, and have it actually fit the video — and provides the data and measurement tools needed to answer that question with evidence rather than intuition.

Key Contributions

1. MTG-AD dataset: 433 English YouTube videos totaling roughly 35.9 hours (mean 5.0 minutes), containing 8,126 AD segments (6,214 validation, 1,912 test), co-authored by two-person teams each pairing one sighted and one blind professional AD writer. Split into 332 validation videos for method development and 101 test videos for multi-model evaluation; includes timestamped ASR transcripts and pre-extracted dense visual event lists.

2. MTG-AD-Eval framework: A reference-free evaluation suite derived from expert AD guidelines (American Council of the Blind, DCMP's Description Key, ADLab's film guide, W3C's WCAG, and YouDescribe's tips), distilled to ten guidelines, six of which are turned into automatic metrics: Prioritization (G1.1), No Redundancy (G1.2), On-Screen Text (G1.3), Timeliness (G2), Audio Overlaps (G7), and Audio Coverage / Avg. Silence (G8).

3. End-to-end audio rendering: Unlike prior work that scores text against a reference on trimmed clips, the framework synthesizes each AD segment with an off-the-shelf TTS engine at a fixed 1.2× speaking rate and measures how the resulting audio track interacts with the original video's speech and silences across the whole video.

4. Human validation: Empirical correlation of the automated metrics with ratings from 8 sighted raters (42 videos, 11 dimensions) and 7 BLV participants (n = 28 video-level evaluations), plus analysis of cross-judge bias and transfer to the external YouDescribe dataset.

Main Findings

• Guideline prompts improve content but can wreck timing. On Gemini-2.5 Flash validation runs, the minimal "Simple AD" prompt under-describes (Coverage 0.41, Avg. Silence 17.95 s, On-Screen Text 0.47). "Guideline only" raises Prioritization to 0.79 and On-Screen Text to 0.69 but over-generates badly — Coverage 1.23 and 59.19 s of speech overlap — a failure mode content metrics alone would miss.

• Explicit timing reasoning cuts overlap. Adding timestamped ASR ("Guideline + ASR") reduces overlap to 21.34 s; the "Gd. + Timing ASR" prompt, which computes each silent gap's duration and caps output at 3 words per silent second (at most three sentences), reduces it further to 12.77 s while keeping Prioritization at 0.76.

• Human AD is still better timed, but scores slightly lower on the automatic prioritization metric. Professional AD yields 7.51 s of overlap and 4.19 s Avg. Silence, with Prioritization 0.73 — lower than some models because the dense visual event list itself is MLLM-extracted and misses subtle human-observed details (e.g., a truck braking, a seal blinking).

• Native audio-video models lead on content. Across nine frontier models on the 101 test videos, Qwen-3.5 and Gemini models score 0.76–0.94 on Prioritization versus 0.51–0.59 for the others.

• Timeliness splits by model family. Recent Qwen-3.5 and Gemini-3.5 describe events within 2–5 s of when they appear; Qwen-2 and Gemini-2.5 stay within 5–7 s; Claude 4/4.5, GPT-5.2, and GPT-4.1 mini drift 20–65 s away.

• No model wins on every metric — models trade overlap against coverage. Gemini-3.5 Flash nearly eliminates overlap (0.26 s) by describing less (Coverage 0.45) and achieves the shortest Avg. Silence (4.85 s). Gemini-2.5 Flash balances higher silence use (Coverage 0.74) with 8.92 s of overlap and 6.63 s Avg. Silence. Qwen models fill more silence (Coverage 0.72–0.94) at the cost of 30.45–52.14 s of overlaps.

• On-screen text is uniformly mediocre. No model escapes the 0.56–0.64 range across all nine, regardless of family.

• Reference-based scoring under-rates good descriptions. Holistic ratings average 2.0–3.3 on Descriptive, Objective, and Clear but only 1.16–1.63 on Reference Alignment (the "Accurate" dimension when judged against human AD), because valid details the human describer omitted are penalized.

• LLM judges rank systems reliably but are not calibrated alike. Inter-judge Pearson correlations range r = 0.38–0.74 across Claude, Gemini, GPT, and Qwen judges, with GPT-4.1 mini and Qwen-2-VL-Max agreeing most; some judges are systematically lenient and others harsh, so comparisons are valid only within a single judge.

• Metrics transfer to YouDescribe. "Premium" human AD trends ahead of "Standard" volunteer AD on Timeliness (2.16 s vs. 2.85 s) and Prioritization (0.82 vs. 0.80), though confidence intervals overlap.

• Timing metrics predict sighted rater judgments. Audio Overlaps tracks ratings of audio issues (ρ = −0.467, p = 0.0018) and Avg. Silence tracks ratings of quiet gaps (ρ = −0.415, p = 0.0063); VideoA11y-Ref Clear (ρ = +0.453) and Descriptive (ρ = +0.328) also correlate significantly.

• Reference-free scoring fixes a negative correlation on BLV accuracy. VideoA11y-Ref Accurate correlates negatively with BLV perceived accuracy (ρ = −0.264); the reference-free Quality-RF variant flips this positive (ρ = +0.244) and substantially strengthens agreement on Descriptive (ρ = +0.583, p = 0.001) and Clear (ρ = +0.415, p = 0.028). For BLV listeners, No Redundancy is also a significant predictor (ρ = +0.386, p = 0.0427).

• BLV users find model-generated AD nearly as good. Participants rated both versions consistently high; 53.6% marginally preferred the human-authored version, and Wilcoxon signed-rank tests showed statistically significant differences only for overall Satisfaction (p < 0.05) and perceived Accuracy (p < 0.05), with near-parity on Quality, Clarity, and Objectivity. Preferences varied by genre: several preferred AutoAD's denser detail for action or travel clips, and human AD's pacing for dialogue-heavy vlogs.

• Sighted rater agreement is moderate-to-good. Krippendorff's α across all metrics was 0.654, led by Timeliness (0.721) and Audio Issues (0.700).

Methodology in Plain English

Building the benchmark. The authors sampled 557 candidate English YouTube videos of roughly 3–7 minutes from channels with at least 1,000 subscribers, aiming for topical diversity and temporal viability (a video is viable only if descriptions can be inserted without pausing playback). Of these, 438 (78.6%) were annotated, 92 (16.5%) were rejected as non-viable, and 27 (4.8%) were skipped as near-duplicates; removing duplicate annotations left 433 unique videos. Each script was produced in a three-step professional workflow: the sighted writer watched the whole video for narrative context, then drafted timestamped descriptions of key visual events and on-screen text; then imported the script into an audio editor against the original soundtrack to adjust phrasing and start times; then the blind writing partner audited the timed playback for standalone comprehension before sign-off.

Generating model AD. Models are prompted on untrimmed multi-minute videos with four escalating prompt designs — minimal instructions; expert AD rules; rules plus a timestamped speech transcript with a 1–3 words/second target; and a version that explicitly computes each silent gap's duration and budgets words per gap. Nine frontier models are compared under the strongest prompt on the 101 test videos.

Evaluating the audio, not just the text. Each generated segment is rendered to speech at a fixed 1.2× rate to get its real spoken duration. The framework then computes how many seconds of AD overlap speech, what fraction of available silence the AD covers (unclipped, so values above 1 flag over-generation), and how long the remaining silent stretches are. A separate LLM judge pairs each AD segment to the visual event it describes to measure timing error, and rates each segment for salience and redundancy against the extracted visual event list and the transcript.

Judging whole-script quality without a reference. The authors take the four VideoA11y rubric dimensions (Descriptive, Objective, Accurate, Clear) and score two ways: the original variant comparing the script to human AD, and a reference-free variant showing the judge the raw video frames and audio plus the ASR transcript, asking it to summarize, reason, and then score.

Validating against people. Sighted raters verified visual-to-text faithfulness and timestamp alignment across many model/prompt pairs, while BLV participants listened to paired human and model AD tracks in randomized, counterbalanced order without knowing which was which, rating each and stating a preference. Spearman correlations between metrics and human ratings test whether the automated numbers mean what they claim.

Why This Matters

Research impact. Prior AD benchmarks evaluate trimmed 2–20 s clips against a single reference, which misses sequence-level coherence, real spoken duration, dead air, and speech overlap — the things that actually shape a listener's experience. This paper reframes the task as an end-to-end audio problem and shows empirically that reference-based "accuracy" scoring can be negatively correlated with what blind listeners perceive as accurate. It also releases 36 hours of professionally co-authored ground truth where prior user-generated-video datasets relied on volunteers, synthetic MLLM output, or repurposed instructional narration.

Real-world applications:

• On-demand audio description for the millions of user-generated YouTube videos that will never receive professional AD.
• Two-stage model selection in production accessibility pipelines: a hard filter on speech overlap and coverage, then optimization of content richness among feasible candidates.
• Automated quality monitoring of human- or volunteer-authored AD archives such as YouDescribe, flagging timing problems without a reference script.
• Prompt and harness design guidance for any system that must fit generated speech into constrained time windows, such as live captioning or narration for slideshows.

Industry relevance. The dataset and evaluation code are released publicly (huggingface.co/datasets/google/mind-the-gap-inline-ad and github.com/google/mtg-ad) by a Google Research team that includes accessibility-focused co-authors. The finding that supplying a timestamped ASR transcript plus explicit per-gap word budgets is what closes most of the overlap gap gives platform teams a concrete, inexpensive recipe. The validated reference-free metrics also matter commercially: they mean generated AD can be scored at scale without paying for human reference scripts or expert listening studies.

Future Directions

• Extend the metric coverage. The automatic metrics cover six of the ten distilled guidelines, leaving active voice (G5) and prosody (G10) unmeasured — both plausible contributors to the narrative-pacing gap the paper identifies.
• Reduce dependence on MLLM-extracted event lists and judge calibration. Prioritization and Timeliness both rely on a Gemini-2.5 Flash-generated visual event list and on judge behavior, which can under-credit subtle human-observed details and cause score shifts when the judge changes.
• Broaden the human studies. The BLV evidence rests on 7 participants and 28 full-length video evaluations; larger cohorts across more genres would better characterize the individual preference differences the authors observed.
• Move beyond a fixed speaking rate. All temporal metrics use a fixed 1.2× TTS rate, so absolute overlap and coverage values depend on it; faster or adaptive per-gap speech could fit richer descriptions and is left open.

Target Audience

Accessibility researchers and practitioners working on audio description, blind and low-vision assistive technology, and video accessibility standards; multimodal LLM researchers interested in long-video, timing-constrained generation and in LLM-as-judge evaluation design; benchmark and dataset builders who need reference-free metrics validated against human listeners; and product engineers building automated description or narration features who want a concrete two-stage selection recipe and an off-the-shelf evaluation suite.

Authors’ abstract

Audio descriptions (AD) are essential for making visual media accessible to blind and low-vision (BLV) users. While Multimodal Large Language Models (MLLMs) offer a scalable solution for generating audio descriptions on-demand, their performance on ``inline'' audio descriptions -- which must fit within existing silences in a video -- remains under-explored, particularly for diverse user-generated content (UGC). We present a comprehensive investigation into MLLM-generated inline audio descriptions. First, we introduce MTG-AD, a professionally annotated dataset of 36 hours of videos, each averaging $\sim$5 mins. and 8k+ descriptions in total, co-created by paired blind and sighted AD professionals to provide a high-quality benchmark for this task. Second, we propose MTG-AD-Eval, an evaluation framework based on audio description expert guidelines that provides fine-grained, reference-free actionable metrics for model improvement. Crucially, our framework evaluates the end-to-end task: it renders the generated scripts into audio tracks to assess their fit within the original video's natural silences, minimizing disruptive overlaps. Our analysis across frontier MLLMs reveals that while models accurately describe visual events and substantially reduce speech overlaps, gaps persist in narrative pacing and flow. Finally, we empirically validate that our automated and reference-free metrics correlate significantly with human quality ratings from sighted raters and 7 BLV participants, finding that while users narrowly prefer human-authored audio descriptions, they find MLLM-generated audio descriptions highly beneficial for comprehension. We provide actionable system design guidance and release MTG-AD at https://hf.co/datasets/google/mind-the-gap-inline-ad and MTG-AD-Eval at https://github.com/google/mtg-ad.