As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does

AI summary

Overview

Research area: Computer vision and multimodal video understanding — specifically character identification in feature-length films from audio-visual content alone.

Technical level: Intermediate. Readers should be comfortable with multimodal large language models (MLLMs), face tracking/clustering, speaker diarization, and reinforcement learning from verifiable rewards (GRPO); the paper's framing and motivation are accessible without that background.

Scope: The paper introduces the "character naming" task — linking a name spoken in dialogue to the correct on-screen face — together with a dataset (MCN), a fine-tuned MLLM solution, and a streaming pipeline that builds a character bank as a film plays.

What This Paper Is About

Films introduce their characters gradually, and viewers work out who is who from dialogue and cinematic cues rather than from a cast list. This paper asks whether a model can do the same: given a name mentioned in the dialogue and the surrounding video, decide which visible character bears that name, using nothing but the film itself (no IMDB cast lists, actor photographs, or pre-built character banks). The authors build a dataset and model for this, then extend it into a system that accumulates character identities, with names, as the movie unfolds.

Key Contributions

1. A new task: character naming from dialogue alone. Given a video clip, its character-level face tracks, and a candidate name from the dialogue, the model must predict the matching face track — or none when the named character is not on screen.

2. The Movie Character Naming (MCN) dataset. Built on MovieNet and Movie101, it contains 23,792 multiple-choice character-naming instances over 407 movies, plus first-appearance annotations for 20 movies (287 characters, 574 timestamps covering both first visual occurrence and first naming).

3. A character-naming model and a comparison against proprietary MLLMs. The authors fine-tune open-source Qwen3VL models with a two-stage recipe and benchmark them against Gemini-3-Flash-Preview and Gemini-3.1-Pro-Preview, including a contamination-controlled test on films released after the proprietary models' training cut-offs.

4. The first automatic method for progressively building a character bank for a feature-length film. The bank is updated shot by shot and evaluated with two new metrics (mean Temporal IoU and Principal Character Recall).

Main Findings

• Proprietary models lead, but a small fine-tuned model closes most of the gap. On the MCN test set, Gemini-3.1-Pro-Preview reaches 82.47 overall / 89.69 known accuracy. The fine-tuned Qwen3VL-4B-Thinking (SFT + RL) reaches 72.76 overall / 81.06 known accuracy, versus 53.60 / 58.83 for pretrained Qwen3VL-8B-Instruct and 61.57 / 67.69 for Qwen3.5-9B. The random baseline is 29.90 / 33.74.

• Two-stage fine-tuning beats naive fine-tuning. Naive fine-tuning of Qwen3VL-8B-Instruct gives 64.92 overall; the two-stage SFT + GRPO recipe on the smaller 4B-Thinking backbone gives 72.76, a reported +7.84% gain over the naive 8B baseline.

• Color-permutation augmentation is a substantial part of the gain. Without it, GRPO lets the model latch onto color-specific shortcuts (e.g. always mapping the protagonist to the most frequent color, "red"). Randomly permuting colors during RL forces identity-based reasoning.

• Accuracy survives on films the models could not have seen. On 102 manually annotated clips from four films released from March 2026 onward (Voicemails for Isabelle, In the Grey, The Death of Robin Hood, Heartstopper Forever) — 159 questions, 25 with none as the answer — the ranking is unchanged: our model gets 64.78 overall / 71.64 known, versus 80.50 / 78.36 for Gemini-3.1-Pro-Preview and 78.62 / 81.34 for Gemini-3-Flash-Preview. All methods drop, by a margin comparable to the random baseline's drop (18.42 / 20.93 here).

• Direct address dominates the test set. Roughly 70% of test questions involve direct address by other characters, 16% third-party reference, 12% self-introduction, and 2% require reasoning from textual cues such as name badges.

• The streaming bank strongly outperforms full-movie proprietary baselines. With mTIoU against first visual occurrence / first naming and PCR@5/10/15: ours 84.90 / 54.40 / 63.00 / 54.54 / 48.19; Gemini-3-Flash-Preview 51.81 / 30.92 / 35.00 / 32.04 / 29.84; Gemini-3.1-Pro-Preview 49.05 / 30.10 / 44.00 / 37.15 / 34.93 — even though the baselines are given full-movie access and ours runs in a single streaming pass.

• Failure modes are shared. Both Gemini and the authors' model struggle with subtle cues (e.g. brief pointing among multiple characters), complex scenes with motion, and over-predicting an on-screen character when the ground-truth character is absent. The remaining gap to Gemini is attributed partly to upstream errors in transcription and active-speaker detection, which Gemini may partly recover from via lip motion.

• The streaming pipeline degrades on crowded films. It performs well on Juno and Meet the Parents (few participants, limited recurring cast) but suffers on Harry Potter and the Goblet of Fire (background characters add clustering noise) and on Four Weddings and a Funeral, with over 50 named characters among hundreds of recurring faces.

• Naming can lag appearance by a long way. In Juno, the protagonist appears in the first shot and in almost every scene but is not named until 7.2 minutes into the film — motivating the separate first-appearance annotations.

Methodology in Plain English

Building the data. The authors start from MovieNet and Movie101, which already map character names to body bounding boxes. They align those boxes with face tracks in space and time to get name-to-face associations for all on-screen characters, keep only associations whose names are also spoken in the dialogue, and mark associations where the named person is off-screen (e.g. referred to in absentia) as none. Face tracks are clustered per character; voices are grouped by speaker diarization and matched to face tracks using TalkNet synchronization scores; dialogue is transcribed with Parakeet, producing subtitles already tagged with who is speaking. Test samples were manually inspected and corrected, partly by examining disagreements with strong proprietary models.

Asking an MLLM to name characters. Instead of feeding bounding-box coordinates, each face track is drawn in a unique color on the raw frames, and the text prompt refers to characters by color. Subtitles are embedded with timestamps and color tags, and the model is asked, e.g., "Which color corresponds to John Constantine?"

Fine-tuning in two stages. First, supervised fine-tuning on synthetic reasoning traces produced by a teacher MLLM that is given the correct answer — teaching the output format. Second, GRPO reinforcement learning with a composite reward: one term for response structure, one for correctness of the name-to-face assignments. Color permutation during RL prevents shortcut learning.

Streaming character bank. The film is processed shot by shot. The bank stores triplets of (face cluster, name, voice cluster), including unnamed entries, mirroring how a viewer remembers a face before learning the name. New tracks are matched to existing clusters by cosine distance to the cluster's face-embedding centroid; if that fails, voice embedding is used as a fallback under a tighter threshold; otherwise a new entry is created. Names are bound by majority vote over a histogram of names seen for each cluster, which handles aliases and nicknames (in Juno, the protagonist is addressed as Mama Bear, Home Skillet, or General MacGuff) and absorbs occasional naming errors. Names learned later are applied retrospectively to earlier unnamed clusters.

Error correction. Three mechanisms counter error accumulation along the chain: an MLLM filters noisy or ambiguous face tracks before clustering; an LLM filters out query names unlikely to refer to anyone on screen; and a greedy voting scheme resolves cross-entry name conflicts after each shot to enforce one-to-one naming.

Evaluation. Character naming is scored as overall accuracy (including none) and known accuracy (only queries with a valid face track). The bank is scored with mean Temporal IoU — the temporal overlap between the interval in which a human knows a character and the interval in which the model has them correctly in the bank, weighted by subtitle mention frequency — in a visual variant and a name variant, plus Principal Character Recall at K, which requires a single bank entry to match both the IMDB name (phonetically) and the IMDB reference face embedding.

Why This Matters

Impact on research. Existing audio-description and transcript systems typically bootstrap from IMDB cast lists, fan wikis, or actor photos, which are unavailable or incomplete for new films, or they avoid naming altogether with placeholders like "Person A". This paper shows the naming signal is present in the film itself and makes it a measurable task with a benchmark, an open fine-tuning recipe, and a streaming formulation that prior dialogue-based work (which optimizes over many mentions across episodes of a recurring TV cast) did not address.

Real-world applications:

• Automatic audio description and subtitling for new releases and archives where no cast metadata exists.
• Content navigation and search — chaptering, "scenes with this character" browsing, and recap generation for long films and series.
• Accessibility tools that announce who is speaking or on screen for blind and low-vision viewers.
• Media QC and cataloguing: automatically deriving cast lists, screen time, and character arcs for large film libraries.

Industry relevance. Streaming services, broadcasters, and post-production houses hold large catalogues with incomplete metadata; a method that derives character identities in a single pass over the video, without a curated cast list, is directly deployable to that setting. The paper also shows that a 4B open-source model with the right fine-tuning is competitive with frontier proprietary APIs on this task, which matters for cost and for processing content that cannot be sent to a third-party service.

Future Directions

• Closing the gap to proprietary models by fixing upstream noise. The authors attribute part of the remaining gap to errors in transcription and active-speaker detection, and hypothesize that Gemini partly recovers via lip motion; better lip-reading or active-speaker modeling is an open route.
• Handling subtle and crowded scenes. Both the proprietary models and the authors' model fail on brief pointing gestures, complex motion, and dense backgrounds — the same conditions that degrade the streaming bank in Harry Potter and the Goblet of Fire.
• Scaling identity disambiguation to large casts. Films with over 50 named characters among hundreds of recurring faces remain hard; longer-context memory and better one-to-many disambiguation are needed.
• Overcoming failure modes with agents or dedicated models. The authors explicitly propose developing models or an agentic framework to address the current failure modes, noting that neither task is fully solved even by proprietary models.

Target Audience

Researchers and engineers working on video understanding, multimodal LLMs, and character-centric video analysis; practitioners building audio description, subtitling, or media-cataloguing systems; and anyone interested in reinforcement-learning recipes for grounded visual reasoning. The paper is also a useful case study for readers evaluating how far open-source MLLMs are from proprietary frontier models on a well-defined multimodal task.

Authors’ abstract

Films introduce characters incrementally as they progress. We study the problem of identifying principal characters and assigning their names directly from the movie itself, without using external cast lists, actor photographs, or pre-built character banks. We introduce the character naming task: given a name mentioned in dialogue and the associated video context, the model must determine which visible character bears that name. This task is challenging because name resolution often depends on cinematic cues such as shot-reverse-shot structure, self-introduction, direct address, third-party reference, gaze, and temporal continuity. We provide a dataset and annotations for evaluating character naming and first character appearances, develop a video-and-dialogue model for the task, and compare it with proprietary multimodal large language models. We further introduce a progressive framework for building a character bank as the film unfolds. Together, these contributions support online, human-like movie understanding in which character identities are inferred from the film itself.