Towards Authentic Movie Dubbing with Retrieve-Augmented Director-Actor Interaction Learning

AI summary

Overview

Research area: Natural Language Processing / speech synthesis — specifically automatic movie dubbing, also called Visual Voice Cloning (V2C), with a focus on emotional expressiveness.

Technical level: Advanced. The method combines retrieval-augmented generation, graph attention networks, large language model (LLM) based multimodal captioning, and neural speech synthesis.

Scope: The paper proposes a movie dubbing framework, Authentic-Dubber, that simulates the real-world director–actor preparation workflow to improve the emotional expressiveness of generated dubbing, and evaluates it on the V2C-Animation benchmark.

What This Paper Is About

Automatic movie dubbing models generate speech from a script, copy a speaker's timbre from a short timbre prompt, and keep the speech lip-synced to silent video. Existing models assume a simplified workflow in which actors dub immediately with no preparation, ignoring the director–actor interaction that happens in real dubbing sessions, where directors give actors emotional reference footage to study first. Authentic-Dubber tries to reproduce that preparation stage — building a library of reference footage, retrieving emotionally similar clips for each target utterance, and progressively feeding that emotional knowledge into speech generation.

Key Contributions

1. First dubbing model built around the authentic workflow. The authors propose Authentic-Dubber, described as the first movie dubbing model that simulates the real director–actor interaction process in order to enhance emotional expressiveness.
2. A Multimodal Reference Footage Library (MRFL). Built from the V2C dataset, it stores four emotion extractors per sample: indirect multimodal emotional information (scene, face, text) and matched direct emotional audio, with LLMs used to produce emotion captions for deep comprehension of multimodal emotional signals.
3. Emotion-Similarity-based Retrieval-Augmentation (ESRG). The target utterance's basic emotion (scene, face, text) is used as separate queries to retrieve the most relevant indirect multimodal emotion information and its matched direct emotional audio from the MRFL, using a speaker-agnostic retrieval strategy.
4. Progressive Graph-based Speech Generation (PGSG). A progressive construct-and-encode paradigm over a Basic Emotion Graph, an Indirect Emotion Extended Graph, and a Direct Emotion Extended Graph, with hierarchical aggregation in an Emotion Knowledge-based Speech Synthesizer.

Main Findings

• Best emotion-related scores among compared models. Authentic-Dubber reaches EMO-ACC of 47.21%, MOS-DE of 3.792 ± 0.055, and MOS-SE of 3.889 ± 0.053, the best of any system in Table 1 on those emotion metrics (ground truth is 99.96, 4.416 ± 0.035, and 4.497 ± 0.044 respectively).
• Comparison points on the other metrics. Authentic-Dubber records WER 25.95, SECS 84.40, and MCD-DTW-SL 9.68. Among baselines, Speaker2Dubber has the lowest WER (18.27), StyleDubber has the lowest MCD-DTW-SL (9.40), and StyleDubber has the highest baseline SECS (83.46). The other baselines are FastSpeech2 (EMO-ACC 42.39, WER 33.30, SECS 25.47, MCD-DTW-SL 14.72, MOS-DE 3.058, MOS-SE 3.063), V2C-Net (43.07, 67.98, 40.65, 19.16, 3.146, 3.149), HPMDubbing (43.94, 135.72, 34.11, 12.64, 3.362, 3.320), and Speaker2Dubber (44.55, 18.27, 81.26, 9.82, 3.432, 3.461).
• Caption-based emotion extraction beats embedding-based alternatives. Replacing the scene caption with I3D embeddings drops EMO-ACC to 46.34; replacing the face caption with EmoFan embeddings gives 46.52; removing both gives 46.02, versus 47.21 for the full model.
• Every retrieval modality contributes. Removing scene retrieval yields EMO-ACC 46.27, face retrieval 46.64, text retrieval 45.99, and all retrieval 45.23. The largest drop occurs when all retrieval is removed.
• The progressive graph components matter. Removing indirect information gives EMO-ACC 45.95; removing direct audio gives 45.30; removing graph-based modeling gives 45.92; removing construct-and-encode gives 46.85; removing hierarchical aggregation gives 46.71. Corresponding MOS-DE and MOS-SE scores fall in each case.
• Speaker-agnostic retrieval wins, and more retrieval is not always better. Speaker-agnostic retrieval achieves the highest EMO-ACC (47.21%) at K = 3 and consistently outperforms Speaker-Specific retrieval; increasing K beyond that point degrades performance under both settings.
• More reference footage helps, up to a point. EMO-ACC rises steadily as the retrieval footage scale grows from 10% to 100%, plateauing between 80% and 100%, with a peak of 47.21%.
• Cosine similarity is the best retrieval metric. Across different Top-K settings, cosine similarity consistently gives the best overall EMO-ACC; dot product fluctuates more, and Euclidean distance is relatively stable but has a slightly lower performance ceiling.
• Qualitative spectrogram evidence. Visualizations of angry and happy speech show that the model more accurately captures high fluctuations in angry speech and produces more natural prosodic variation in happy speech than the baselines.

Methodology in Plain English

The researchers start from an analogy: in real films, a director gives actors emotional reference footage before a dubbing session, and the actors study it until they have internalized the emotional context.

To simulate the director's material, they build a library of reference footage from the V2C dataset. For each clip, four extractors pull out emotional signals: a scene emotion vector (using an LLM video-understanding model to caption the scene's emotional atmosphere, aided by low-level visual cues like hue, lightness, and saturation), a face emotion vector (from a caption of facial emotion changes), a text emotion vector (combining a "self" emotion from the script with a "react" emotion generated by a commonsense model), and an audio emotion vector from a universal emotion representation model. The captions are converted into emotion vectors by a RoBERTa-based emotion recognition model.

To simulate the actor studying that footage, the target utterance's basic emotions act as queries. Scene, face, and text emotion vectors are each compared against the library by cosine similarity, and the Top-K matches are retrieved together with their matched emotional audio, found by index lookup. This retrieval is speaker-agnostic, which matters because animated characters are virtual and have limited speaker-specific footage.

To simulate the final performance, the model builds three graphs in sequence. The Basic Emotion Graph connects the target's own scene, face, and text emotion nodes pairwise. The Indirect Emotion Extended Graph adds the retrieved indirect emotion nodes, attached to the basic nodes of the same modality. The Direct Emotion Extended Graph adds the matched emotional audio nodes, attached to the basic source that issued the query. Each graph is encoded with a Graph Attention Encoder, and each stage is initialized from the previous one. Finally, a synthesizer aggregates the learned emotional knowledge via cross-attention and Conv1D layers on top of the cross-modal alignment output, and a Mel decoder plus vocoder produces the final waveform.

Training details: frames sampled at 25 fps, audio resampled to 22.05 kHz, STFT window 1024, frame size 1024, hop length 256, all emotional features projected to 256 dimensions, Graph Attention Encoder output dimension 256, Conv1D output dimension 256, Adam optimizer with β1 = 0.9, β2 = 0.98, ε = 10⁻⁹, learning rate 0.00625, on an A800 GPU.

Evaluation uses V2C-Animation (26 Disney animated movies, 153 characters, 10,217 video clips, split 60% training / 10% validation / 30% testing), which the authors note is currently the only publicly available movie dubbing dataset with emotion annotations. Objective metrics are EMO-ACC, WER, SECS, and MCD-DTW-SL; subjective metrics come from a MOS test with 20 trained raters scoring 12 generated dubbed videos and speech samples on a 1-to-5 scale.

Why This Matters

The work reframes dubbing as a preparation-driven process rather than a one-shot cross-modal mapping, and shows measurable gains in emotion-related metrics. It also shows that LLM-generated emotion captions are more useful than raw emotion embeddings for this task, and that retrieved emotional audio adds value beyond what the target utterance alone provides.

Real-world applications:

• Animated film and streaming localization, where a single script must be re-voiced in many languages while preserving the original emotional performance.
• Commercial AI-generated content for movie production, which the paper explicitly cites as a motivation for V2C.
• Voice-over and character dialogue for games, virtual avatars, and animated shorts built from silent video.
• Post-production dubbing workflows where directors want to specify or verify the target emotional tone of a performance.

Industry relevance: the pipeline is modular — a reference footage library, a retrieval stage, and a graph-based synthesis stage — so studios could swap in their own footage archives or emotion extractors. The reliance on a speaker-agnostic retrieval strategy is a practical design choice for animation, where a character's available footage is scarce. The paper also states that code and demos are publicly available.

Future Directions

• Extending the framework to control additional expressive attributes beyond emotion, specifically timbre, speaking rate, and prosody, as stated in the future work section.
• Investigating why retrieval performance degrades past a certain Top-K, since excessive retrieval introduces redundant information under both speaker-agnostic and speaker-specific settings.
• Exploring retrieval similarity functions further, given that the model is sensitive to the choice of similarity metric and that dot product and Euclidean distance behaved differently from cosine similarity.
• Testing whether the approach transfers beyond animated film, since the paper notes that speaker-specific reference footage is limited for virtual characters but does not report evaluation on live-action or non-V2C data.

Target Audience

Researchers and graduate students working on speech synthesis, multimodal generation, retrieval-augmented generation, and graph neural networks; practitioners building dubbing or voice-cloning systems for film, animation, and games; and readers interested in how real production workflows can be encoded as learning schemes. Some familiarity with speech synthesis pipelines and graph attention networks helps, but the high-level framing of the paper is accessible without it.

Authors’ abstract

The automatic movie dubbing model generates vivid speech from given scripts, replicating a speaker's timbre from a brief timbre prompt while ensuring lip-sync with the silent video. Existing approaches simulate a simplified workflow where actors dub directly without preparation, overlooking the critical director-actor interaction. In contrast, authentic workflows involve a dynamic collaboration: directors actively engage with actors, guiding them to internalize the context cues, specifically emotion, before performance. To address this issue, we propose a new Retrieve-Augmented Director-Actor Interaction Learning scheme to achieve authentic movie dubbing, termed Authentic-Dubber, which contains three novel mechanisms: (1) We construct a multimodal Reference Footage library to simulate the learning footage provided by directors. Note that we integrate Large Language Models (LLMs) to achieve deep comprehension of emotional representations across multimodal signals. (2) To emulate how actors efficiently and comprehensively internalize director-provided footage during dubbing, we propose an Emotion-Similarity-based Retrieval-Augmentation strategy. This strategy retrieves the most relevant multimodal information that aligns with the target silent video. (3) We develop a Progressive Graph-based speech generation approach that incrementally incorporates the retrieved multimodal emotional knowledge, thereby simulating the actor's final dubbing process. The above mechanisms enable the Authentic-Dubber to faithfully replicate the authentic dubbing workflow, achieving comprehensive improvements in emotional expressiveness. Both subjective and objective evaluations on the V2C Animation benchmark dataset validate the effectiveness. The code and demos are available at https://github.com/AI-S2-Lab/Authentic-Dubber.