Hermes the Polyglot: A Unified Framework to Enhance Expressiveness for Multimodal Interlingual Subtitling

AI summary

Overview

Research area: Natural language processing, specifically multimodal machine translation (MT) — applying large language models to interlingual subtitling, the task of translating the subtitles of films, TV series, and other visual media into a target language.

Technical level: Advanced. The paper assumes familiarity with speaker diarization, embedding-based clustering, knowledge distillation, and preference-optimization methods such as DPO and GRPO.

One-sentence scope: The paper presents Hermes, an LLM-based subtitling framework that fuses three components — Speaker Diarization, Terminology Identification, and a new Segment-wise Adaptive Preference Optimization (SAPO) method — to produce subtitle translations that are more contextually coherent, terminologically consistent, and expressive across six language directions.

What This Paper Is About

Interlingual subtitling is treated in this paper as an underexplored subfield of machine translation, distinct from general MT because subtitles are short, strongly context-dependent dialogue lines tied to both video and speech. The authors argue that today's LLMs still fail on four specifics: keeping meaning coherent across line boundaries, resolving pronouns correctly when languages differ in honorifics or pronoun omission, translating and consistently reusing specialized and proper-noun terminology, and producing translations that read naturally and vividly rather than literally. The goal is a single framework that supplies an LLM with visual and audio prior information plus a tailored preference-optimization method, so that subtitle translations match what audiences expect from localization.

Key Contributions

1. Task framing: The paper defines interlingual subtitling as a multimodal machine translation task and systematically lays out its practical challenges — semantic coherence across line segments, pronoun reference, terminology translation, and translation quality/expressiveness.
2. The Hermes framework: An LLM-based subtitling framework built on three modules — Speaker Diarization (combining visual and speech modalities), Terminology Identification (using LLM knowledge distillation), and Expressiveness Enhancement (the SAPO method with LLM-as-a-Judge).
3. An evaluation framework: A multidimensional LLM-as-a-Judge evaluation scheme for interlingual subtitling covering pronoun accuracy, terminology consistency, translation accuracy, naturalness, and vividness, plus a pairwise human win-rate evaluation.
4. A released dataset: Multilingual parallel subtitle corpora collected from the online video platform Youku, covering en⇒de, en⇒fr, en⇒zh, ko⇒zh, zh⇒en, and zh⇒th, with code and data available at the linked GitHub repository.

Main Findings

• Speaker diarization performance: On three manually annotated test sets (five Chinese programs, three challenging mainly-dialectal Chinese programs, and five English programs), Hermes' SD module reports lower DER and Text DER than VBx, spectral clustering, and the tri-modal E2CP method. For example, on the English set Hermes (SC) reaches DER 0.10272, JER 0.31331, and Text DER 0.09418, versus E2CP at 0.11951 / 0.39785 / 0.11098. Spectral clustering outperforms VBx throughout. On the Chinese-Hard set, Hermes (SC) reaches DER 0.10180 versus E2CP's 0.21125.
• Accuracy and naturalness beat human references, vividness lags: The paper reports that Hermes and frontier LLMs such as GPT-5 significantly outperform traditional translation models like MADLAD, reaching higher accuracy and naturalness than human references, while human translations excel in vividness because human translators incorporate video context and translate more liberally. In the reported tables, Hermes (π_st, SAPO) records the highest vividness scores across all directions.
• SAPO improves more than vividness: Compared with the supervised fine-tuned (SFT) backbone, SAPO-trained models show clear gains not only in vividness but also in accuracy and naturalness. In en⇒zh, the SFT model scores 86.5 accuracy / 82.1 naturalness / 59.2 vividness, while SAPO reaches 90.6 / 84.2 / 76.6.
• Strong gains on lower-resource directions: Hermes particularly excels across all dimensions in ko⇒zh and zh⇒th; in zh⇒th it records PA 85.6, TC 93.2, translation 91.9, naturalness 84.7, vividness 74.2, against a human reference of 76.6 / 75.1 / 66.3.
• Adaptive strategies matter, the gate most of all: Removing the gating function reduces en⇒zh vividness by 6.4 points, removing the importance score by 4.2, and removing the combined weight w(s_i) by 9.2. Effects are smaller in zh⇒th (down 2.9, 2.2, and 3.8 respectively).
• Terminology identification has high recall but poor translation on its own: In-context learning identifies terms with 96.9% recall on a test set of 20 annotated programs, but cannot produce suitable translations because only monolingual subtitles are available at inference time — motivating the knowledge-distillation training of the terminology LLM.
• Backbone and loss sensitivity: SAPO improves all tested backbones over SFT, with Qwen2.5-14B giving the strongest and most stable results; LLaMA-3.1-8B is weaker in en⇒zh and zh⇒en because of its weaker Chinese performance. Across preference losses, DPO gives consistently stable and superior results, while GRPO — which uses all sampled results rather than only preference pairs — achieves the best performance only in the en⇒zh direction (this section is truncated in the available text).
• Human evaluation agrees with the LLM judges: Pairwise win:tie:loss comparisons in en⇒zh and zh⇒th against the human gold reference, the SFT model, GPT-4o, and DeepSeek-R1 are reported as consistent with the LLM-as-a-Judge table, which the authors offer as validation of their automated evaluation framework.

Methodology in Plain English

Hermes works as a pipeline that feeds an LLM better information before it translates.

Step 1 — Speaker Diarization. Subtitle lines are dialogues between characters, so knowing who says what improves pronoun translation and helps group lines into coherent chunks. The system uses subtitle timing to cut matching video and audio segments per line. For video, TalkNet finds the active speaker, and face embeddings from CurricularFace are clustered; for audio, ERes2NetV2 produces timbre embeddings that are clustered separately. Because roughly 35% of lines in their test set are spoken while the speaker is off-screen, the visual path alone is incomplete, so visual clusters act as the anchor: audio clusters vote within each visual cluster to register a speaker and its average timbre prototype. Lines without a detected speaker are matched to the nearest prototype by cosine similarity, and a supplementary strategy uses adjacent-line similarity (threshold ε = 0.35) to group lines and an average score threshold (η = 0.4) to register previously missed speakers.

Step 2 — Terminology Identification. An off-the-shelf LLM (Qwen-Max) one-shot extracts terms, their types, and translations from bilingual line pairs. Because the same term can be typed or translated inconsistently, the raw candidates are filtered and voted on, then retrieved with a prefix tree to build a training set for a dedicated terminology LLM (Qwen2.5-14B) — a knowledge-distillation-like setup.

Step 3 — Expressiveness Enhancement. A subtitle translation model is first trained by supervised fine-tuning, with speaker labels and inferred speaker age/gender (from Qwen2.5-VL-7B-Instruct) added to the prompt and identified terminology inserted via the prefix tree. Because translating a line depends on its context, the authors define the problem as local preference optimization — rewards apply to segments of a response, not to the whole response. Their SAPO method holds out 20% of the training corpus, samples k = 15 candidate translations per line using previously chosen lines as prefix, adds the human reference when available, scores candidates with an LLM evaluator (Qwen2.5-14B-Instruct, with DeepSeek-V3.1 as reward model), and keeps the top-scoring line as the prefix for the next segment. Each line receives an adaptive weight combining a gate (turned off when candidates are too few or too similar, e.g. |T_i| ≤ 3 or max(E_i) − min(E_i) ≤ 5) and an importance score proportional to candidate diversity, and the weighted loss is compatible with DPO, SimPO, or GRPO.

Why This Matters

The paper positions interlingual subtitling as essential to entertainment localization and argues it has been largely untreated in machine translation research despite growing demand from streaming and online video platforms. It also contributes a released parallel subtitle corpus and a reusable evaluation design, and shows that a 14B open model trained with domain-specific signals can beat much larger frontier LLMs on domain-specific measures such as pronoun accuracy and terminology consistency.

Real-world applications:

• Streaming and online video localization — translating library content for Netflix- and Disney+-style platforms and for platforms such as Youku, where the data was collected.
• Subtitle consistency at scale — keeping character names, places, and invented terms translated identically across an entire series season.
• Multilingual content discoverability — the paper notes subtitles make multimedia content more searchable, comprehensible, and shareable in a global web environment.
• Speaker-aware dubbing and accessibility — the diarization output (who speaks which line, plus inferred age and gender) supports downstream voice work and accurate pronoun rendering.

Industry relevance is direct: the workflow targets professional localization pipelines where terminology consistency and natural-sounding, vivid phrasing are commercial quality criteria, and the authors include collaborators from media and entertainment groups.

Future Directions

• Closing the vividness gap with human translators. The paper attributes human vividness advantages to translators watching the video and translating more liberally; how to transfer that deeper video grounding into the model is left open.
• Extending and strengthening low-resource and dialectal coverage. The Chinese-Hard set (mainly dialectal shows) remains the most difficult diarization condition, and directions such as ko⇒zh and zh⇒th are where Hermes' advantages are largest — an implicit invitation to test more language pairs.
• Broadening the preference-loss comparison. The available text is truncated mid-sentence while reporting that GRPO wins only on en⇒zh, leaving the full comparison across DPO, SimPO, and GRPO unresolved in this excerpt.
• Reducing dependence on proprietary judge and reward models. SAPO relies on Qwen2.5-14B-Instruct as evaluator and DeepSeek-V3.1 as reward model, and the evaluation framework relies on DeepSeek-V3.1, Claude Sonnet 4, and GPT-5; whether smaller or open judges suffice is not reported.

Target Audience

Machine translation and multimodal NLP researchers, especially those working on audiovisual translation, speech-and-vision fusion for language tasks, or preference optimization beyond whole-response alignment. It is also relevant to localization engineers and streaming-platform technologists evaluating LLM-based subtitle pipelines, and to NLP practitioners interested in how distilled domain knowledge and LLM-as-a-Judge evaluation can be combined for a specialized translation domain.

Authors’ abstract

Interlingual subtitling, which translates subtitles of visual media into a target language, is essential for entertainment localization but has not yet been explored in machine translation. Although Large Language Models (LLMs) have significantly advanced the general capabilities of machine translation, the distinctive characteristics of subtitle texts pose persistent challenges in interlingual subtitling, particularly regarding semantic coherence, pronoun and terminology translation, and translation expressiveness. To address these issues, we present Hermes, an LLM-based automated subtitling framework. Hermes integrates three modules: Speaker Diarization, Terminology Identification, and Expressiveness Enhancement, which effectively tackle the above challenges. Experiments demonstrate that Hermes achieves state-of-the-art diarization performance and generates expressive, contextually coherent translations, thereby advancing research in interlingual subtitling.