Breaking Babel: A Self-Evolving Multi-Agent System for Long-Form Subtitle Translation

AI summary

Overview

Research area: Natural Language Processing — long-form machine translation, specifically subtitle translation, using LLM-based multi-agent systems that adapt at test time.

Technical level: Advanced. Comfort with LLM agent architectures, prompt optimization, MQM-style evaluation metrics, and MT benchmarking conventions is needed to follow the methodology.

Scope: The paper proposes SMART, a self-evolving multi-agent system for translating entire TV series with persistent series-level memory and test-time prompt/routing adaptation, and introduces the Subtitle Arena benchmark (192 series, 70,664 aligned bilingual episode pairs, 15 locales) plus the SubMQM evaluation rubric (7 dimensions, 19 error types).

What This Paper Is About

Translating subtitles for a full TV series is harder than translating isolated sentences: hundreds of sentences per episode depend on discourse and cultural context that spans scenes and episodes, while terminology, character names, and style must stay consistent across the whole series and still satisfy display constraints like line length and reading speed. Existing single-LLM approaches work sentence-by-sentence without persistent context, and existing multi-agent approaches use static workflows that cannot adapt to how difficult a given scene is. SMART addresses both by evolving its agent prompts and routing policy during a test-time training stage on a portion of each series, then freezing that evolved configuration for the rest.

Key Contributions

1. SMART, a self-evolving multi-agent framework for long-form subtitle translation. It has a test-time training stage that jointly adapts agent prompts and routing policies while maintaining persistent series-level memory and invoking contextual/verification tools, followed by a test-time inference stage that translates the remaining content with the evolved configuration frozen (no LLM parameter updates).

2. Subtitle Arena, a long-form subtitle translation benchmark spanning 14 genres, 2–198 episodes per series, production years from 1959 to 2023, and 15 target locales, with explicit support for evaluating cross-episode consistency and contextual adaptation. It contains 192 television series, 6,267 English source episodes, and 70,664 aligned bilingual episode pairs.

3. SubMQM, a subtitle-adapted automatic MQM (Multidimensional Quality Metrics) framework covering seven dimensions and 19 fine-grained error categories for analyzing semantic, linguistic, contextual, and subtitle-specific technical errors.

4. Extensive evaluation showing SMART achieves the best Overall MQM score in all 15 Subtitle Arena directions with a 6.9% lower average penalty than the strongest agent baseline, the best model result across all 4 MuSC language pairs, and the best human evaluation result at 4.50/5.

Main Findings

• Best on Subtitle Arena in every direction: SMART obtains the lowest Overall SubMQM penalty in all 15 English→locale directions and in all 15 locale→English directions. For English→locale it reduces the mean Overall penalty from 1.20 (TransAgent) to 1.11, a 6.9% relative reduction. For locale→English it reduces the mean from 0.41 to 0.37, a 10.0% relative reduction.

• Beats stronger single-call models despite a weaker default backbone: SMART's default backbone is Claude Sonnet 4.6, yet it outperformed single-call GPT-5.5 (mean 1.44) and Claude Opus 4.8 (mean 1.55) on average for English→locale.

• Gains are spread across dimensions, not one error type: Relative to TransAgent, SMART lowered the mean Accuracy penalty from 1.83 to 1.67 and the mean Technical penalty from 1.83 to 1.72, while also reducing Terminology, Fluency, Linguistic Conventions, Locale Conventions, and Audience Appropriateness.

• Transfers to the public MuSC benchmark: SMART achieved the best model result on all Accuracy, Naturalness, and Vividness measurements across all 4 directions (en→zh, ko→zh, zh→en, zh→th). Its scores were en→zh 94.3/88.2/80.4, ko→zh 86.8/85.9/78.4, zh→en 92.3/88.6/83.7, and zh→th 94.2/87.0/80.5. Against the strongest non-SMART result in each column, it improved by roughly 3.0 points on average, with gains from 1.2 to 7.9 points. Largest gains were in Vividness: 3.8, 7.9, 2.0, and 6.3 points respectively.

• Gains are not solely from a stronger base model: With Gemma 3 4B as backbone, en→zh Overall penalty dropped from 1.46 to 0.67; with DeepSeek-V3.2 it dropped from 0.93 to 0.53, with similar reductions in the reverse direction and other pairs.

• Evaluation is robust to judge choice: Swapping the default Claude Sonnet 4.6 judge for GPT-5.5 while holding translations fixed changed Overall penalties only marginally, e.g., 0.48 versus 0.47 on en→zh and 0.87 versus 0.87 on en→de.

• Human evaluation: SMART achieved the best result with an overall score of 4.50/5.

• Multimodal expansion: Adding video and audio tools powered by Qwen3-Omni reduced the average SubMQM penalty from 0.63 to 0.57 across six representative directions and outperformed the multimodal subtitle systems ViDove and Hermes.

• Temporal generalization: On 200 TV series released in 2025–2026, including titles beyond the reported knowledge cutoff of Claude Sonnet 4.6, SMART outperformed TransAgent across all six evaluated directions.

• Ablation results are incomplete in the supplied content: The paper's component ablation table (Table 6) is described across six directions using Overall SubMQM penalties, but the text is truncated mid-caption, so specific ablation numbers are not reported here.

Methodology in Plain English

SMART treats a TV series as a long-running translation job rather than a list of independent sentences.

Persistent series memory. The system keeps a series-level memory organized as four parts: a terminology pool, character profiles, domain/scene knowledge, and an idiom bank. Before translating an episode it loads everything accumulated from earlier episodes; after translating, newly confirmed information is merged back. This gives the system long-range state that exceeds any single model call's context window.

Dynamic routing and a pool of specialists. Instead of sending every sentence through the same pipeline, a learnable router inspects properties such as sentence length, scene tone, and lexical cues like slang or profanity, then activates a subset of specialized translator roles. The role pool emphasizes different priorities: semantic faithfulness, naturalness, expressiveness, subtitle-length control, and colloquial rendering.

Mixture-of-agents generation with tools. Each active translator produces a candidate translation and can call tools for surrounding context, confirmed terminology, similar previously translated sentences, subtitle constraints, domain knowledge, idiom lookup, and target-language fluency checks.

Judge–refiner loop. A judge scores each candidate on semantic accuracy, fluency, style and register, long-range consistency, and subtitle display constraints, and also emits a textual critique. The highest-scoring candidate is refined using that critique, and the accepted translation is written back to memory. After each episode, an episode-level pass revisits subtitles to repair residual consistency errors.

Self-evolution at test time training. After each training batch, an evaluator aggregates candidate scores and critiques into a structured feedback signal. This feedback updates translator prompts and the routing policy — both expressed in natural language, and both constrained rewrites rather than free-form edits: prompt revisions preserve an agent's role and workflow, and routing rewrites operate under a floor of two agents per category. Updates happen once per training epoch from batch-aggregated feedback, and the underlying LLM parameters are never modified.

Training/inference split. Episodes of each series are split 3:7 into test-time training and held-out inference data (the implementation section also describes using the first 30% of available content for training and the remaining 70% for held-out inference). Prompts and routing are frozen for inference, while memory continues to accumulate because it represents task state. The episode-level consistency check operates over overlapping windows of size 5, 10, and 20 with 50% overlap.

Evaluation. Subtitle Arena was built from OpenSubtitles2024 by recovering bilingual correspondences and associating subtitle files using IMDb identifiers, season indices, and episode indices, preserving original timecodes, normalizing formatting, removing empty/one-sided/unparsable alignments, and grouping aligned episodes by series. Hypotheses are scored with SubMQM: an LLM evaluator assigns error penalties of 0, 5, or 10 (no, minor, severe) across seven dimensions that decompose into 19 error types, with Overall weighting semantic fidelity more heavily. All SubMQM results are reported as penalties, where lower is better, using the same evaluator and rubric for every system.

Why This Matters

Impact on research. The paper reframes subtitle translation as a stateful, self-improving long-form task rather than a sentence-level MT problem. It shows that adapting prompts and routing policies at test time — without touching model weights — can beat both stronger single-call LLMs and fixed-workflow translation agents. It also supplies two reusable artifacts: a series-level benchmark with long-range consistency evaluation, and SubMQM, a subtitle-specific error taxonomy. The comparison against strong reasoning models and a fine-tuned ALPO baseline frames self-evolution as a complement to, not a substitute for, model capability.

Real-world applications:

• Streaming platform localization, where a single series must be translated consistently into many locales across an entire season or run.
• Dubbing and audio-description workflows, which need the same terminology and character-voice consistency and can leverage the multimodal extension.
• Catalog and archival localization, since Subtitle Arena covers production years from 1959 to 2023 and the system generalizes to 2025–2026 series.
• Subtitle quality assurance tooling, where SubMQM's seven dimensions and 19 error types can be used as an automatic checking layer rather than only a research metric.

Industry relevance. The finding that routing, persistent memory, and tool calling produce gains on top of off-the-shelf backbones means improvements can be realized without retraining or fine-tuning large models, which matters for cost and deployment timelines. The 15-locale coverage (including zh-CN, pt-BR, es-ES, es-MX, fr-FR, ro-RO, tr-TR, pt-PT, de-DE, it-IT, nl-NL, sv-SE, da-DK, no-NO, ko-KR) matches the locale mixes that global distribution platforms actually ship, and the technical dimension of SubMQM targets line breaking, characters per line, and lines per box — constraints that production subtitle pipelines enforce.

Future Directions

• Broaden the multimodal signal. The current multimodal expansion uses Qwen3-Omni video and audio tools over six directions; scaling this across all 15 locales and quantifying how much visual grounding contributes to long-range consistency remains open.
• Reduce dependence on a strong judge. The judge–refiner loop drives all prompt and routing updates, and the paper only tests swapping Claude Sonnet 4.6 for GPT-5.5. Whether the scaffold can evolve with weaker or cheaper judges is not established.
• Extend the test-time adaptation budget and analysis. The paper reports the training/inference split and the 30%/70% implementation split but leaves open how much test-time training data a series needs before returns diminish, and the component ablation is the natural place to answer that.
• Scale to longer horizons and more series. Subtitle Arena covers 2–198 episodes per series; whether memory accumulation stays beneficial over much longer runs, or saturates, is not addressed by the reported results.

Target Audience

Researchers working on machine translation, LLM-based multi-agent systems, and self-evolving or test-time-adapting agents; practitioners building subtitle and media localization pipelines who need series-level consistency and locale coverage; and benchmark or evaluation researchers interested in MQM-style, multidimensional error analysis adapted to a specific production domain. Readers without a background in agent architectures or MT evaluation will find the methodology sections dense, but the problem framing and the benchmark description are accessible.

Authors’ abstract

Long-form subtitle translation requires reasoning over discourse and cultural context spanning episodes or entire series, while maintaining consistent terminology and style. Existing single-LLM methods are largely sentence-level, and multi-agent systems often use static workflows that do not adapt to scene complexity or production context. We propose SMART, a Self-evolving Multi-Agent system for long-foRm subtitle Translation. During test-time training, SMART builds persistent series-level memory and translates a subset of sentences through a dynamic router and Mixture-of-Agents layer with tools for terminology verification, subtitle constraint validation, and contextual retrieval. A judge-refiner loop scores candidates and uses textual critiques to update agent prompts and routing policies without retraining the underlying LLMs. During test-time inference, the evolved configuration translates the remaining series. We also introduce Subtitle Arena, covering 14 genres, 2--198 episodes per series, production years 1959--2023, and 15 target locales, together with SubMQM, a subtitle-adapted MQM framework with seven dimensions and 19 error categories. SMART achieves the best overall MQM score in all 15 Subtitle Arena directions, reducing average penalty by 6.9% over the strongest competing agent system. On the public MuSC benchmark, SMART obtains the best model result across all four language pairs and also achieves the best human-evaluation result, with an overall score of 4.50/5.