All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts

AI summary

Overview

Research area: Computer vision / optical character recognition — specifically multilingual scene text recognition (STR), the task of reading text from cluttered natural images across many writing systems.

Technical level: Advanced. The paper assumes familiarity with Transformer decoders, autoregressive vs. non-autoregressive decoding, CTC, and Mixture-of-Experts routing. The core ideas are explained here in plain language, but the underlying architecture is research-grade.

Scope in one sentence: The paper builds a large synthetic multilingual training set (TextMuSS-10M) plus a real evaluation benchmark (TextMuSS-Bench) and proposes ScriptMoE, a single recognizer that routes each image to script-specialized experts, aiming to be simpler than per-language expert systems, lighter than vision-language models, and more accurate than both.

What This Paper Is About

Multilingual scene text recognition is hard for two reasons: most languages have almost no large-scale real training data, and one dense model has to serve scripts with radically different structure (stacked consonants in Hindi and Tibetan, cursive ligatures and right-to-left order in Arabic, and Chinese/Japanese/Korean character sets that are tens to hundreds of times larger than the Latin alphabet). Existing fixes either deploy one recognizer per language (costly, and errors in a language-identification step propagate) or use massive vision-language models (expensive and often inaccurate on long-tail scripts). The authors want one recognizer that beats both approaches.

Key Contributions

1. TextMuSS-10M — a large-scale, balanced synthetic scene text dataset spanning 10 scripts and 229 languages, with 1M synthesized samples per script (10M total), built by adapting the UnionST synthesis engine.

2. TextMuSS-Bench — a real scene text benchmark covering all ten scripts with 10,899 images total, assembled by extending the MLT2019 test split with newly collected and expert-annotated Russian, Thai and Tibetan images.

3. ScriptMoE — a script-aware Mixture-of-Experts recognizer that shares a single visual encoder (SVTRv2) and replaces the decoder's dense feed-forward network with a sparse MoE block. An image-level router dispatches each image to Top-2 script-aligned experts, an always-on shared expert absorbs cross-script knowledge, and a lightweight script-classification head provides auxiliary supervision.

4. A systematic all-in-one multilingual STR evaluation under a unified training protocol, comparing ScriptMoE against 15 representative STR models, 9 general OCR systems, and multiple VLMs on both recognition and end-to-end OCR.

Main Findings

• Best average accuracy on TextMuSS-Bench: ScriptMoE reaches 82.06% average word accuracy, beating the strongest STR baseline (SVTRv2-AR, 80.75%) by 1.31%. The pure autoregressive no-MoE baseline sits at 80.75%.

• Gains concentrate on difficult low-resource scripts: Arabic improved by 2.98%, Thai by 2.40%, and Tibetan by 1.96% relative to the strongest baseline. The abstract reports roughly 18% improvement over PP-OCRv5 MLT and Qwen3.5-9B, and the results section notes generalist systems trail by 18–70% average points, with some collapsing outright on Arabic, Bangla and Tibetan.

• End-to-end OCR improves substantially: Keeping the PP-OCRv5 detector fixed and replacing only the recognizer with ScriptMoE lifts F1 on the CC-OCR multilingual task from 65.71% to 80.89% — a 15.18% absolute jump. This slightly surpasses the best zero-shot general VLM (Qwen3.5-9B at 80.73%), the best OCR-specialized VLM (Qianfan-OCR at 76.70%), and GoogleOCR (71.78%).

• Efficiency: ScriptMoE activates 41.13M of its 45.85M parameters per image, one to two orders of magnitude fewer than VLM-based systems. The paper reports accuracy versus activated parameters in Figure 4 but does not report inference latency or throughput numbers.

• Synthetic data is mandatory for low-resource scripts: Training on real data only collapses on Russian, Thai and Tibetan (0.00% each). Training on TextMuSS-10M alone outperforms real-only training on the MLT2019 average by 2.49%. Combining synthetic and real data raises the MLT2019 average by 8.40% over real-only training.

• Four experts is the best configuration: Ablation over expert count gives 0 experts = 80.75%, 2 experts = 81.52%, 4 experts = 82.06%, 10 experts = 81.32%. Scaling to one expert per script hurts because each expert sees too few samples.

• Top-2 routing beats alternatives: Top-1 reaches 81.60% and Top-4 reaches 81.94%, versus 82.06% for Top-2. Top-4 activates more parameters for negligible return and hurts Japanese and Korean.

• Both script-aware components matter: Removing the script-classification signal costs 0.43% on average. Removing the shared expert costs 0.71% on average, with the drop concentrated on Arabic (−3.62%) and Tibetan (−3.65%).

• A cross-script trade-off exists: Optimizing aggregate accuracy sacrifices per-script peaks. The Russian result specifically suffers from Cyrillic–Latin character confusion, compounded by Latin's dominance in real training data. Adding synthetic data lowers Latin, while adding real data lowers Russian.

• Routers learn script structure: Routing case studies show four representative lines from different script groups are each dominated by their expected expert, evidence of learned script-to-expert specialization.

Methodology in Plain English

The authors start from the observation that although the world has hundreds of languages, grouping them by writing system collapses the problem dramatically — PP-OCRv5 MLT, for example, covers 106 languages with about ten per-script models. They therefore consolidate languages into ten scripts: Latin, Cyrillic, Chinese, Japanese, Korean, Arabic, Hindi, Thai, Bangla and Tibetan. These ten scripts cover 229 languages.

Because real data exists almost only for English and Chinese, they synthesize 1M images per script. The synthesis draws text from four sources with a fixed mixing ratio: 40% real words from per-language lexicons, 20% character-shuffled words, 20% vocabulary-expansion strings that oversample rare characters, and 20% sentences from News Crawl newspaper corpora. Each of the 8k background images comes from the SynthText release, and each effect (curved layout, multi-directional/vertical layout, perspective distortion) is applied independently with 20% probability. Chinese, Japanese and Korean get a larger share of vertical-text synthesis (20%, versus 5% for other scripts), and Arabic is rendered right-to-left in logical order.

For the model, the enabling insight is that a scene text image almost always contains a single script. That means a mixture-of-experts router faces a nearly unambiguous decision. They take a strong visual encoder, feed its tokens into a Transformer decoder, and swap the dense feed-forward network for an MoE layer containing several routed experts plus one always-on shared expert. Crucially, the router input is computed once per image by mean-pooling the visual tokens, so every output token of an image follows the same route. This cuts routing cost, avoids token-level "ping-ponging" that destabilizes training, and makes each expert readable as a script specialist. Experts are further grouped by morphological similarity into four families: alphabet (Latin + Cyrillic), CJK, Arabic family, and Others (Hindi, Bangla, Tibetan, Thai).

A separate four-way classification head, sharing the router input but not the router weights, is trained to predict the script group from the ground-truth transcription's Unicode ranges. This gives a gentle nudge toward script alignment without forcing the router's hand. The total loss is the standard autoregressive cross-entropy plus this auxiliary term weighted at 0.1.

Why This Matters

Impact on research: The paper argues that script-level consolidation is a more practical unit than language-level modeling for multilingual OCR, and demonstrates that sparse expert routing can substitute for either per-language model proliferation or massive VLM scale. It also contributes two reusable artifacts — a 10M-image synthetic training resource and a 10-script real benchmark — that address a documented data scarcity in long-tail scripts.

Real-world applications:

• Document digitization and archival in multilingual regions where scanned material mixes scripts, such as government records, newspapers and historical archives.
• Deployed OCR pipelines where an existing detector (like PP-OCRv5) stays in place and only the recognizer is swapped, as the paper demonstrates end-to-end on CC-OCR.
• Edge and mobile deployment, since ScriptMoE activates 41.13M of 45.85M parameters per image compared with VLM systems the paper describes as having a hundred times more parameters.
• Multilingual user content processing, covering the ten scripts the system supports across 229 languages, including right-to-left Arabic and vertical CJK text.

Industry relevance: The direct comparison against PP-OCRv5 MLT, GoogleOCR and multiple commercial-class VLMs on a per-language end-to-end benchmark gives concrete evidence for practitioners deciding between expert pipelines and generalist models. The released code links (ScriptMoE and OpenOCR repositories) suggest intended practical adoption.

Future Directions

• Continual learning for new scripts: The authors explicitly plan to explore strategies that allow new scripts to be added without retraining the full model.
• Stronger detectors: They are interested in pairing ScriptMoE with better text detectors, since current end-to-end scores are capped by the upstream PP-OCRv5 detector, whose missed and false detections are most visible on Latin scripts under strict word-level evaluation.
• Resolving the Cyrillic–Latin confusion: The Russian result remains the weakest among the non-minority scripts (61.20% accuracy but a normalized edit distance of .72, notably lower than peers), traced to visually identical glyphs mapping to different characters. The paper states this gap remains difficult to close entirely because Cyrillic relies solely on synthetic substitutes.
• Handling bilingual and mixed-script images: The single-image-few-script prior assumes one script per image, with occasional bilingual mixes noted but not deeply addressed — a natural next case for the routing mechanism.

Target Audience

Researchers and engineers working on OCR, scene text recognition, or multilingual document understanding; practitioners choosing between per-language recognizers, expert cascade systems and vision-language models for deployment; and anyone interested in Mixture-of-Experts routing applied outside of large language models. Readers should be comfortable with Transformer architectures, autoregressive decoding, and dataset construction for vision tasks.

Authors’ abstract

Multilingual scene text recognition (STR) remains challenging due to the scarcity of training data for most languages and the difficulty of serving diverse scripts within a single model. Existing solutions either deploy one recognizer per language, inflating cost and introducing error accumulation, or rely on massive vision-language models (VLMs) that are expensive and still inaccurate on many scripts. In this work, we pursue an all-in-one multilingual recognizer that is simpler than per-language experts, lighter than VLMs, and more accurate than both. First, we construct TextMuSS-10M, a large-scale synthetic scene text dataset spanning 10 scripts and 229 languages. It provides balanced and sufficient supervision where real data is unavailable. Second, we propose ScriptMoE, a script-aware Mixture-of-Experts (MoE) architecture. It shares a single visual encoder and replaces the dense decoder with a sparse MoE block, which consists of an image-level router dispatches each image to the top-2 script-aligned experts and a shared expert absorbs cross-script knowledge. Extensive experiments on our assembled TextMuSS-Bench (10 scripts, 10,899 images) show that ScriptMoE achieves the highest accuracy of 82.06%, outperforming the strongest STR baseline by 1.31%. On the CC-OCR end-to-end multilingual task, replacing only the recognizer in PP-OCRv5 with ScriptMoE lifts F1 score from 65.71% to 80.89%, slightly surpassing the best VLM (80.73%) at a fraction of the parameter count.