What Is Wrong with Synthetic Data for Scene Text Recognition? A Strong Synthetic Engine with Diverse Simulations and Self-Evolution

AI summary

Overview

Research area: Scene Text Recognition (STR) — specifically the generation of synthetic training data and semi-supervised (self-evolution) training to reduce real-annotation cost.

Technical level: Intermediate. The paper is data-centric rather than architecture-heavy, but it assumes familiarity with STR benchmarks, CTC vs. autoregressive decoding, pseudo-labeling, and standard metrics (word accuracy, NED, FID).

Scope: The paper diagnoses why rendering-based synthetic text data underperforms real data, proposes a stronger rendering engine (UnionST) plus two datasets (UnionST-S, UnionST-P), and shows that a self-evolution learning framework can match or exceed fully supervised real-data training while cutting manual annotation by 91%.

What This Paper Is About

Scene Text Recognition models need large, category-balanced, perfectly labeled text images, which are hard to collect from the real world. Rendering-based synthetic data is cheap and has exact labels, but models trained on it lag far behind models trained on much smaller real datasets — a domain gap. This paper asks whether that gap is caused by the synthetic engines themselves, and whether fixing their limited diversity in corpus, font, and layout can close it.

Key Contributions

1. A diagnostic comparison of the two synthetic paradigms. The authors evaluate rendering-based engines against learning-based (text-editing and diffusion) generators and argue that rendering remains preferable because it guarantees absolute label correctness and is far cheaper — relying on CPUs at roughly 1/20 the cost of diffusion-based TextSSR and 1/10,000 the cost of the closed-source Nano Banana.

2. UnionST, a rendering-based synthetic engine with diverse simulations. It enriches the corpus (Common, Contextless, Incomplete, Multi-Words), expands the font set, and models per-character position, orientation, and size so curved, multi-oriented, multi-sized, and salient text can be rendered. Its corpus contains 113.8K entries versus 1.4K for MJ, 1.2K for ST, 3.6K for SynthTIGER, and 3.1K for UnrealText.

3. Two released datasets. UnionST-S (5M samples) built from the curated corpus, and UnionST-P (5M samples) built by using pseudo-labels from unlabeled real data as a new corpus; combining them yields UnionST-SP (10M samples).

4. A self-evolution learning (SEL) framework with Iterative Self-Refinement (ISR). Pseudo-labels high-confidence unlabeled real data, iteratively fine-tunes on it, and reserves manual annotation for the low-confidence hard cases. It reports a new SOTA of 91.39% average accuracy on Union14M-Benchmark and a 91% reduction in annotation requirement.

Main Findings

• Generative synthesis is unreliable for training data. On the ScenePair benchmark, the best-performing editing model, TextCtrl, reaches only 84.67% accuracy (NED 0.936, FID 43.78), while Flux.1 Kontext reaches 12.81% accuracy (NED 0.460, FID 41.59), AnyText 51.12%, Qwen-Image-Edit 53.16%, TextSSR 79.69%, and MOSTEL 37.69%.

• Diversity, not volume, is the bottleneck. Evaluating mainstream rendering-based synthetic datasets totaling 36M samples shows they are dominated by single, semantically rich short words in conventional fonts with simple horizontal layouts, causing poor performance on multi-word, contextless, artistic, curved, and multi-oriented text.

• Fewer but harder samples win. UnionST-S with only 5M samples outperforms existing 2D-based synthetic datasets (up to 36M samples) by 0.42% on Common benchmarks and by 9.64% on the more challenging Union14M-Benchmark.

• Large gains on the hardest subsets. UnionST-S improves over ST-2D by 5.64% on the Curve subset and 21.26% on the Multi-Oriented subset, approaching real-data performance levels.

• Model choice matters. Replacing SVTRv2's CTC decoder with an attention-based autoregressive decoder (yielding SVTRv2-AR) gives an average accuracy of 83.00% on Union14M-Benchmark using only synthetic training data.

• Pseudo-corpus helps, more data does not. At equal scale, UnionST-P improves average accuracy by 1.30% over UnionST-S on Union14M-Benchmark, whereas scaling UnionST-S from 5M to 10M samples produces only marginal improvement (83.29% versus 83.00%).

• New state of the art. Pre-training on UnionST-SP and fine-tuning on real data (UnionST-SP + R, 10.0M + 3.22M) reaches 91.39% average accuracy on Union14M-Benchmark, which the authors state is the first time STR performance has exceeded 90% on this benchmark. Pre-training with UnionST-SP adds 4.17% average accuracy on U14M-Bench, 1.68% more than the gain from ST-2D.

• Minimal annotation works. Fine-tuning the UnionST-S-trained model with only 1% of real data (32.2K samples) yields 87.26% on U14M-Bench, comparable to the 87.22% obtained from the full real U14M-Filter dataset (3.22M).

• Self-evolution approaches full supervision. After the first ISR iteration accuracy reaches 89.12% on U14M-Benchmark and after the second 89.81%, outperforming models trained on the full real dataset by 2.59%. A third iteration produces no further gain because added pseudo-label errors offset the increase in labeled samples.

• Hard-case annotation closes the gap. Adding 290K manually annotated hard samples on top of the second iteration reaches 91.23%, only 0.16% below the fully supervised SOTA, while reducing annotation from 3.22M to 290K (a 91% reduction).

• Choosing what to pseudo-label matters. With the same threshold (0.9) and real data, using MJ+ST as the ISR base reaches 73.30%, ST-2D reaches 80.51%, UnionST-S reaches 88.09%, and UnionST-SP reaches 89.12% — showing the pseudo-corpus augmentation is indispensable.

• The confidence threshold has an optimum near 0.9. Fine-tuning with descending confidence thresholds (Top-1K, Top-10K, Top-100K, Top-1M, Top-2.35M at threshold 0.9, and Top-3.22M at threshold 0) improves performance initially and then declines, whereas human-annotated data keeps improving with volume.

• Ablations confirm each component. Curved placement strongly affects the Curve subset; multi-oriented variation affects Curve, Multi-Oriented, and Salient; corpus augmentation primarily benefits Contextless and Multi-Words but can slightly lower Common benchmark scores; DTAug notably improves the General subset; font diversity has little effect at small data scale but reducing font variety at 5M scale markedly drops Artistic performance.

Methodology in Plain English

The authors first take apart existing rendering-based synthetic data engines. Each such engine selects text and a font, renders the text, and pastes it onto a background — so the authors improve each step in turn.

For the corpus, they move beyond isolated semantic words. They add Contextless text (random character strings, length 2–25, 10K per length, 240K total, drawn from 94 characters), Incomplete text (words with an initial, terminal, or internal character randomly removed, 264K samples), and Multi-Words text (400K common phrases plus substrings of varying lengths, 1–25, 120K per length, 3M total, from the ST newspaper corpus). The Common portion is MJ and ST corpora with case transformations, giving 264K and 13M samples. They also curate 113,788 public font files, filtering out fonts whose uppercase and lowercase glyphs are indistinguishable.

For layout, they stop treating text as one rigid block. Each character i of a text string gets its own position, orientation, and size. Positions follow a parameterized curve with curvature parameter a sampled from [20, 200] (a = 0 meaning straight text), and a global rotation angle sampled uniformly from [0, 2π) supplies varied orientations. Vertical text is handled by swapping the horizontal and vertical axes. Characters are rendered as separate layers, then given effects such as elastic deformation, perspective transformation, and borders. A randomly chosen background is augmented with random text instances, shadows, or embossing, and text color comes from a predefined colormap.

Training also uses online augmentation: downsampling and transmission distortion augmentation (DTAug) simulate small or blurry text.

They build the recognizer from SVTRv2 by swapping its CTC decoder for an attention-based autoregressive decoder (SVTRv2-AR), because CTC assumes monotonic character alignment, which breaks on curved or multi-oriented text.

Finally, the self-evolution learning loop

Authors’ abstract

Large-scale and categorical-balanced text data is essential for training effective Scene Text Recognition (STR) models, which is hard to achieve when collecting real data. Synthetic data offers a cost-effective and perfectly labeled alternative. However, its performance often lags behind, revealing a significant domain gap between real and current synthetic data. In this work, we systematically analyze mainstream rendering-based synthetic datasets and identify their key limitations: insufficient diversity in corpus, font, and layout, which restricts their realism in complex scenarios. To address these issues, we introduce UnionST, a strong data engine synthesizes text covering a union of challenging samples and better aligns with the complexity observed in the wild. We then construct UnionST-S, a large-scale synthetic dataset with improved simulations in challenging scenarios. Furthermore, we develop a self-evolution learning (SEL) framework for effective real data annotation. Experiments show that models trained on UnionST-S achieve significant improvements over existing synthetic datasets. They even surpass real-data performance in certain scenarios. Moreover, when using SEL, the trained models achieve competitive performance by only seeing 9% of real data labels. Code is available at https://github.com/YesianRohn/UnionST.