Real In, Real Out: What If We Only Use Real Data for Scene Text Editing?

AI summary

Overview

Research area: Computer Vision — Scene Text Editing (STE), conditional diffusion models, and synthetic data generation for Scene Text Recognition (STR).

Technical level: Advanced. The paper assumes familiarity with diffusion models, ControlNet-style conditioning, VAEs, and STR architectures.

Scope: A data-centric study showing that Scene Text Editing can be trained purely on real images through self-paired supervision (shuffling and cropping), and that the resulting model produces more realistic edits and useful long-text recognition data.

What This Paper Is About

Scene Text Editing aims to replace the text in a photo with new text while keeping the original color, texture, perspective, and lighting intact. Every existing method needs paired before/after images, which do not exist naturally, so the field relies on synthetic data — and that synthetic-to-real gap causes leftover artifacts, hallucinated or misspelled characters, and an overall "synthetic flavor." The authors ask a deliberately simple question: what if each real image is turned into its own training pair by scrambling its text content while preserving its style?

Key Contributions

1. TextRIRO, a self-paired STE method trained only on real data. A single real text image is pushed through a content-destructive but style-preserving operator (column-wise Shuffle followed by Crop) to become a pseudo-source image, which the model must reconstruct conditioned on the original text label. No synthetic triplets and no explicit or implicit disentanglement modules are used.

2. A systematic comparison of self-pairing operators. The paper ablates Shuffle-only, Crop-only, Shuffle&Crop, Mask, Block, and identity (no perturbation), and shows Shuffle&Crop gives the best accuracy/realism trade-off (82.42% ACC, 0.960 NED, 43.59 FID in the core ablation).

3. State-of-the-art results judged by "train-on-edits" proxy. TextRIRO reaches 86.67% average accuracy when its edited images are used to train a recognizer, a 9.78% gain over the next-best approach, and 11.69% on irregular samples — a stronger criterion than recognizer-scored editing alone.

4. TextRIRO-3M, a large-scale long-text recognition dataset. Filtered edited images are concatenated with their originals (doubling sequence length) to build about 3M style-consistent samples. Mixing this with real data raises Union14M-Benchmark average accuracy by 1.91% and the Multi-words subset by 3.28% for MAERec.

Main Findings

• Realism–accuracy trade-off is real and measurable. On ScenePair (1,280 real pairs), RS-STE posts the highest word accuracy at 91.80% — above the real-image upper bound of 88.44% — but its FID is 49.59 versus TextRIRO's 41.53. TextRIRO reaches 85.23% ACC, 0.964 NED, and 41.53 FID, staying close to real data without looking synthetic.

• Variable-length editing is where the gap becomes large. On the new RIROCommon benchmark, TextRIRO gets 79.31% average equal-length accuracy versus 70.67% for TextCtrl and 60.24% for RS-STE; on variable-length editing it gets 47.64%, against 11.98% and 11.73% for the same two baselines. The paper notes equal-length accuracy is 19.07% higher than RS-STE.

• Synthetic-dominated training has a ceiling. Under the train-on-edits proxy, TextCtrl and RS-STE differ by only 0.90% in average accuracy (75.99 vs 76.89), which the authors read as evidence that architecture changes do not close the synthetic domain gap. TextRIRO reaches 86.67% against a real-data upper bound of 96.77%, leaving a 10.10% gap attributed to accumulated edit failures in the training set.

• Shuffle and Crop play complementary, layout-dependent roles. Shuffle dominates on straight text (79.00% on IC13 alone vs 77.83% for Crop only), while Crop dominates on curved (60.93% vs 54.83% on CUTE) and multi-oriented text (69.76% vs 66.67% on SVTP). The combined operator is best on straight text at 80.16%.

• Text conditioning must be learned on real scene text. Removing pre-training collapses accuracy to 9.38%; glyph pre-training gives 53.44%; an STR-pretrained encoder gives 56.72% at the same 256×256 setting. With the full configuration (64×256 input, SD 2.1), STR pre-training reaches 82.42% versus 81.95% for glyph pre-training and 45.96 FID versus 45.96→45.96 comparison noted as 45.96 for glyph.

• Rectangular inputs and a stronger VAE matter. Moving from 256×256 to 64×256 lifts accuracy from 56.72% to 78.91%; switching the VAE from SD 1.5 to SD 2.1 lifts it further to 82.42%.

• Degenerate self-pairing fails badly. Masking collapses to 13.13% ACC and 58.08 FID; Block shuffling reaches 79.36% ACC but 50.67 FID; identity pairing (no perturbation) degenerates to copying at 4.37% ACC.

• ControlNet beats a DiT-style alternative. An OminiControl-based variant reaches only 52.89% ACC and 0.803 NED on ScenePair (FID 39.65), versus 85.23% ACC and 0.964 NED for the ControlNet-based model (FID 41.53). Multi-scale injection is judged better for fine-grained character fidelity.

• Results are not recognizer-specific. With three independent recognizers (official TRBA, SVTRv2, PPOCR-v6-rec), TextRIRO scores 85.23 / 85.86 / 86.25 while RS-STE scores 91.80 / 90.39 / 91.80, confirming RS-STE is recognizer-friendly but less realistic.

• Synthetic data usefulness depends on style consistency, not just volume. Concatenating two independent real images (Real-Concat) reaches 87.33% Union14M average — better than most editing baselines but below TextRIRO's 87.98% at 1M scale and 88.00% at 3M, which the authors attribute to style inconsistency between stitched samples.

• Scaling saturates unevenly. With MAERec, going from 1M to 3M TextRIRO samples moves Union14M average from 87.98 to 88.00 while Multi-words rises from 89.08 to 90.17; with SVTRv2-S the average moves from 85.28 to 85.81.

• The shuffle probability is tuned, not arbitrary. At p_shuffle 0.5 / 0.7 / 1.0, ACC is 80.78 / 82.42 / 81.72 and FID is 42.76 / 43.59 / 44.01, so 0.7 is adopted as the default.

Methodology in Plain English

The central trick is to manufacture training pairs out of unpaired real photos. Take a real text image and its label. Split the image along its long axis into as many equal-width columns as there are characters, then with probability 0.7 randomly permute those columns. Finally, crop a random contiguous run of columns. The result is an image that still carries the original font weight, color, lighting, and perspective — but no longer reads as coherent text. The model is asked to reconstruct the untouched original from this scrambled, cropped version plus the target string. Because the only reliable way to do that is to copy style from the input and take character shapes from the text conditioning, the model learns editing behavior without ever seeing a synthetic render.

The architecture is intentionally conventional. A ViT-based style encoder reads the scrambled image and produces multi-scale features at four pyramid levels, injected into a U-Net through ControlNet-style zero-convolutions. Text is encoded character by character with an encoder pre-trained on real scene text using an autoregressive STR objective rather than synthetic glyph or CLIP pre-training. The VAE and text encoder are frozen; the style encoder and U-Net are trained with a standard noise-prediction diffusion loss on 64×256 inputs. Training data is U14M-Filter (3.22M annotated instances).

For the downstream dataset, the trained model edits each real image to a same-length different string, an STR model filters out samples whose predicted text does not match the target, and accepted edits are concatenated with their originals to double the sequence length — producing TextRIRO-3M.

Why This Matters

The paper reframes the STE problem as a data problem rather than an architecture problem, and backs that reframing with an ablation showing that two very different synthetic-trained architectures (TextCtrl and RS-STE) land within 0.90% of each other while a data change moves the number by 9.78%. That is a useful corrective for a subfield that has mostly competed on modules.

Real-world applications:

• Advertising and e-commerce localization — swapping pricing, product names, or slogans in existing photographs without reshooting or re-rendering.
• Film, gaming, and signage post-production — replacing storefront or billboard text in footage while preserving perspective, lighting, and grain.
• Privacy redaction with plausible replacement — substituting real signs or plates with visually consistent fake text instead of blurring.
• Training data for OCR/STR systems in data-scarce scripts or layouts — long-text and multi-word samples, the categories where the reported gains are largest (Multi-words +3.28%).

Industry relevance: the work targets the exact bottleneck faced by teams that need cheap, realistic labeled OCR data. The 3M-sample dataset construction is a directly reusable recipe, and the finding that ControlNet-style injection beats a token-efficient DiT alternative is actionable for anyone building conditioned diffusion pipelines.

Future Directions

• Closing the remaining 10.10% gap to real-data training. The authors attribute it to edit failures that slip past the STR filter and accumulate; better filtering or failure-aware training is an open problem.

• Multilingual editing in low-resource settings. The paper explicitly states TextRIRO excels in English because real scene text data is abundant, and names accurate, realistic multilingual STE as future work.

• More specialized DiT-based fine-grained control. The OminiControl variant reached only 52.89% ACC, and the authors leave stronger DiT conditioning designs as future work.

• Moving beyond label-dependent self-pairing. The authors note that fully label-free training is "theoretically infeasible" because STE requires specifying the edited text, leaving open how to reduce reliance on character counts and transcriptions.

Target Audience

Researchers and graduate students working on scene text editing, text-image synthesis, and diffusion-based image editing; OCR/STR practitioners who need realistic synthetic training data; and applied engineers in post-production or document/AI data pipelines evaluating whether a real-data-only training scheme can replace synthetic render pipelines. Readers without background in diffusion models will find the high-level argument accessible but the results tables difficult to interpret.

Authors’ abstract

Aiming to modify textual content while preserving the original style, Scene Text Editing (STE) has significant practical value in many applications. STE typically requires paired images before and after editing for training. However, recent approaches rely heavily on synthetic paired data and complex disentanglement strategies. They struggle to faithfully capture the diversity and complexity of natural scenes, often producing artifacts or hallucinations. In this work, we challenge the common practice and investigate whether STE can be driven by turning each real sample into its own paired supervision. Concretely, we explore multiple ways to disrupt the original image (e.g., Crop, Shuffle, Mask) to construct self-paired training data, and build a strong baseline (TextRIRO}) upon a conditional diffusion architecture. Our experiments show that combining character-column shuffling with image cropping (Shuffle&Crop) effectively destroys text semantics while preserving key style cues, enabling the model to learn editing behaviors directly from the real text image. With this concise yet real-oriented design, TextRIRO achieves promising performance in both accuracy and visual realism across real-world benchmarks, substantially mitigating the synthetic-to-real domain gap. Furthermore, we leverage TextRIRO to build TextRIRO-3M, a large-scale realistic dataset for long-text recognition by concatenating edited and original images. Experimental results show that training recognition models with TextRIRO-3M significantly improves recognition accuracy, demonstrating that TextRIRO provides highly useful training resources for downstream tasks. Overall, TextRIRO lifts STE to realistic, reliable, and widely applicable new levels. Code is available at https://github.com/YesianRohn/TextRIRO.