InnoText: A Unified Model for Visual Text Generation and Editing

AI summary

Overview

• Research area: Computer Vision — diffusion-based visual text generation and editing (typographic image synthesis), with a focus on bilingual English–Chinese text.
• Technical level: Intermediate. The paper assumes familiarity with diffusion models, Diffusion Transformers (DiT), LoRA fine-tuning, and standard image-generation evaluation metrics (LPIPS, NED, FID, CLIPScore).
• Scope: The paper proposes a single DiT-based model, InnoText, that performs both visual text generation and visual text editing, along with three training modules and a new bilingual dataset and benchmark.

What This Paper Is About

Generating and editing text inside images is harder than general image synthesis because the text has to be structurally regular and legible, and this gets worse for small fonts and for non-Latin scripts such as Chinese. Existing UNet-based models often produce distorted or artifact-ridden text, while newer Diffusion Transformer (DiT) models are usually built for only one of the two tasks (generation or editing), which leads to redundant training pipelines, inconsistent visual styles, and weaker cross-task generalization. InnoText aims to unify both tasks in one DiT-based model with scale-aware and small-character-aware training, supported by a newly curated bilingual dataset and benchmark.

Key Contributions

1. A unified DiT framework. InnoText reformulates visual text tasks through in-context learning with specialized input patterns, allowing one model to switch between visual text generation and editing. Generation is formulated as simultaneous text-to-image synthesis rather than the text-guided image-to-image "insertion" used by conventional unified models.
2. Font Size-Aware Modulation (FSAM). A module that uses a grayscale "size map" (encoding inverse font height per detected text region) to produce scale and shift modulation parameters that recalibrate hidden features across different font scales.
3. Small-Character Aware Augmentation (SCAA) plus a Task-Specific Region Weighted Loss. SCAA stochastically enlarges small-font regions during training to improve fine-grained fidelity; the task-specific loss applies region-weighted supervision for editing and global supervision for generation.
4. InnoText-30K dataset and InnoText-Bench. A bilingual dataset (20K Chinese, 10K English) built by extending Lex-10K into Chinese, adding a 10K subset of Anyword-3M, and filtering with aesthetic scoring, PPOCR-v4 re-annotation, and DeepSeek-VL2 recalibration; plus a diagnostic benchmark of 1,500 images.

Main Findings

• Editing performance leads on most metrics. On the AnyText-Benchmark editing task, InnoText reaches English Sen. Acc 0.7988, NED 0.9016, LPIPS 0.0786, and Chinese Sen. Acc 0.7257, NED 0.8579, LPIPS 0.0591 — the highest sentence accuracy and lowest LPIPS among compared methods.
• Generation performance also leads. On the generation task, InnoText scores English Sen. Acc 0.6586, NED 0.8066, LPIPS 0.4787 and Chinese Sen. Acc 0.5907, NED 0.6837, LPIPS 0.5311, surpassing Flux-Fill, AnyText, AnyText2, and TextFlux.
• FSAM is the single most impactful module. Removing FSAM raises the generation LPIPS from 0.5311 to 0.5928, described as the largest degradation among the ablations.
• SCAA matters most for small text. Excluding SCAA consistently lowers sentence accuracy and visual similarity, with the effect described as especially pronounced in editing.
• The Task-Specific Region Weighted Loss contributes moderately. Removing it causes a 1.93% drop in editing accuracy.
• Fine-tuning baselines on the new data is not enough. AnyText-FT and AnyText2-FT improve over their originals after fine-tuning on InnoText-30K but remain inferior to InnoText.
• Human evaluation favors InnoText. A double-blind side-by-side study with 15 participants over 100 cases (50 generation, 50 editing) against AnyText2 gave InnoText an overall preference rate exceeding 78%; an expanded study used 45 participants and added Flux-Text (editing) and EasyText (generation) as baselines.
• Strong results on MARIO-Eval. Using 5,000 random MARIO-Eval samples, Ours (30K) achieves FID 3.84, CLIPScore 0.3356, OCR-Accuracy 0.6501, OCR-F1 0.8605; Ours (3M) achieves FID 3.76, CLIPScore 0.3318, OCR-Accuracy 0.6471, OCR-F1 0.8583.
• Higher LoRA rank helps. Editing Sen. Acc rises from 0.5902 (rank 32) to 0.6693 (rank 256), and generation Sen. Acc from 0.5133 to 0.5889, with rank 128 used in the main setup.
• Compute comparison. Table 6 lists about 960 GPU hours for Flux-Text (3M), 128 for Ours (30K), and 768 for Ours (3M).
• Known failure modes. Editing can duplicate local text when mask regions are overly large, and small densely structured characters can show minor stroke inaccuracies; generation struggles with redundant characters and high-stroke-density Chinese glyphs.

Methodology in Plain English

The team starts from Flux.1 Fill dev, an inpainting model built on the Flux backbone, and keeps its generative capacity while changing how inputs are constructed. For editing, the model receives a masked image, a glyph image rendered from the target text, and the mask itself, concatenated along the spatial dimension. For generation, the masked image is replaced by an entirely black canvas and the mask by a constant value, so the same architecture produces text from scratch. Training alternates between the two tasks with equal probability of 0.5.

To stop the model from losing quality on small text, FSAM builds a grayscale size map: for each detected character box it estimates a font height as the smaller of the box height and width, then writes the inverse height into that spatial location and normalizes the map to [0, 1]. The map is patchified into tokens, linearly projected into latents, and decoded by two MLP branches into scale and shift parameters. Scale is bounded with a sigmoid and amplified by a nonlinear function; shift uses a Tanh activation scaled by a fixed parameter. These modulate the hidden states of the DiT block.

SCAA takes characters whose estimated height falls below a threshold and stochastically resizes those regions by a base factor plus random perturbation. This is applied at inference for editing to keep train-test consistency, but skipped for generation so the original font-size distribution is preserved. The loss treats the two tasks differently: editing uses a size-map-weighted flow matching loss squared, emphasizing modified regions, while generation uses the unweighted global flow matching loss.

Training uses batch size 2, the Prodigy optimizer with weight decay 0.01, LoRA rank 128, 8 NVIDIA A100 GPUs, 30,000 steps, and 512×512 images. Data was built by translating Lex-10K captions into Chinese, expanding them semantically with Intern-VL3, generating Chinese image–text pairs with the Seedream engine, merging with an Anyword-3M subset, then filtering by aesthetic score, re-annotating with PPOCR-v4, and recalibrating captions with DeepSeek-VL2.

Why This Matters

• Research impact: The paper argues that isolating generation and editing into separate DiT models wastes training compute and hurts cross-task generalization, and it offers a concrete unification recipe plus a bilingual dataset and benchmark where high-resolution Chinese visual text data was previously scarce. Writing (dense glyph structures) and tiny text were largely unsolved, so the FSAM and SCAA designs give the field reusable ideas for scale-aware conditioning and foveal-style augmentation.
• Real-world applications:
  • Multilingual design and advertising tools that need to render or restyle text in both English and Chinese images.
  • Augmented reality and digital media, where text inside images must be replaced dynamically while blending into the scene.
  • Accessibility, such as modifying or clarifying text in images for visually impaired users.
  • Education and interactive content, where personalized or localized text overlays are generated on the fly.
• Industry relevance: The results are reported against commercially relevant baselines (Flux-Fill, AnyText, AnyText2, TextFlux, Flux-Text, EasyText) using released pretrained weights, and efficiency is reported in GPU hours. The LoRA-rank ablation (32 to 256) and the 30K-vs-3M data comparison directly inform practical fine-tuning budgets. The paper also discusses misuse risks such as deceptive content and forged visual evidence, and recommends watermarking, usage controls, and responsible deployment.

Future Directions

• Improve spatial controllability in editing, since overly large mask regions can still produce duplicated text.
• Address character-level structural consistency for high-stroke-density Chinese glyphs and repetitive character patterns, which remain the main generation failures.
• Close the remaining gap on small and densely structured characters under extreme font scales, where stroke-level inaccuracies persist despite FSAM and SCAA.
• Explore scale-aware and multimodal foundational models more broadly, as the authors suggest, and evaluate whether the unified formulation extends to scripts beyond English and Chinese.

Target Audience

Researchers and engineers working on diffusion-based image generation, controllable generation, and diffusion transformers, especially those interested in text rendering, multilingual (English–Chinese) synthesis, and unifying multiple visual tasks in one model. It is also relevant to practitioners who need to fine-tune or evaluate text-in-image models on a limited compute budget, and to readers looking for a bilingual dataset and benchmark for typographic generation.

Authors’ abstract

Diffusion models have recently achieved remarkable success in high-fidelity image synthesis, yet their application to visual text generation and editing remains relatively underexplored. Unlike general image generation, visual text tasks demand precise structural regularity and legibility, which may pose additional challenges for small-scale text and non-Latin scripts such as Chinese. Existing UNet-based models often struggle to produce clear and coherent text, while DiT-based models, though more expressive, are typically limited to a single task, which may lead to redundant training pipelines, inconsistent visual styles, and reduced cross-task generalization. To address these challenges, we propose InnoText, a unified DiT-based framework capable of performing both text generation and editing within a single model. We introduce a Font Size-Aware Modulation (FSAM) module to enhance representations across font scales, a Small-Character Aware Augmentation strategy to improve fine-grained fidelity, and a Task-Specific Region Weighted Loss for adaptive optimization. To support training and evaluation, we also construct a high-quality bilingual (English-Chinese) visual text dataset covering diverse fonts, sizes, and backgrounds. Experimental results demonstrate that our method achieves superior generation accuracy and editing quality, producing visually appealing and realistic text images.