SkyReels-Text: Fine-Grained Font-Controllable Text Editing for Poster Design

AI summary

Overview

Research area: Computer vision / generative image editing, specifically region-level text editing with explicit font-style control, applied to poster design.

Technical level: Advanced. The paper assumes familiarity with diffusion models, DiT/MMDiT backbones, VAEs, LoRA fine-tuning, ControlNet-style conditioning, and distribution-matching distillation.

Scope (one sentence): The paper introduces SkyReels-Text, a framework that edits the text content of posters while transplanting a user-supplied font style from a cropped glyph patch, without font labels or test-time fine-tuning. Code and models are stated to be publicly available at https://github.com/SkyworkAI/SkyReels-Text (arXiv:2511.13285v2).

What This Paper Is About

General-purpose image editing models can already insert and modify content, but they are unreliable at the specific job professional designers care about: changing the words on a poster while faithfully reproducing the original (often unusual or calligraphic) typeface, position, and layout. SkyReels-Text addresses this by treating text editing as a regional modification task, where the model is given a target text to write, a region to write it in, and a reference glyph patch that defines the desired typography. The stated goal is state-of-the-art performance on both text fidelity and visual realism, with simultaneous editing of multiple text regions that each use a different font style.

Key Contributions

1. Visual conditioning from reference glyph patches: A dual-stream conditioning strategy that feeds the model both a plain-text reference (target content and layout) and a glyph map (font style), enabling fine-grained font control without font labels or test-time fine-tuning.
2. A VLM-based OCR system: A fine-tuned Qwen2.5-VL 7B model that recognizes and localizes ornamental, calligraphic, and custom-designed fonts that conventional OCR methods handle poorly.
3. A content–style decoupling data pipeline: An LLM text-replacement step, local editing with Nano Banana, SAM2 mask extraction, and two-pronged verification (OCR for content, DINO v2 features for style), producing a dataset of 100K aligned original–edited image pairs.
4. Comprehensive evaluation: Experiments on the SkyReels-Text benchmark, the AnyText benchmark (English and Chinese), and handwritten datasets (IAM, CVL), reporting state-of-the-art results in semantic accuracy and typographic style fidelity.

Main Findings

• SkyReels-Text benchmark (200 curated samples): SkyReels-Text reaches a Sentence Accuracy (Sen. Acc.) of 0.8334, NED of 0.9502, spatial IoU of 0.7506, DINO of 0.8503, and B-PSNR of 34.17. Among compared methods, FLUX-Text has the closest Sen. Acc. (0.8266) and a slightly higher B-PSNR (34.53), but a much lower DINO (0.6679); strong commercial models do worse on text fidelity, e.g., Seedream 4.0 at 0.7772 Sen. Acc. and Nano Banana at 0.7290.

• AnyText benchmark (1,000 images from Wukong and LAION): SkyReels-Text reports the best result on every reported metric in both languages. English Sen. Acc. 0.8536, NED 0.9406, FID 6.12, LPIPS 0.0246; Chinese Sen. Acc. 0.7710, NED 0.8764, FID 5.44, LPIPS 0.0192. The paper states this surpasses FLUX-Text by 1.37% (English) and 7.50% (Chinese) in Sen. Acc.

• Zero-shot handwritten text: Without training on dedicated handwritten data, SkyReels-Text reports HWD of 1.32 on IAM (161 writers, CSA-GAN protocol) and 0.89 on CVL (27 writers), compared with DiffBrush at 1.41 and 1.06 respectively. IS is 1.90 on IAM and 1.71 on CVL.

• User preference study: Against four commercial systems, SkyReels-Text reports win rates of 71.8% versus Qwen-Image-Edit and 84.5% versus FLUX-Kontext-Pro, and a 60.2% win rate against both Nano Banana and Seedream 4.0 with roughly a 10% loss rate. The number of study participants is not reported.

• Font Style Reference ablation: Removing the font style reference raises Sen. Acc. (0.9327 vs 0.8334) and NED (0.9734 vs 0.9502), but collapses DINO from 0.8503 to 0.6995. The authors attribute the higher text accuracy to the simpler single-condition task and to OCR difficulty on highly stylized output.

• Text-region weighted loss ablation: Increasing λ from 0 to 10 steadily improves Sen. Acc. (0.7998 → 0.8363) and NED (0.9378 → 0.9558), but λ = 5 is chosen for the best balance, maximizing spatial IoU (0.7506) and DINO (0.8503); λ = 10 lowers B-PSNR to 33.13.

• Inference speed: Distillation via the DMD2 paradigm produces an 8-step student model that removes the need for classifier-free guidance and delivers a speedup of over 10x. Multi-font editing in one inference pass is reported at approximately 8 seconds on a single A800 GPU.

• Small-text and paragraph-level editing: The model corrects small text or garbled AI artifacts down to roughly 20 pixels. For paragraph-level editing, continual LoRA training on 10K samples (5K from the content-style decoupled dataset and 5K synthetic) works for standard fonts, but the paper states that highly unconventional or artistic styles at paragraph level remain challenging due to scarce dense stylized data.

• OCR benchmark (SkyReels-OCR benchmark, 200 images): The fine-tuned OCR model reports 0.9446 Sen. Acc. on regular fonts and 0.9276 on non-regular fonts, versus 0.8386 and 0.6319 for the baseline. Qwen3-VL-8B-Instruct scores higher on regular-font Sen. Acc. (0.9659) but lower on non-regular fonts (0.8925). PP-OCRv4 drops to 0.3358 Sen. Acc. on non-regular fonts.

Methodology in Plain English

The system is built by fine-tuning Qwen-Image-Edit with LoRA (rank 64, batch size 64, AdamW, learning rate 10⁻⁴, 2 epochs, roughly 512 A100-hours). Two streams of visual information guide each edit: a plain-text reference, made by inpainting the masked text region with the target words so the layout is unambiguous, and a glyph map, made by rendering the user's reference glyph onto a canvas the size of the input so the style is unambiguous. A frozen VAE encodes both, and their features are concatenated with the noisy image latent along the sequence dimension before entering MMDiT blocks with trainable LoRA modules. Qwen2.5-VL handles the multimodal editing instruction. No ControlNet, no font labels, no test-time optimization.

Training data is generated rather than simply rendered. Real posters are filtered for resolution, legibility, and quality, then annotated by a fine-tuned Qwen2.5-VL 7B OCR model (about 72 A100-hours of fine-tuning) that copes with calligraphic and custom letterforms. To force the model to separate what is written from how it looks, an LLM (Qwen3-8B) generates replacement words that are semantically disjoint from the originals but match sequence length and casing. Nano Banana then re-renders the text locally, preserving typography, and two checks filter the result: the OCR model must confirm the new content (NED threshold of 0.9) and DINO v2 features must confirm style preservation (similarity threshold of 0.8). Because text covers less than 10% of a typical image, the reconstruction loss is spatially reweighted by a SAM2-derived text mask with λ = 5.

Why This Matters

Impact on research: The paper reframes font control as a regional, reference-conditioned editing problem rather than a prompt-conditioning problem, and shows that explicit glyph patches can serve as a substitute for font labels. It also contributes two datasets (SkyReels-OCR-40K and a 100K-sample content-style decoupled set) and a benchmark, plus an OCR model that substantially outperforms PP-OCRv4/v5 on non-regular fonts.

Real-world applications (as supported by the paper):

• Updating slogans, brand names, dates, or multilingual captions on existing posters while preserving typographic identity.
• Repairing garbled or nonsensical small text in AI-generated artwork, down to roughly 20 pixels.
• Multi-region poster redesign where each text block keeps a distinct font, done in a single pass rather than repeated interaction rounds.
• Handwritten and calligraphic text generation from a single style reference, without per-language or per-domain training.

Industry relevance: The evaluation targets commercial design tools directly, benchmarking against Nano Banana, Seedream 4.0, FLUX-Kontext-Pro, and Qwen-Image-Edit, and reporting user preference win rates against all four. If the claims hold, this points toward design software where a designer's existing typeface library—including custom and non-library fonts—can be applied to generated or edited layouts without manual retyping.

Future Directions

• Paragraph-level artistic typography: The paper explicitly leaves highly unconventional or artistic styles at paragraph scale to future work, citing scarcity of dense, stylized data.
• Closing the accuracy/style trade-off: The ablation shows that adding the font style reference lowers reported Sen. Acc. and NED because OCR systems struggle with highly stylized output; better stylized-text evaluation (or better OCR) would sharpen this measurement.
• Extending the decoupled data pipeline: The current pipeline uses thresholds (NED ≥ 0.9, DINO ≥ 0.8) and an "Effective Text" definition of less than 20% character overlap; how sensitive the model is to these choices is not analyzed in the provided content.
• Efficiency and deployment: The distilled 8-step model is reported at roughly 8 seconds per multi-font edit on a single A800 GPU; further latency reduction for interactive design tools is an open practical question.

Target Audience

Researchers and engineers working on diffusion-based image editing, visual text generation, and typography-aware generative models. The paper is also relevant to applied teams building design and marketing tools who need font-faithful text replacement, and to dataset builders interested in VLM-based OCR and content-style decoupling pipelines. Readers without a background in diffusion models and transformer-based image backbones will find the method section demanding, though the problem framing and results tables are accessible.

Authors’ abstract

Artistic design, particularly poster design, often demands rapid yet precise modification of textual content while preserving visual harmony and typographic intent, especially across diverse font styles. Although modern image editing models have grown increasingly powerful, they still fall short in fine-grained, font-aware text manipulation, limiting their utility in professional workflows. To address this issue, we present SkyReels-Text, a novel font-controllable framework for precise poster text editing. Our method enables simultaneous editing of multiple text regions, each rendered in distinct typographic styles, while preserving the visual appearance of non-edited regions. Notably, our model requires neither font labels nor test-time fine-tuning: users can simply provide cropped glyph patches corresponding to their desired typography - even if the font is not included in any standard library. Extensive experiments on multiple benchmarks demonstrate that SkyReels-Text achieves state-of-the-art performance in both text fidelity and visual realism, offering unprecedented control over font families and stylistic nuances. This work bridges the gap between general-purpose image editing and professional-grade typographic design. Code and models are publicly available at https://github.com/SkyworkAI/SkyReels-Text.