Ovis-Image Technical Report

AI summary

Overview

Research area: Computer Vision — text-to-image generation, specifically accurate in-image text rendering (typography inside generated images).

Technical level: Intermediate. The report assumes familiarity with diffusion models, flow matching, MMDiT, and preference-optimization methods such as DPO and GRPO, though its main claims are stated in accessible terms.

Scope: A technical report describing Ovis-Image, a 7B text-to-image model from the Ovis Team at Alibaba Group that targets high-quality bilingual (Chinese and English) text rendering while remaining deployable on a single high-end GPU.

What This Paper Is About

Reliable, high-quality text rendering inside generated images is difficult at low computational cost, because a model must master both fine-grained visual synthesis and language understanding at once. In practice, strong text rendering has been limited either to very large open models that are hard to deploy or to closed-source systems that cannot be integrated, customized, or reproduced. The paper's goal is to build a specialized 7B text-to-image model that prioritizes text rendering and keeps visual fidelity on general concepts, without scaling to tens of billions of parameters.

Key Contributions

1. A compact 7B text-to-image model for text rendering. Ovis-Image upgrades the prior Ovis-U1 framework by pairing a diffusion-based visual decoder with the more capable Ovis 2.5 multimodal backbone and scaling the vision-side generator to 7B parameters.

2. A text-centric training pipeline. The model is trained through four stages — pretraining, supervised fine-tuning, DPO, and GRPO — with data and objectives deliberately organized around text rendering, including synthetic typographic samples and a GRPO prompt set focused on text placement and styling.

3. An efficient architecture. The report describes a simplified structure relative to Ovis-U1: MMDiT with RoPE, 6 double-stream blocks and 27 single-stream blocks, 24 attention heads, SwiGLU activations, a frozen FLUX.1-schnell VAE, and removal of the refiner structure so that the final Ovis hidden states condition image generation directly.

4. Empirical results across text-rendering and general generation benchmarks. The model is evaluated on CVTG-2K, LongText-Bench, DPG-Bench, GenEval, OneIG-Bench, plus a comparison of inference time and GPU memory, with reported results competitive against larger open models and approaching closed-source systems.

Main Findings

• Text rendering on CVTG-2K: On this benchmark of 2,000 prompts requiring 2 to 5 English text regions, Ovis-Image achieves the highest reported average word accuracy at 0.9200 (2 regions 0.9248, 3 regions 0.9239, 4 regions 0.9180, 5 regions 0.9166), the highest NED at 0.9695, and the highest CLIPScore at 0.8368. For comparison, Qwen-Image reports an average of 0.8288, NED 0.9116, CLIPScore 0.8017, and GPT4o reports an average of 0.8569, NED 0.9478, CLIPScore 0.7982.

• Long-text rendering on LongText-Bench: Ovis-Image reports 0.922 on LongText-Bench-EN and 0.964 on LongText-Bench-ZN. Qwen-Image reports 0.943 EN and 0.946 ZN; GPT4o reports 0.956 EN and 0.619 ZN. The paper highlights Chinese text performance and long English text generation as particular strengths relative to its parameter count.

• General generation on DPG-Bench: Over 1,000 dense prompts, Ovis-Image reports Overall 86.59 (Global 82.37, Entity 92.38, Attribute 90.42, Relation 93.98, Other 91.20). Larger/other systems reported include Qwen-Image Overall 88.32 and Seedream 3.0 Overall 88.27.

• Compositional generation on GenEval: Ovis-Image reports Overall 0.84, with Single object 1.00, Two object 0.97, Counting 0.76, Colors 0.86, Position 0.67, and Attribute binding 0.80.

• Bilingual evaluation on OneIG-Bench: On OneIG-EN, Ovis-Image reports Alignment 0.858, Text 0.914, Reasoning 0.308, Style 0.386, Diversity 0.186, Overall 0.530; on OneIG-ZN, Alignment 0.805, Text 0.961, Reasoning 0.273, Style 0.368, Diversity 0.198, Overall 0.521. The paper emphasizes its performance in the text dimensions.

• Computational overhead: Measured for 1024×1024 images, 50-step sampling, BF16 inference, Ovis-Image uses 24959 MB and 30.56 s on A100, and 24276 MB and 13.74 s on H100. Qwen-Image is reported at 59329 MB / 45.16 s (A100) and 59354 MB / 20.27 s (H100); Flux.1-dev at 34637 MB / 23.51 s (A100) and 34661 MB / 11.03 s (H100); Ovis-U1 at 10528 MB / 8.41 s (A100) and 11937 MB / 4.29 s (H100).

• Model composition: The report lists MMDiT at 7.37B parameters, the text encoder (AIDC-AI/Ovis2.5-2B) at 2.57B, and the VAE (black-forest-labs/FLUX.1-schnell) at 0.08B, for a total of 10.02B. The paper notes Ovis2.5-2B demonstrates better performance than Qwen2.5-VL-7B on the OpenCompass benchmark suite.

• Training stability from a DPO safeguard: The authors adopt a Diffusion-SDPO winner-preserving gradient scaling factor and report that omitting it leads to models that frequently produce noisy or artifact-prone images.

Methodology in Plain English

The researchers start from their earlier Ovis-U1 model and change three things: they use a stronger multimodal backbone (Ovis 2.5) as the text encoder, enlarge the vision-side generator to 7B parameters, and simplify the structure by removing the refiner stage, feeding the text encoder's final hidden states directly into image generation. The visual decoder is an MMDiT trained with flow matching and a standard noise-prediction loss, using a frozen VAE.

Training proceeds in four stages, each initialized from the previous checkpoint. First, large-scale pretraining on a heterogeneous image–text mixture (web-scale, licensed, and synthetic data) with large-scale recaptioning in Chinese and English, plus text-salient slices such as posters, banners, logos, and UI layouts, and synthetic samples from a rendering engine. Second, supervised fine-tuning on a higher-quality subset emphasizing clean visuals, prompts up to 1024×1024 resolution, and varied aspect ratios. Third, Direct Preference Optimization using preference pairs built by taking a high-quality image and a model-generated image for the same prompt and letting an ensemble of automatic scorers (including HPSv3, CLIP, and PickScore) decide the winner. Fourth, GRPO, an on-policy reinforcement stage where multiple candidate images are generated per prompt, scored by reward models, and used to update the policy, run for approximately 500 steps and focused specifically on text-rendering prompts.

Infrastructure details include PyTorch with Hybrid Sharding Data Parallel, gradient checkpointing, activation offloading, bfloat16 mixed precision with FP32 master weights, Flash Attention, regional compilation, and distributed checkpointing.

Why This Matters

Impact on research: The results suggest that frontier-like text-aware generation does not inherently require extremely large models; it can emerge from moderate-scale models when architecture, data curation, and alignment objectives are explicitly organized around in-image text rather than treated as a byproduct of generic image synthesis. The paper also demonstrates a working recipe that combines a multimodal backbone with a diffusion decoder and a staged alignment pipeline.

Real-world applications (as described in the paper):

• Posters, banners, and title cards requiring legible, correctly spelled typography.
• Logos and product labels where text must match the intended linguistic content.
• UI mockups and UI elements where layout and text placement must be tightly controlled.
• Infographics and other text-heavy, layout-sensitive design assets, including stylized compositions and illustrations.

Industry relevance: The model is positioned as deployable on a single high-end GPU with moderate memory, supporting low-latency interactive use and batch production serving. This matters for applications where tens-of-billions-parameter models are impractical, and for organizations that need integration, customization, and reproducibility that closed-source systems do not offer.

Future Directions

1. Closing the remaining gap to frontier systems. Ovis-Image reports text rendering on par with much larger open models and approaching closed-source systems such as Seedream and GPT4o, but the paper does not report matching them; the remaining distance, especially on the general-generation benchmarks where Qwen-Image and Seedream 3.0 report higher Overall scores, is a natural next target. The paper does not report a planned approach for this.

2. Extending beyond text rendering. Ovis-U1 covered multimodal understanding, text-to-image generation, and image editing in one framework; Ovis-Image specializes in generation, so how editing and understanding capabilities would be retained or restored at this scale is an open question the report does not address.

3. Efficiency trade-offs. Ovis-Image uses more memory and time than the smaller Ovis-U1 (24959 MB and 30.56 s versus 10528 MB and 8.41 s on A100), so further inference-cost reduction is a logical direction. The paper does not report planned work on this.

4. Systematic study of the training recipe. The report attributes gains to data curation, the DPO safeguard, and the GRPO text-focused prompt set, but does not report ablation studies isolating each choice, leaving the relative importance of individual components unclear from the provided content.

Target Audience

Researchers and engineers working on text-to-image generation, particularly those interested in in-image text rendering, efficient diffusion architectures, multimodal backbones, and preference-optimization pipelines (DPO and GRPO). It is also relevant to practitioners who need deployable, customizable generation models for design-oriented content such as posters, banners, logos, and UI mockups, and to readers following the Ovis model family as an extension of Ovis-U1.

Authors’ abstract

We introduce $\textbf{Ovis-Image}$, a 7B text-to-image model specifically optimized for high-quality text rendering, designed to operate efficiently under stringent computational constraints. Built upon our previous Ovis-U1 framework, Ovis-Image integrates a diffusion-based visual decoder with the stronger Ovis 2.5 multimodal backbone, leveraging a text-centric training pipeline that combines large-scale pre-training with carefully tailored post-training refinements. Despite its compact architecture, Ovis-Image achieves text rendering performance on par with significantly larger open models such as Qwen-Image and approaches closed-source systems like Seedream and GPT4o. Crucially, the model remains deployable on a single high-end GPU with moderate memory, narrowing the gap between frontier-level text rendering and practical deployment. Our results indicate that combining a strong multimodal backbone with a carefully designed, text-focused training recipe is sufficient to achieve reliable bilingual text rendering without resorting to oversized or proprietary models.