OvisOCR3 Technical Report

AI summary

Overview

• Research area: Computer vision / document AI — optical character recognition, document parsing, and document understanding with vision-language models.
• Technical level: Intermediate. The task definitions and evaluation are accessible, but the training recipe (on-policy distillation, loss-normalization fusion, GRPO-based reinforcement learning) assumes familiarity with modern LLM post-training.
• Scope (one sentence): The report describes OvisOCR3, a single 0.8B-parameter model trained to handle five OCR-related task families — document parsing, general OCR, key information extraction, document figure captioning, and cross-page table reconstruction — and reports benchmark comparisons against general-purpose vision-language models and specialized OCR models.

What This Paper Is About

Most OCR systems are built for one job: transcribing text accurately. Real document workflows also demand page structure, field extraction against a schema, descriptions of figures, and tables that continue across page boundaries — each of which has traditionally required its own specialized model. OvisOCR3 asks whether one compact 0.8B model can cover all of these task families at specialist-level quality, and proposes a staged training recipe to get there.

Key Contributions

1. A unified five-task OCR model in 0.8B parameters. OvisOCR3 shares one set of parameters across document parsing, general OCR (ten distinct interfaces), key information extraction, document figure captioning, and cross-page table reconstruction, with task-specific instructions defining what to output and in what representation.

2. A four-stage training recipe for multi-domain balancing. Continued pretraining on roughly 72 million sampled examples, domain-balanced mid-training on roughly 8.8 million examples, multi-teacher on-policy distillation from five domain-specific 4B teachers, and mixed-task reinforcement learning with domain-specific rewards.

3. A loss-normalization fusion trick for imbalanced response lengths. The authors note that balanced sample counts do not produce balanced token-level supervision (general OCR is about 22.8% of Stage 2 examples but only 5.2% of supervised response tokens; KIE is 16.1% of examples but 2.5% of tokens), so they train two branches with token-level and sample-level loss normalization and merge their weights at the end of mid-training.

4. A two-stage, recursive procedure for cross-page tables. Pages are parsed independently into Markdown with HTML tables, then the same model takes adjacent page images plus their parsed representations to detect continuations and merge tables; multi-page tables are handled by processing adjacent pairs from the end of a page sequence toward its beginning.

Main Findings

• Document parsing: OvisOCR3 reaches an overall score of 96.74 on OmniDocBench v1.6 (1,651 pages), above the reported baselines in the visible comparison table, including Qwen3.8-Max at 93.70, Gemini 3 Pro at 92.91, and Gemini 3 Flash at 92.62. It also reports the highest overall scores on PureDocBench, Real5-OmniDocBench, and MDPBench.

• General OCR: It ranks first on OCRBench v2, scoring 65.9 on the English split and 61.3 on the Chinese split, and outperforms the baselines on OCRBench.

• Key information extraction: It outperforms baselines on all four KIE benchmarks, covering open, specified-field, and schema-based extraction.

• Figure captioning: On DocFigCapBench it achieves the highest localization F1 among compared models and a caption score close to Qwen3.8-Max, leading specialized OCR models on both metrics. Captions are constrained to English text of no more than 100 words, with optional Mermaid representations for flowcharts.

• Cross-page tables: It leads on Relation F1 on CrossTabBench and on truncated-table TEDS on MPDocBench-Parse.

• Distillation choice: Top-K distillation outperformed the policy-gradient implementation of on-policy distillation, particularly on document parsing and general OCR — the authors attribute this to relative preferences over multiple candidate tokens helping a capacity-constrained student absorb teacher knowledge for coordinated text and structure generation.

• Loss normalization matters by task: Sample-level loss empirically outperformed token-level loss on short-response tasks such as general OCR and key information extraction.

• Reported figures are incomplete here: The paper content is truncated mid-way through Table 3, so full per-metric document-parsing baselines beyond the Overall column are not available in this text; latency, memory footprint, and full parameter counts for some baselines are not reported.

Methodology in Plain English

• Base model and data. The model starts from Qwen3.5-0.8B, with vision encoder, multimodal connector, and language model all updated jointly. Training data mixes real and synthetic examples across document genres, languages, layouts, and acquisition conditions (including photographed and degraded pages).

• Stage 1 — continued pretraining. A broad visual-text mixture of about 72 million sampled examples, with document parsing at roughly 57% and general OCR at roughly 29% of the sampling budget and the remaining three domains sharing the rest.

• Stage 2 — domain-balanced mid-training. About 8.8 million examples, with each domain contributing roughly 16–23% of the sample budget, plus deliberately hard cases: multilingual photographed pages, small or dense figures, extraction targets with empty fields, and hard negative cross-page relations. Because long responses still dominate token-level supervision, two branches are trained with token-level and sample-level loss normalization and their weights are merged.

• Stage 3 — multi-teacher on-policy distillation. Five teachers, each initialized from Qwen3.5-4B and fine-tuned then RL-tuned on one domain, supervise the shared 0.8B student on the student's own sampled responses (reverse KL over the student's top-K tokens, with a per-position cap). Each example is supervised only by its domain's teacher.

• Stage 4 — mixed-task reinforcement learning. GRPO optimizes the model jointly with domain-specific rewards: normalized edit distance for text, TEDS for tables, CDM for formulas, F1 and IoU for localization and figure grounding, field-value matching with penalties for wrong or missing fields for KIE, and a relation-first reward for cross-page tables where TEDS only counts if the page associations are correct.

• Output conventions. Localization throughout uses bounding boxes given by upper-left and lower-right corners, with coordinates normalized to integers in 0–999 relative to image width and height.

Why This Matters

The report argues that a single small model can replace a portfolio of task-specific OCR systems while improving on them, which matters for deployment cost and maintenance. For research, it offers a concrete recipe — rebalancing by loss normalization, multi-teacher distillation from per-domain teachers, and mixed-task RL — for handling tasks whose correct outputs differ wildly in length and in what counts as success.

Real-world applications:

• Document digitization and archiving: full-page parsing into Markdown/JSON with layout, reading order, formulas, and tables preserved.
• Back-office automation: receipts, invoices, forms, and medical documents extracted into template-guided JSON for downstream systems.
• Accessibility and content repurposing: figure captioning and flowchart-to-Mermaid conversion for documents whose meaning lives partly in images.
• Financial and scientific reporting: reconstruction of tables that break across consecutive pages, where merging fragments correctly is the bottleneck.

Industry relevance: the model's small size and single-checkpoint deployment target settings where per-page inference cost and model-serving complexity matter, and the released checkpoint on Hugging Face makes it directly usable.

Future Directions

• Closing the remaining gaps: the caption score on DocFigCapBench is described only as close to Qwen3.8-Max, so further work on figure description quality is implied.
• Scaling and teacher design: the recipe uses five 4B teachers; whether more or larger teachers, or different distillation objectives, improve the student is left open.
• Beyond the covered tasks: the paper's five families do not include, for example, handwriting-specific or multi-page reasoning tasks; extending the unified formulation is a natural next step.
• Efficiency reporting: the provided content does not report inference latency, throughput, or memory, which would be needed to substantiate deployment claims.

Target Audience

Researchers and engineers working on document AI, OCR, and multimodal LLMs — particularly those interested in compact models, multi-task training recipes, distillation from domain teachers, and reinforcement learning with task-specific rewards. Practitioners building document-processing pipelines will also benefit from the detailed prompt specifications for each task interface.

Authors’ abstract

We introduce OvisOCR3, a 0.8B model for a broad range of OCR-related tasks. OvisOCR3 unifies document parsing, general OCR, key information extraction (KIE), document figure captioning, and cross-page table reconstruction in a single model. We train the model on real and synthetic data through continued pretraining and use domain-balanced mid-training to balance supervision across tasks with different data volumes and response lengths. Multi-teacher on-policy distillation transfers expertise from five domain-specific 4B teachers into the shared model, followed by mixed-task reinforcement learning with domain-specific rewards. We compare OvisOCR3 with general-purpose vision-language models and specialized OCR models on public and in-house benchmarks covering all five task families. In these comparisons, it achieves an overall score of 96.74 on OmniDocBench v1.6 and the highest overall scores on PureDocBench, Real5-OmniDocBench, and MDPBench. On OCRBench v2, it ranks first, scoring 65.9 and 61.3 on the English and Chinese splits, respectively. It also outperforms the baselines on OCRBench and all four KIE benchmarks. On DocFigCapBench, it achieves the highest localization F1 and a caption score close to that of Qwen3.8-Max. For cross-page tables, it leads in Relation F1 on CrossTabBench and truncated-table TEDS on MPDocBench-Parse. These results demonstrate that a compact 0.8B model can achieve strong performance across diverse OCR-related tasks. OvisOCR3 is available at https://huggingface.co/ATH-MaaS/OvisOCR3.