PolyOCR-Venus: Unified OCR Foundation Models for Text-Centric Visual Intelligence

AI summary

Overview

Research area: Computer Vision / multimodal foundation models, specifically Optical Character Recognition (OCR) and text-centric visual intelligence.

Technical level: Advanced. The paper assumes familiarity with multimodal large language models (MLLMs), reinforcement learning post-training (GRPO), knowledge distillation, and document-parsing benchmarks.

One-sentence scope: This technical report from Ant Group's GuangJian Team describes PolyOCR, a family of unified OCR foundation models (2B, 9B, and a specialized 2.7B multilingual variant) trained on a ~60-million-instance OCR corpus with a new competence-aware post-training method called Competence-Guided Policy Optimization (CGPO), evaluated on OCRBench v2.1, CC-OCR, an in-house KIE Benchmark, OmniDocBench v1.6, and MDPBench.

What This Paper Is About

Existing OCR systems tend to fall into two camps: specialized models that are strong on one task (scene text, document parsing, tables, or structured extraction) but fragment knowledge across scenarios, and general-purpose multimodal models that handle OCR only as an implicit side effect of broad pretraining and therefore perform unevenly on precise recognition, localization, and structured extraction. The paper's goal is to build a single unified OCR foundation model that balances recognition, localization, parsing, extraction, and reasoning under one instruction-following interface, supported by a large-scale data engine and an adaptive post-training strategy that keeps heterogeneous OCR tasks from overwhelming one another.

Key Contributions

1. PolyOCR model family: Unified OCR foundation models integrating text recognition, text localization, document parsing, information extraction, visual text understanding, and OCR reasoning into one generative framework. The family includes general-purpose variants (2B and 9B, built on Qwen3.5) and a specialized multilingual configuration, PolyOCR-2.7B, which combines a 1.2B-parameter ViT vision encoder with a 1.5B-parameter Qwen2.5 text decoder and is trained on an in-house corpus covering 21 language/script groups.

2. A large-scale OCR data engine: Converts heterogeneous visual resources into quality-controlled multi-task supervision via task-oriented construction, controllable synthesis, and verification. After capability balancing and sampling, it yields approximately 60 million high-quality OCR training instances.

3. Competence-Guided Policy Optimization (CGPO): A competence-aware post-training framework that combines verifier-based GRPO with on-policy distillation, routed sample-wise according to teacher reliability and the evolving teacher–student competence gap, to balance heterogeneous task learning and avoid catastrophic forgetting.

4. OCRBench v2.1 and expanded evaluation: A revision of OCRBench v2 with manually verified annotation corrections and task-aligned scoring metrics, alongside evaluation across CC-OCR, an in-house KIE Benchmark, OmniDocBench v1.6, and MDPBench for general, multilingual, and industrial OCR scenarios.

Main Findings

• State-of-the-art or highly competitive results: Extensive experiments across OCRBench v2.1, CC-OCR, in-house KIE Benchmark, OmniDocBench v1.6, and MDPBench show PolyOCR achieving state-of-the-art or highly competitive performance. The paper states the family exhibits strong and balanced capabilities across recognition, localization, document parsing, information extraction, visual text understanding, and OCR reasoning. (The truncated content does not report the full per-benchmark score tables.)

• GRPO versus GRPO + OPD: The ablation reported (Table 9) gives a CC-OCR score of 81.72 for GRPO and 82.18 after adding fixed-weight OPD, supporting the use of complementary supervision.

• The equal-reward problem in long-text OCR: The paper illustrates a financial document with 1,000 characters after normalization containing the amount 18,650. If eight student rollouts each substitute one character (e.g., transcribing it as 18,850 or 18,350), normalized edit similarity assigns all eight the same reward of 1 − 1/1000 = 0.999. All group-relative advantages then become zero, so the group contributes no reward-driven policy gradient to correct the digit.

• Capability imbalance in the SFT corpus: Across approximately 60 million SFT instances, the four data groups — Text Perception (TP), Document Structure Understanding (DS), OCR Understanding and Reasoning (OR), and General Multimodal Alignment (GA) — account for 32.5%, 30.0%, 22.5%, and 15.0% respectively overall.

• Curriculum schedule matters: The three-phase schedule (Early / Middle / Late, at 20M instances each) shifts TP from 40.0% to 25.0%, DS from 35.0% to 25.0%, OR from 10.0% to 35.0%, while GA stays fixed at 15.0%.

• Cross-model agreement as a quality signal: For document parsing, sampled PDF pages are independently parsed by PaddleOCR-VL and MinerU2.5-Pro, and inter-model similarity is used as a proxy for annotation quality; low-agreement pages are corrected by Qwen-3.5 using both outputs plus the original image.

• Table annotation consistency via TEDS: For table localization and parsing, outputs from PaddleOCR-VL and MinerU2.5-Pro are normalized into HTML and compared with the TEDS score as a reference for annotation quality.

• The title and the model family differ in naming: The paper is titled "PolyOCR-Venus," the GitHub repository is inclusionAI/PolyOCR-Venus, and the model family described throughout is PolyOCR.

Methodology in Plain English

The approach has three parts.

First, data. The team organized training data around six capability dimensions rather than around datasets: text recognition, text localization, document parsing, information extraction, visual text understanding, and OCR reasoning. They curated public and in-house resources for broad coverage, then built a targeted synthesis engine for capabilities that are underrepresented because public data is scarce or annotation is expensive. That engine has three pipelines: (1) scene-text and translation synthesis, where text regions are annotated by two independent tools (PP-OCRv6 and LocateAnything) and cross-validated by spatial overlap and transcription consistency, with disagreements sent to human annotators; (2) structured KIE, document, and table synthesis, where templates are reconstructed into editable HTML/CSS, contents replaced, and pages rendered into new images with aligned annotations, and where PDF pages are clustered, stratified by a layout complexity score, and parsed by two systems whose agreement indicates quality; and (3) reasoning-oriented synthesis, where counting tasks are generated deterministically from verified annotations (so answers inherit exact ground truth) and more complex OCR-VQA reasoning traces are generated by a strong reasoning model (DeepSeek-V4-Pro) and then validated. Three forms of quality control run across pipelines: agreement-based validation, structure-aware validation, and source-grounded validation. Samples that fail are removed rather than repaired.

Second, supervised fine-tuning. The corpus is partitioned into TP, DS, OR, and GA groups with explicit sampling quotas, then trained in three phases of 20 million instances each, shifting weight toward OCR reasoning later in training while keeping general multimodal alignment fixed. Each group is sampled without replacement according to its phase quota and shuffled.

Third, Competence-Guided Policy Optimization. SFT alone imitates rather than directly optimizing verifier-defined outcomes, so the second stage uses GRPO — which assigns scalar rewards and derives token advantages from relative reward within a rollout group — combined with on-policy distillation from a teacher. Instead of always distilling, CGPO estimates teacher and student competence by scoring their generations against a shared reference, calibrates teacher quality and probability of superiority on held-out generations, and uses that sample-level weight to modulate how much distillation flows to each example. Both model variants use the same SFT and CGPO settings; the CGPO stage uses a learning rate of 10⁻⁶, with Qwen3.5-122B-A10B as the teacher for offline competence estimation and token-level OPD.

Why This Matters

Impact on research. The paper argues that neither specialized OCR systems nor general multimodal models alone deliver balanced text-centric visual intelligence, and it offers a recipe for the missing middle: unified task formulation plus adaptive, competence-aware optimization. Its contrast with prior outcome-optimization work is explicit — olmOCR 2 uses verifiable unit-test pass rates as a GRPO reward, Infinity-Parser2 performs joint RL over eight tasks, OvisOCR2 distills an RL-enhanced 4B teacher into a 0.8B student, SCOPE routes verified student trajectories to self-reinforcement or distillation, and Reward-Gated OPD selects trajectories by agreement between verifier feedback and the teacher–student likelihood gap. CGPO instead calibrates teacher quality and probability of superiority on held-out generations. The release of OCRBench v2.1 with manually verified corrections and task-aligned scoring also provides the community a stricter measuring stick.

Real-world applications (as supported by the tasks and benchmarks described):

• Document digitization at scale — full-page parsing of academic papers, annual reports, research reports, and books into Markdown and plain text, preserving hierarchy, lists, tables, and formulas.
• Industrial key information extraction — structured extraction and key–value association from receipts, forms, certificates, and labels, which the paper also evaluates with an in-house KIE Benchmark.
• Bilingual document and scene translation — region-aware Chinese–English and English–Chinese translation that keeps spatial grounding to the source text instance.
• Table and chart understanding in visually rich documents — joint table localization plus HTML structure recovery across fully bordered, borderless, and partially bordered tables, and chart parsing within pages.
• Multilingual document reading — the PolyOCR-2.7B variant targets diverse writing systems and acquisition conditions across 21 language/script groups.

Industry relevance. The work is a technical report from Ant Group, and the benchmark set spans general (OCRBench v2.1), multilingual (CC-OCR, excluding multilingual OCR in the Figure 1 comparison), industrial (in-house KIE), and document (OmniDocBench v1.6) settings, plus MDPBench. That mix — plus the deployment-oriented detail that the OmniDocBench comparison uses the PolyOCR-9B layout-guided pipeline while MDPBench uses PolyOCR-2.7B — signals a focus on production-grade OCR rather than a single academic leaderboard.

Future Directions

• Closing the fundamental tension the paper names: balancing capability breadth, task-specific accuracy, and cross-scenario generalization within one OCR model remains an open challenge, and the paper does not claim to have solved it outright.
• Making outcome supervision work for long structured outputs: the 1,000-character example shows page-level similarity rewards can be identical across rollouts that all get a critical digit wrong, leaving zero advantage. Better reward shaping for fine-grained, high-stakes errors is a clear next step.
• Extending competence routing: CGPO calibrates teacher reliability and the teacher–student gap on held-out generations; how this routing behaves as teachers, students, and task mixtures change (and whether it generalizes beyond OCR) is untested in the available content.
• Data-engine scale-up and verification cost: the engine depends on cross-model agreement, human adjudication for hard scene-text disagreements, and correction passes by Qwen-3.5, so scaling the roughly 60-million-instance corpus while keeping verification rigorous is a continuing question.

Target Audience

Researchers and engineers building OCR or multimodal document-understanding systems; teams working on post-training methods (GRPO, on-policy distillation, multi-task balancing) who want a concrete case study in competence-aware routing; practitioners in document digitization, finance, and enterprise document processing who need structured extraction, table parsing, and multilingual OCR; and benchmark designers interested in the corrections and task-aligned scoring introduced in OCRBench v2.1. Readers looking for a full results table should note that the provided content is truncated and does not report the complete per-benchmark numbers beyond the CC-OCR ablation values of 81.72 and 82.18.

Authors’ abstract

Optical Character Recognition (OCR) is evolving from plain-text transcription toward general visual intelligence, requiring models to recognize, localize, and reason over textual information in complex visual environments. However, existing OCR systems often excel at only some tasks and struggle to balance recognition, parsing, and reasoning across scenarios. In this report, we present PolyOCR, a family of unified OCR foundation models of varying scales. PolyOCR combines a shared instruction-following framework with a large-scale data engine that converts heterogeneous visual resources into quality-verified OCR supervision. We introduce Competence-Guided Policy Optimization, which combines verifier-based Group Relative Policy Optimization with on-policy distillation through sample-wise routing based on teacher reliability and the teacher--student competence gap. We also introduce OCRBench v2.1, our revision of OCRBench v2 with manually verified annotation corrections and task-aligned scoring metrics. Extensive experiments across OCRBench v2.1, CC-OCR, in-house KIE Benchmark, OmniDocBench v1.6 and MDPBench demonstrate that PolyOCR achieves state-of-the-art or highly competitive performance.