WeVisDoc: From Coverage to Capability for Robust End-to-End Document Parsing

AI summary

Overview

Research area: Document AI / end-to-end document parsing with vision-language models, approached from a data-centric (rather than architecture-centric) training perspective.

Technical level: Advanced. The paper formalizes training exposure in terms of loss-bearing target tokens, uses NED/TEDS/CDM error metrics, and builds a diagnostic protocol on top of visual–structural clustering, so familiarity with autoregressive multimodal training and document benchmarks is assumed.

Scope: The paper proposes a two-stage framework — broad data coverage (Stage I) followed by capability diagnosis and targeted refinement (Stage II) — and releases 2B and 4B parsers that rank first among compared end-to-end systems on OmniDocBench v1.6 and three PureDocBench robustness tracks.

What This Paper Is About

Document parsers must turn page images into structured text while preserving content, element types, reading order and layout — but training corpora are skewed toward common document types and clean, born-digital pages. The authors argue that simply expanding data coverage does not tell you which of a parser's remaining weaknesses still need data, so they build a framework that first covers the space broadly, then diagnoses the trained model's residual errors and spends a fixed training budget where those errors concentrate. The goal is robust end-to-end parsing across languages, layouts and degraded acquisition conditions (scans, photos, photocopies, screen recaptures).

Key Contributions

1. A two-stage data-centric framework (WeVisDoc) that separates expanding empirical support (Stage I, roughly 40 million records) from increasing exposure to existing or newly validated support (Stage II, roughly 5 million records), each with its own target-token budget.

2. A Stage I coverage pipeline combining open-source datasets, in-house collections, targeted web/PDF crawling and generated documents; converting source annotations into unified page/region/component targets; labeling uncertain pages with three expert parsers (MinerU2.5-Pro, PaddleOCR-VL-1.6, dots.mocr) under normalized-edit-distance agreement checks; compiling executable page programs from semantic HTML for exact image–target pairs; and applying source-conditioned appearance degradation (paper, copying, transmission, physical acquisition).

3. A Stage II refinement protocol built around a held-out allocation probe disjoint from training, hard-example mining and final evaluation, combining document-level NED with component-level text/table/formula errors (NED, 1−TEDS, 1−CDM), a verification/audit step that separates annotation errors, decoding failures and unreadable content from genuine model errors, and visual–structural clustering (frozen SigLIP2 embeddings + PCA concatenated with structural metadata) to produce stabilized residual priorities per cluster.

4. An explicit token-accounting formalism and released models: loss normalized per supervised target token, source balancing with an exponent ρ, and conversion of desired token shares into record-sampling probabilities — instantiated from Qwen3-VL-Instruct at 2B and 4B scales, with checkpoints released (WeVisDoc-2B and WeVisDoc-4B) alongside a project page and GitHub repository.

Main Findings

• Top scores on OmniDocBench v1.6: WeVisDoc-4B achieves an Overall score of 95.38, and WeVisDoc-2B achieves 95.06.
• Top average across PureDocBench tracks: the 4B model reaches a mean Overall score of 75.54 across the three PureDocBench tracks; the 2B model reaches 73.86 (reported as Avg₃).
• First place in all four settings: the 4B model ranks first among the compared end-to-end parsers on OmniDocBench v1.6 and on the three PureDocBench tracks (Clean, Digital, Real).
• Half-size model stays competitive: the 2B model is described as delivering top-tier end-to-end performance at half the parameter count of the 4B model.
• Stage II improves both scales on both benchmarks: the complete Stage II protocol raises Overall scores for the 2B and 4B models on both OmniDocBench v1.6 and PureDocBench, relative to Stage I.
• Degraded pages benefit most: gains are larger on the degraded PureDocBench tracks than on clean pages, including a 4.03-point gain for the 4B model on the Real Degraded track.
• Comparison set: Figure 1 compares the two WeVisDoc models against five representative end-to-end systems — HunyuanOCR-1.5, Unlimited-OCR, FD-RL, Logics-Parsing-v2 and Qianfan-OCR.
• Stage I data composition: before training-specific sampling, roughly 40% of records provide full-page supervision, 25% text or mixed regions, 25% isolated tables and 10% isolated formulas.
• Caveat stated by the authors: because Stage II combines several interventions, the stage-wise results do not identify the effect of residual-aware allocation in isolation.
• Reported language and source coverage: Simplified Chinese, English, Traditional Chinese and mixed-language pages; acquisition coverage spans born-digital pages, scans, photographs, screenshots and screen recaptures.

Methodology in Plain English

Stage I — build broad coverage. The authors assemble roughly 40 million training records from public datasets, in-house production documents, targeted crawling (education, newspapers, handwriting, mixed-language pages) and synthetic pages. Every source annotation is converted into a unified target format: Markdown for text and reading order, HTML for tables, LaTeX for formulas. Records can be full pages, text crops, mixed-content regions, isolated tables or isolated formulas. Where source labels are missing or inconsistent, three expert parsers label the page; if all three agree closely (measured by character-level normalized edit distance), the target can be used directly, if two agree it goes to image-based review, and if all three disagree the record is withheld from ordinary training and reviewed by a stronger multimodal model plus humans. Synthetic pages are created by writing "page programs" in semantic HTML: one compilation path renders the image and node geometry, another traverses the same tree to emit the labels, so image and target are exactly aligned. Finally, appearance degradations (paper aging and texture, printing/photocopy/scan artifacts, compression, perspective and curvature, uneven illumination, screen recapture) are applied along source-appropriate acquisition paths, and a transformed page keeps its original target only if all supervised content stays visible and readable.

Accounting for training exposure. The authors do not count records; they count loss-bearing target tokens. Long pages contribute many more supervised tokens per sampled record than short crops, so they define a training objective normalized by supervised token length, balance data sources by their available token counts with an exponent ρ (ρ = 1 follows token counts, ρ = 0 gives equal shares), and convert desired token shares into record-sampling probabilities with a length correction.

Stage II — diagnose and refine. A frozen Stage I checkpoint is evaluated on a held-out probe that is kept out of training, hard-example mining and final benchmark evaluation, so it can be reused for both 2B and 4B scales. Errors are measured at page level with NED over the full serialized output, and at component level separately for text (NED), tables (1 − TEDS) and formulas (1 − CDM); figures and charts are excluded from the component metrics. High-error records are then audited to check whether the error is actually the model's fault — targets may be wrong, outputs may fail to stop, or content may be unreadable. A record enters the verified hard set only if the mining checkpoint still makes substantial errors against the final reliable target. Separately, Stage I records are clustered by visual structure and composition (frozen SigLIP2 embedding reduced by PCA, concatenated with normalized metadata about element composition, density, language, layout, acquisition condition and target length), and cluster-level residuals are computed on reliable probe records and shrunk toward the pooled mean. Together, coverage priorities (small or diffuse clusters) and residual priorities decide whether to add validated data or increase exposure to existing records, and the revised mixture is trained under the fixed Stage II token budget. Stage II freezes the visual encoder and updates the language model; the backbone architecture, output serialization and autoregressive objective are kept fixed within each scale.

Why This Matters

The paper reframes document-parser improvement as a data-allocation problem rather than a purely architectural one. It gives a repeatable procedure for deciding whether a failure is a coverage gap, a labeling problem or a decoding problem — a distinction that matters because adding nearby examples cannot fix a wrong target or an unreadable crop. This is directly useful for anyone building or auditing large multimodal training corpora, and the released checkpoints make the recipe testable at two scales.

Real-world applications:

• Digitizing scanned contracts, receipts, forms and invoices where the input is photographed under poor lighting or at an angle.
• Converting academic papers and technical reports into machine-readable text, tables and formulas for retrieval or search.
• Processing multilingual enterprise documents that mix Simplified Chinese, English, Traditional Chinese and mixed-language content.
• Handling historical or archival material affected by aging, stains and ink fading where the same page must be parsed from both clean and degraded views.

Industry relevance: the work comes from WeChat Vision, Tencent Inc., targets production-scale document pipelines, and ships public artifacts (WeVisDoc-2B and WeVisDoc-4B on Hugging Face, a GitHub repository and a project page). The token-budget framing also maps cleanly onto industrial training constraints, where compute and supervised-token budgets are the binding resource rather than the amount of raw data on disk.

Future Directions

• Isolate the interventions. The authors explicitly note that Stage II bundles several changes, so an ablation separating residual-aware allocation from validated additions and hard-example oversampling is the natural next step.
• Extend component-level diagnosis. Figures and charts are currently excluded from the component error metrics (Equation 7), leaving a class of document elements diagnosed only through the whole-page score.
• Scale and deployment analysis. Results are reported only at 2B and 4B; behavior at other scales, and inference cost or latency, are not reported in the available text.
• Broaden language and condition coverage beyond the reported set. The reported language coverage is Simplified Chinese, English, Traditional Chinese and mixed-language pages, so generalization to other scripts remains an open question.
• Reproducibility of the diagnostic machinery. Details that would let others replicate the exact refinement decisions — such as the support coefficient used for residual shrinkage, the annotation-agreement threshold and the number of clusters — are not reported in the available text, which was truncated mid-equation in Section 3.3.2.

Target Audience

Researchers and engineers working on document AI, OCR and multimodal large language models, especially those responsible for training-data curation, mixture design or robustness evaluation. It is also relevant to practitioners deploying page-parsing systems in production who need parsers that hold up on scanned, photographed and recaptured documents, and to benchmark designers interested in how diagnostic probes can be kept separate from final evaluation.

Authors’ abstract

Document parsing converts document images into structured content and requires reliable performance across diverse layouts and acquisition conditions. Yet training corpora are biased toward common document types and clean digital pages, while expanding coverage alone does not specify how to address a parser's remaining weaknesses. We present WeVisDoc, a two-stage data-centric framework for robust end-to-end document parsing. Stage I broadens semantic, structural, and appearance coverage through heterogeneous data and structure-preserving degradation synthesis. Stage II uses a held-out probe to measure the Stage I parser's residual errors within fixed visual-structural clusters. These diagnostics guide targeted data construction and reallocation of the target-token budget. WeVisDoc-4B achieves an Overall score of 95.38 on OmniDocBench v1.6 and a mean Overall score of 75.54 across the three PureDocBench tracks, ranking first among the compared end-to-end parsers in all four settings. Compared with Stage I, Stage II improves Overall scores for the 2B and 4B models on both benchmarks, with larger gains on the degraded PureDocBench tracks, including a 4.03-point gain for the 4B model on the Real Degraded track.