Docs2Synth: A Synthetic Data Trained Retriever Framework for Scanned Visually Rich Documents Understanding

AI summary

Overview

Research area: Multimodal document understanding — specifically Visually Rich Document Understanding (VRDU) for scanned documents in regulated, privacy-sensitive domains, combining synthetic data generation, vision-language retrieval, and multimodal large language model (MLLM) inference.

Technical level: Advanced. The paper assumes familiarity with MLLMs, layout-aware Vision-Language Pre-trained Models (VLPMs), key information extraction (KIE), and retrieval-augmented generation.

Scope: The paper introduces Docs2Synth, a synthetic-supervision framework that generates verified QA pairs from raw document collections, tunes a lightweight visual retriever on them, and uses that retriever in an iterative retrieval–generation loop with an MLLM to ground responses without human annotations.

What This Paper Is About

Scanned documents in regulated domains contain sensitive, evolving, domain-specific knowledge, but there are no manual annotations available to adapt models to them, and pretrained models struggle to stay current with domain-specific facts. Generative MLLMs show strong zero-shot ability but hallucinate and lack domain grounding, while discriminative VLPMs ground well but need costly annotation. Docs2Synth's goal is to let a system adapt to a new, private document collection automatically — generating its own supervision and using retrieval to keep MLLM answers anchored to document evidence.

Key Contributions

1. Docs2Synth, a framework that automatically digests a new document collection, generates high-quality synthetic QA pairs via an agent-based system, and trains a domain-adapted visual retriever to support retrieval-guided iterative MLLM inference for private and low-resource domains.
2. An iterative inference procedure in which a tuned retriever proposes top-k entity indices, their bounding boxes are highlighted in red on the document image, and an MLLM re-answers using the retrieved content, with that new answer feeding back to refine the retriever's next top-k selection.
3. Docs2Synth as an open-source, easy-to-use Python package that automates data processing, synthetic corpus creation, retriever training, and retrieval-enhanced inference, configured through a single config.yml and runnable with a unified command (e.g., docs2synth run).
4. A modular, production-ready implementation supporting multiple OCR backends (Docling, PaddleOCR), a unified agent wrapper across proprietary APIs (OpenAI, Gemini, QwenVL) and local inference providers (Ollama, vLLM, HuggingFace), an optional lightweight human annotation interface, and a side-by-side interface for comparing simple RAG baselines against the iterative strategy.

Main Findings

• Synthetic-data-tuned retriever beats the best MLLM: As shown in Figure 3(a), the system equipped with a synthetic-data-tuned retriever consistently outperforms the best-performing MLLM across various scenarios. The reported improvement is particularly notable on the FormNLU-printed (F-P) set.
• Printed-domain advantage attributed to sample fidelity: The authors attribute the notable FormNLU-printed gain to higher fidelity of the generated synthetic samples, which better capture the visual and structural characteristics of the printed domain.
• Retriever quality is solid but not perfect: Figure 3(b) reports that the tuned retriever achieves solid and reliable performance, though occasional retrieval errors remain. The consistent improvement is presented as evidence of the value of synthetic data for retrieval quality and downstream accuracy.
• Iterative refinement helps both components: The qualitative case study in Figure 3(c) shows an MLLM initially producing an incomplete student name; after the retriever supplies more focused and relevant content as supplementary context, the MLLM generates the correct prediction. The authors describe this as bidirectional information flow producing mutual performance improvements.
• No manual annotations required: The paper states that experiments on multiple VRDU benchmarks show substantial enhancement of grounding and domain generalization without human annotations.
• Feasible on limited on-premise hardware: The authors report that with proprietary LLM APIs only retriever training requires moderate resources and can run on CPU; fully local deployment requires a GPU, and their experiments on an RTX 5090 (32 GB) show the system remains feasible with limited on-premise GPU capacity.
• Specific numeric results are not reported in the text: The provided paper content refers to quantitative outcomes only through Figure 3(a)–(c); no numeric scores, percentages, or per-dataset metric values appear in the text, so none can be stated here.

Methodology in Plain English

The framework has four stages.

1. Document processing. Each document image in a new collection is run through an OCR tool to extract semantic entities, where each entity is a pair of text content and its bounding box. XY-cut reading-order methods aggregate the full document text and its bounding box list. These entities pass through pretrained language and vision backbones to produce textual and visual features.

2. Synthetic QA generation. Instead of hand-labeling, an MLLM is prompted to generate a question for each extracted entity such that the entity's text content is the answer — for example, given "Ordinary Shares," it might ask "What is the type of shares listed in the document?" The same MLLM then acts as a verifier, checking the relevance and clarity of the question and confirming that the entity text is a valid answer. Each document yields a question set and corresponding answer set, with the number of pairs as an adjustable parameter.

3. Retriever tuning. A layout-aware pretrained VRDU model is fine-tuned on the synthetic QA pairs. Each training sample combines the question, document image, full text, bounding boxes, entity-level textual and visual features, and an initial answer prediction from an MLLM. The retriever learns a scoring function that predicts the index of the correct answer entity among all candidates, trained by minimizing cross-entropy loss against the ground-truth index.

4. Iterative inference. At each step the retriever predicts a set of top-k entity indices (chosen from the retriever's logits). Their text contents form the retrieved content set, and their bounding boxes are highlighted in red on the document image. The MLLM then receives the question, the modified image, the full text, and the retrieved content, and produces an updated answer. That answer is fed back into the retriever to refine the next top-k selection, and the loop continues. Both k and the number of iterations are user-specified.

Evaluation uses three benchmark datasets: Form-NLU (financial forms with printed and handwritten formats, 12 key fields including "Substantial Holder Name"), CORD (receipts, fine-grained fields such as "store name"), and Ephoie (scanned Chinese exam headers with handwritten elements, fields like "Score" and "Student Name"). Baselines are open-source MLLMs (Qwen2-VL, Idefics2, InternVL2) and proprietary MLLMs (GPT-4o, Gemini 1.5), all evaluated under their default HuggingFace inference settings. PaddleOCR is the default OCR for extracting text and bounding boxes. For optional warmer tuning, the batch size is 16, the learning rate is 2 × 10⁻⁵, and AdamW is the optimizer. All tools and checkpoints used to produce the reported results are released.

Why This Matters

Impact on research. The paper frames a specific gap: whether fine-tuning lightweight discriminative VLPMs on large-scale synthetic domain-specific corpora can handle newly scanned documents without manual annotations, and whether such fine-tuned models can support or enhance stronger MLLMs at inference time. It offers an answer in the form of a retrieval-guided paradigm where the tuned retriever grounds MLLM responses, positioning synthetic data as a route to domain adaptation where annotations are unavailable.

Real-world applications:

• Financial and regulatory document review, using Form-NLU-style forms with printed and handwritten fields such as "Substantial Holder Name."
• Receipt and expense processing at CORD-style fine granularity, for example extracting "store name."
• Handwritten exam header processing of the Ephoie type, extracting fields like "Score" and "Student Name."
• Privacy-sensitive enterprise deployments, where the unified agent wrapper allows local inference providers (Ollama, vLLM, HuggingFace) instead of proprietary APIs, and only retriever training needs moderate resources and can run on CPU.

Industry relevance. Docs2Synth is explicitly packaged to bridge research prototypes and industrial deployment: a modular, extendable, production-ready Python package with pluggable OCR backends, a single configuration file, a one-command workflow, one-command retriever training/validation/packaging, and a side-by-side interface for comparing retrieval strategies. The paper reports that the full system remains feasible on a single RTX 5090 (32 GB), which matters for organizations that cannot send sensitive documents to external APIs.

Future Directions

• Improving OCR and document parsing. The authors state that with ongoing advances in document parsing and data synthesis technologies, they expect the retriever to deliver even greater benefits for future document understanding tasks.
• Reducing residual retrieval errors. The paper acknowledges that occasional retrieval errors remain in the tuned retriever, leaving headroom in retrieval quality and downstream accuracy.
• Explaining and exploiting domain variation. The FormNLU-printed set improved most, which the authors link to higher-fidelity synthetic samples capturing printed visual and structural characteristics; whether the pipeline needs different synthesis strategies for handwritten or low-fidelity scans is left open.
• Optimizing the human-in-the-loop path. The optional lightweight human annotation interface for refining generation is described but its effect on reliability in private domains is not quantified in the provided text.
• Choosing inference settings. Since k and the number of iterations are user-specified, how to select them per domain or per query type is an open practical question.

Target Audience

Researchers and engineers working on document AI, multimodal retrieval-augmented generation, and key information extraction; practitioners in finance, retail, and government who need to extract structured fields from scanned, handwritten, or otherwise messy documents without labeled data; and developers looking for a deployable Python package that wires together OCR, synthetic QA generation, retriever tuning, and MLLM inference in privacy-sensitive settings. Readers should be comfortable with MLLM and retrieval terminology; the paper is not an introductory treatment.

Authors’ abstract

Document understanding (VRDU) in regulated domains is particularly challenging, since scanned documents often contain sensitive, evolving, and domain specific knowledge. This leads to two major challenges: the lack of manual annotations for model adaptation and the difficulty for pretrained models to stay up-to-date with domain-specific facts. While Multimodal Large Language Models (MLLMs) show strong zero-shot abilities, they still suffer from hallucination and limited domain grounding. In contrast, discriminative Vision-Language Pre-trained Models (VLPMs) provide reliable grounding but require costly annotations to cover new domains. We introduce Docs2Synth, a synthetic-supervision framework that enables retrieval-guided inference for private and low-resource domains. Docs2Synth automatically processes raw document collections, generates and verifies diverse QA pairs via an agent-based system, and trains a lightweight visual retriever to extract domain-relevant evidence. During inference, the retriever collaborates with an MLLM through an iterative retrieval--generation loop, reducing hallucination and improving response consistency. We further deliver Docs2Synth as an easy-to-use Python package, enabling plug-and-play deployment across diverse real-world scenarios. Experiments on multiple VRDU benchmarks show that Docs2Synth substantially enhances grounding and domain generalization without requiring human annotations.