TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition

AI summary

Overview

Research area: Computer vision and document AI — specifically table recognition (TR), a core component of document parsing, advanced through self-supervised fine-tuning of vision-language models.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (specifically GRPO), vision-language model architectures, attention mechanisms, and document-structure metrics such as TEDS. The conceptual framing is accessible, but the method details are not beginner material.

Scope: The paper proposes a self-supervised fine-tuning framework that lets a pretrained VLM learn table recognition from unlabeled table images by using table question-answering as a proxy reward, and demonstrates a 3B-parameter model beating far larger proprietary and open systems.

What This Paper Is About

Table recognition — turning a photo or scan of a table into structured markup like HTML or Markdown — normally requires large amounts of expensively hand-labeled data, which is why open-source models trail proprietary ones like Gemini 2.5 Pro. This paper asks whether unlabeled table images scraped from the wild can substitute for those labels. The authors answer yes, by building a training loop where the model is rewarded for producing recognized tables that let a separate language model correctly answer automatically generated questions about the table's contents.

Key Contributions

1. TRivia framework: A self-supervised fine-tuning method that trains VLMs on unlabeled table images using Group Relative Policy Optimization (GRPO), where the reward comes from a table question-answering proxy task rather than ground-truth markup.

2. Response-consistency sampling: A data selection strategy that measures pairwise TEDS similarity among multiple recognition outputs for each image, then preferentially trains on images that elicit inconsistent (i.e., uncertain, informative) responses.

3. Attention-guided QA generation: A mechanism that uses VLM attention distributions to determine which table regions each generated QA pair depends on, then greedily selects QA sets with minimal visual overlap — ensuring broad table coverage without redundant questions, cross-checked for validity by a separate model.

4. TRivia-3B: An open-source 3B-parameter model fine-tuned from Qwen2.5-VL-3B that sets a new state of the art on three table recognition benchmarks, outperforming both specialist document-parsing models and proprietary general-purpose models several orders of magnitude larger.

Main Findings

• State-of-the-art at small scale: TRivia-3B achieves 89.88 overall TEDS and 93.60 S-TEDS, beating MinerU2.5 (86.82/90.81), Gemini 2.5 Pro (88.93/91.23), PaddleOCR-VL (83.36/87.77), and GPT-5 (78.30/86.21) across OmniDocBench, CC-OCR, and OCRBench v2.

• Distillation from the teacher would have failed: Using the QA-generation model (Qwen2.5-VL-72B) to produce HTML pseudo-labels and fine-tuning on them caused an average drop of 8.37 TEDS with SFT and 4.92 TEDS with GRPO. The QA-based reward sidesteps the teacher's inability to produce correct markup.

• Response-consistency sampling is a major accelerator: Compared to random sample selection, it raised TEDS from 52.0 to 63.5 and sped up convergence, confirming that high-uncertainty samples are the most valuable for GRPO.

• Illegal-sample filtering stabilizes training: Discarding invalid recognition outputs (which would otherwise receive zero reward and distort relative advantages) cut convergence steps by roughly 25% and improved final performance by about 3 TEDS.

• Attention-guided QA diversity matters: Ablating it causes a large performance drop, with the model specifically weakening on structurally complex or visually ambiguous tables.

• TRivia works as a data annotator: A Stage-2 model fine-tuned on TRivia-3B-generated pseudo-labels reached 89.99 overall TEDS, essentially matching TRivia-3B (89.88) — and slightly exceeding it on CC-OCR, the hardest benchmark.

• Generalizes to other OCR tasks: In a preliminary joint table recognition and key information extraction experiment, adding KIE QA rewards improved KIE performance from 76.08 to 91.01 while holding TR performance nearly constant.

Methodology in Plain English

The approach has three stages.

First, a general-purpose VLM (Qwen2.5-VL-3B) is warmed up on roughly 700K synthetic tables rendered into OTSL, a compact table markup format that encodes cell adjacency rather than explicit row/column spans. Only the language component is trained here; the visual encoder stays frozen.

Second, the model is fine-tuned on about 50K real-world table images with all parameters unfrozen, establishing the ceiling reachable through ordinary supervised learning.

Third, and this is the novel part, the model trains on roughly 50K unlabeled table images assembled from web PDFs. Since no labels exist, the supervision comes from questions: for each table image, a large VLM generates many candidate question-answer pairs. Each pair is validated by checking that another model can answer it with the image but not without it — guaranteeing the question genuinely requires looking at the table. Attention maps identify which image regions each answer relies on, and questions with heavily overlapping evidence are pruned, leaving roughly 28 diverse questions per image.

During training, the TR model produces several recognized-table outputs per image. Each output is fed alongside a question to a language model (Qwen3-8B), which attempts to answer from the recognized structure. The reward is the F1 score between that answer and the known answer, averaged over all questions. Standard GRPO then updates the TR model to maximize this reward — so the model learns to produce recognized tables that actually preserve the information needed to answer questions about them. Invalid outputs are filtered out before computing advantages to keep training stable. The sample selection, meanwhile, favors images where the model's own outputs disagree with each other, since those carry the most learning signal.

Why This Matters

Impact on research: The paper demonstrates that reinforcement learning with automatically verifiable proxy tasks can break the labeled-data bottleneck in document AI. It also shows something counterintuitive and important: a teacher model that cannot produce good direct annotations can still supply useful supervisory signal when the supervision is reframed as questions rather than labels. This reframing is generalizable to other structured-prediction problems.

Real-world applications:

• Privacy-sensitive document processing: An open, 3B-parameter model that runs offline lets banks, hospitals, and law firms parse tables in confidential documents without sending them to commercial APIs.
• Automated data annotation pipelines: TRivia-3B can generate high-fidelity pseudo-labels for unlabeled table corpora, replacing costly and slow manual annotation.
• Retrieval-augmented generation over enterprise documents: Accurate table parsing is a prerequisite for making financial reports, invoices, and scientific tables searchable by downstream LLM systems.
• Low-resource deployment: A compact model that beats much larger systems can run on modest hardware, widening access for smaller organizations and researchers.

Industry relevance: Document AI vendors currently compete on annotation volume and proprietary teacher models. This work suggests a cheaper path — collect unlabeled documents, generate verifiable questions, and train with RL — that undercuts the data-acquisition advantage of large players and reduces reliance on commercial APIs whose terms may forbid distillation.

Future Directions

• Online sample selection: Response-consistency sampling currently runs offline for computational reasons. Whether periodic online re-evaluation would yield further gains, and at what cost, remains open.

• Scaling beyond table recognition: The preliminary TR+KIE result suggests QA-based rewards transfer to other document tasks (form understanding, chart extraction, layout analysis), but this has only been tested on a small mixed dataset.

• Richer reward design: The current reward is answer F1 averaged across questions. Whether weighting questions by difficulty, structural importance, or uncertainty improves learning is unexplored.

• Data-domain matching: The pipeline depends on a QA-generation model and a validation model. How sensitive performance is to the choice of these components, and whether a single model could serve both roles, is not characterized.

Target Audience

Researchers and engineers working on document AI, vision-language models, or reinforcement learning from verifiable rewards. It is most valuable for those building production document-parsing systems who face annotation cost or privacy constraints, and for RL researchers interested in how proxy tasks and attention-based data curation can substitute for labeled supervision. Readers without background in VLM training or GRPO will find the framing clear but the methodology dense.

Authors’ abstract

Table recognition (TR) aims to transform table images into semi-structured representations such as HTML or Markdown. As a core component of document parsing, TR has long relied on supervised learning, with recent efforts dominated by fine-tuning vision-language models (VLMs) using labeled data. While VLMs have brought TR to the next level, pushing performance further demands large-scale labeled data that is costly to obtain. Consequently, although proprietary models have continuously pushed the performance boundary, open-source models, often trained with limited resources and, in practice, the only viable option for many due to privacy regulations, still lag far behind. To bridge this gap, we introduce TRivia, a self-supervised fine-tuning method that enables pretrained VLMs to learn TR directly from unlabeled table images in the wild. Built upon Group Relative Policy Optimization, TRivia automatically identifies unlabeled samples that most effectively facilitate learning and eliminates the need for human annotations through a question-answering-based reward mechanism. An attention-guided module generates diverse questions for each table image, and the ability to interpret the recognition results and answer them correctly provides feedback to optimize the TR model. This closed-loop process allows the TR model to autonomously learn to recognize, structure, and reason over tables without labeled data. Leveraging this pipeline, we present TRivia-3B, an open-sourced, compact, and state-of-the-art TR model that surpasses existing systems (e.g., Gemini 2.5 Pro, MinerU2.5) on three popular benchmarks. Model and code are released at: https://github.com/HKU-TASR/TRivia