DR Pixel: Page-Native Deep Research with Vision-Language Models

AI summary

Overview

Research area: Multimodal deep-research (DR) agents, visual document retrieval, and vision-language model (VLM) post-training — positioned at the intersection of computer vision, information retrieval, and agentic AI.

Technical level: Intermediate. The paper assumes familiarity with retrieval-augmented generation, ReAct-style agent loops, and supervised fine-tuning, but the core idea (stop OCR-ing documents, look at the pages instead) is easy to grasp.

Scope (one sentence): The paper asks whether deep-research agents can retrieve and reason directly over rendered document page images rather than OCR-extracted text, builds an open-source agent (DR Pixel) and benchmark (DRPixelBench) to test this, and introduces a credit-assignment training method to make an open-weight VLM competitive with proprietary models.

What This Paper Is About

Complex documents like scientific papers carry essential evidence in tables, figures, charts, and equations — content that gets flattened, distorted, or dropped entirely when a PDF is linearized into plain text for a conventional "OCR-then-retrieve" pipeline. The authors ask whether a deep-research agent can instead be page-native: keep every page as a rendered image, retrieve over those images with a visual retriever, and let a VLM reason directly over the retrieved page images across many turns of search. To answer this, they build DR Pixel (an agent operating over 1.06M rendered pages), DRPixelBench (500 verifiable questions requiring multi-step retrieval plus visual reasoning), and a training recipe called counterfactual credit reweighting that makes an open-weight 27B model competitive with proprietary systems.

Key Contributions

1. DR Pixel, a page-native deep-research agent. The first open-source end-to-end multimodal DR agent that uses rendered page images as its sole retrieval and reasoning unit, operating over a corpus of 1.06M pages. It preserves the standard ReAct loop but swaps the text retriever and text corpus for a VLM-based page-image embedder (Qwen3-VL-Embedding-8B) and a corpus of page images, so evidence is never converted to plain text at any point in the trajectory.

2. DRPixelBench. A 500-question benchmark over 99,055 papers (1,062,153 pages) built specifically to require agentic multi-step search, multimodal evidence, non-templated questions, and short verifiable answers — a combination the authors show no existing benchmark satisfies. Questions are generated from enriched abstracts (abstract plus atomic facts extracted from the full paper) so that the question is driven by curiosity rather than by evidence already in view, then filtered through four automatic checks and human rewriting.

3. Counterfactual credit reweighting. A fine-grained credit-assignment method for agentic SFT that weights each reasoning and search step by its counterfactual contribution to the gold answer, estimated by removing the step from a teacher trajectory and measuring the resulting change in gold-answer log-likelihood under a frozen base policy.

4. A systematic comparison of page-native versus OCR-based pipelines across three model families, plus ablations on corpus size, page resolution, and training-data modality.

Main Findings

• Page-native DR wins even with no training. Against the stronger text+figure OCR baseline, DR Pixel improves accuracy by 5.6, 5.1, and 5.2 points for GPT-5.4 (46.3 → 51.9), Gemini-3.5-Flash (41.3 → 46.4), and Qwen3.6-27B (34.9 → 40.1) respectively, while using 20% to 30% fewer retrieval calls.

• The gain concentrates exactly where the pipeline is lossy. GPT-5.4's accuracy broken down by evidence source shows DR Pixel highest on questions requiring tabular and figure evidence, and on par for text- or equation-grounded questions.

• Adding an image-search tool to a text pipeline barely helps. The text+figure baseline yields only marginal gains because agents rarely invoke the figure-search tool despite explicit prompt instructions.

• Open-weight models lag in both retrieval and reasoning. Under the DR Pixel setting, Qwen3.6-27B trails GPT-5.4 by 11.8 points (40.1 vs 51.9). Decomposition: it retrieves the gold page 9.4 points less often, 5.3% of its queries duplicate an earlier one, and 21.6% of its trajectories contain at least one duplicate. When the gold page comes from the latest search the reasoning gap is only 5.1 points, but it widens to 12.3 points when the page was retrieved in an earlier search.

• Training closes the gap. With counterfactual credit reweighting, Qwen3.6-27B goes from 40.1 to 52.1 accuracy on DRPixelBench — a 12.0-point (30% relative) gain that slightly surpasses GPT-5.4's 51.9 — and page-hit rate rises by roughly 15 points (36.3 → 52.0). Gains are attributed mainly to retrieval: the trained model retrieves the gold page 6.3 points more often than GPT-5.4 while using 8.9 searches per question versus GPT-5.4's 9.2.

• Credit reweighting beats vanilla SFT by suppressing wasted searches. Vanilla SFT reaches 49.0 accuracy with a slightly higher page-hit rate (53.6 vs 52.0) but requires 57% more searches; it exhausts the 20-call budget on 35.8% of questions versus 9.8% for DR Pixel Trained (GPT-5.4: 17.8%; untrained backbone: 19.2%).

• Gains transfer to external scientific benchmarks. On IRPAPERS, M3SciQA, and AirQA, the trained model improves over the untrained backbone by 9.6 to 14.4 points (IRPAPERS 65.4 → 79.4, M3SciQA 49.8 → 64.2, AirQA 42.7 → 52.3), surpasses GPT-5.4 in document retrieval on two of the three (95.6 vs 92.2 on IRPAPERS; 89.7 vs 87.9 on AirQA), and adapts search effort to difficulty — searching less on IRPAPERS and AirQA but more on M3SciQA, where the backbone's retrieval gap to GPT-5.4 was largest (15.0 points).

• Out-of-distribution generalization to non-scientific documents. On ViDoRe-V3, post-training improves both page-based and text-based retrieval by 4.1 points, despite training only on page-native trajectories. With page retrieval the trained model reaches 70.8 global versus a published page Oracle of 72.6 and text Oracle of 70.6. The two modalities stay close because only 1.7% of ViDoRe-V3 questions genuinely require visual content.

• Robustness to corpus growth. Scaling the corpus from 20,000 to 99,000 papers, DR Pixel Trained degrades by only 8.0 page-hit points versus 12.5 for GPT-5.4, widening its retrieval advantage from 1.8 points to 6.3 points.

• Resolution is a cheap knob. Capping pages at 1.05 megapixels (roughly 50% of native resolution) cuts input tokens by 30% to 48% for a 1.0-to-3.8-point accuracy drop, and page retrieval still beats OCR text retrieval by 2.7 to 7.4 points while using 22% to 33% fewer input tokens.

• Visual training data matters more than text training data. Fine-tuning on 2,000 Text trajectories instead of 2,000 Visual ones yields 46.0 vs 50.2 accuracy, with the text-trained model searching less and finding the gold page less often. Visual fine-tuning stays within 1.4 points of text fine-tuning on Text questions, while text fine-tuning trails by 6 to 11 points on Visual questions.

Methodology in Plain English

Building the corpus. Every PDF is rendered page by page at 150 dpi, so a "document" is now just a stack of images. Each page image is encoded into a fixed-size vector with a VLM embedding model, which makes it searchable by a text query. Nothing is parsed, OCR'd, or linearized at any point.

Running the agent. The agent follows the standard think-act-observe loop: at each turn it produces a thought and either a search query or a final answer. A search returns the top 3 page images, which are dropped straight into the agent's context as images and stay there as visual evidence for the rest of the trajectory. The agent gets at most 20 searches. Nothing is ever converted back to text.

Building the benchmark. Questions are generated from a paper's abstract — but first the abstract is enriched with atomic facts extracted from the full paper (a comparison in a table, a trend in a figure), so the resulting question is inspired by the paper's headline contribution without the answer already being visible in the prompt. The generator then locates the supporting evidence in the full paper and produces a short answer, and each question is labeled Text or Visual depending on whether a table, figure, or equation is required. Automatic filters remove coined names, dangling references ("this method"), internal codes, multi-part questions, malformed answers, and questions answerable from parametric knowledge; human experts then rewrite for naturalness.

Comparing pipelines fairly. From the same 99,055 papers the authors build three corpora — OCR text chunks of 256 words (k=5), extracted figure crops (k=3), and page images (k=3) — choosing k in each setting to roughly match the agent's retrieved-token budget. Parsing uses MinerU-2.5-Pro, preserving tables as HTML and equations as LaTeX. Answers are graded by GPT-5.4-mini under a strict prompt with majority vote over three independent judgments.

Training. Teacher trajectories are sampled from GPT-5.4 on automatically generated training questions (no human rewriting or verification). The problem: these traces are noisy, full of redundant or failed searches, and naive imitation teaches the model to burn its search budget. The fix is to score each step. For a search step, the credit is how much the returned results raise the log-likelihood the frozen base model assigns to the gold answer; for a reasoning step, it is how much removing that thought from the final history lowers that log-likelihood. Non-positive credits get weight zero; positive credits are converted to rank-based weights scaled so the average positive weight stays near 1, keeping the loss on the same scale as standard SFT. Reasoning and action weights are further coupled — a reasoning step is masked if its action was masked or its likelihood gain falls below the median. The final objective is a token-level weighted loss with an extra constant weight (λ = 1.5) on the answer tokens. Training is LoRA on Qwen3.6-27B, one epoch over about 2,500 trajectories, learning rate 1e-4, effective batch size 16, on four H200 GPUs.

Why This Matters

The paper argues that the default "OCR-then-retrieve" stage in document AI is not merely unnecessary but actively suboptimal — it discards precisely the tables, charts, and equations that make specialized documents valuable, and its parsing errors propagate silently into retrieval and reasoning. Replacing it with page rendering removes a brittle, format-specific, high-maintenance component from the stack, and the results suggest you also get better answers with fewer tool calls and fewer tokens.

Real-world applications:

• Financial research. Earnings reports, 10-Ks, and filings where the signal lives in tables and charts; the authors specifically cite financial documents as a target for layout-rich private corpora.
• Legal document review. Contracts and case files with layout-dependent structure where maintaining bespoke OCR/parsing pipelines is costly and fragile.
• Scientific literature assistants. Literature-review agents that must read experimental results out of figures and tables, not just prose — the setting DRPixelBench is built around.
• Enterprise document QA over private corpora. Because the method requires only rendering rather than per-format parsing, a new document type needs no new extraction pipeline.

Industry relevance: The most immediately transferable artifact is arguably not the agent but the training recipe. Counterfactual credit reweighting is a general-purpose fix for the well-known problem that agentic SFT traces are noisy, and it requires no reward model, no RL infrastructure, and no environment rollouts — just a frozen base model and the ability to score gold answers under perturbed histories. For teams that already have teacher trajectories and want to avoid the instability of outcome-based RL, that is a practical path.

Future Directions

• How much of the win is retrieval versus reasoning? The authors' own decomposition shows open-weight models lose ground on using earlier-retrieved pages (a 12.3-point gap on pages found in an earlier search versus 5.1 points on the latest one). Whether longer-context training, explicit evidence management, or better cross-turn memory closes this is left open.

• Can the credit signal be pushed further? The method weights steps but does not add or remove them. Combining counterfactual reweighting with trajectory editing, rejection of low-value steps, or online RL on top of the reweighted SFT initialization is a natural extension.

• What is the real cost ceiling? Page images are token-expensive, and the resolution ablation (1.05 megapixels, 30% to 48% token savings) shows a modest accuracy trade. Adaptive per-turn resolution — high fidelity only for pages the agent decides to read closely — is unexplored.

• Is ViDoRe-V3 too text-heavy to test the claim? Only 1.7% of its questions genuinely require visual content, which the authors acknowledge is why page and text retrieval land so close. A benchmark of non-scientific, layout-rich documents where vision is load-bearing (financial filings, forms, slide decks) would sharpen the out-of-distribution picture.

• Can the benchmark generation pipeline scale further? Questions are LLM-generated from enriched abstracts with human rewriting of all 500 items; whether the pipeline can be scaled to thousands of questions without human review, and whether such questions remain non-templated and unambiguous, is an open practical question.

Target Audience

Most valuable for: researchers and engineers working on document AI, retrieval-augmented generation, and agentic systems, particularly those building over scientific, financial, or legal corpora and currently maintaining OCR/parsing pipelines. It is also relevant to practitioners doing agentic fine-tuning — the counterfactual credit reweighting recipe applies well beyond documents — and to benchmark designers, since the paper articulates a clear four-property criterion (agentic, multimodal, non-templated, verifiable) that existing benchmarks fail in different ways.

Less essential for: readers looking for new vision architectures or new embedding models; both the retriever (Qwen3-VL-Embedding-8B) and the base policy (Qwen3.6-27B) are off-the-shelf, and the contribution is in the system, benchmark, and training method rather than in model design.

Authors’ abstract

Complex documents such as scientific papers contain rich information in tables, figures, and equations; however, existing deep research (DR) agents typically reduce papers to OCR-extracted text. We instead ask whether vision-language models (VLMs) can enable page-native deep research: directly retrieving and reasoning over rendered document page images. We introduce DR Pixel, the first open-source end-to-end multimodal DR agent that uses rendered page images as its sole retrieval and reasoning unit, operating over a corpus of 1.06M pages. To evaluate this setting, we construct DRPixelBench, a benchmark on scientific papers requiring reasoning over text, figures, tables, and equations. Surprisingly, page-native DR already works well without training, outperforming OCR-based baselines across state-of-the-art models like GPT-5.4, Gemini-3.5-Flash, and Qwen3.6-27B, while requiring fewer tool calls. We next turn to DR training. A key challenge is that teacher trajectories for agentic supervised finetuning are noisy, mixing critical reasoning and action steps with redundant or failed attempts. To address this, we introduce counterfactual credit reweighting, a fine-grained credit assignment method that weights individual reasoning and action steps by their counterfactual contribution to the final answer. Our training improves retrieval recall by about 15 points and final accuracy by over 10 points on DRPixelBench, and likewise outperforms the untrained baseline on four existing benchmarks. On out-of-distribution ViDoRe-V3, DR Pixel comes within 1.8 points of an oracle that is given the gold pages directly. Our results suggest that the conventional OCR-then-retrieve pipeline is not only unnecessary but suboptimal: directly retrieving and reasoning over visual document pages makes complex DR both more effective and more efficient.