Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark

AI summary

Overview

Research area: Computer vision and information retrieval, specifically cross-modal document image retrieval and visual document understanding (VDU).

Technical level: Intermediate. The paper assumes familiarity with contrastive vision-language models, OCR pipelines, and retrieval metrics, but its core idea, retrieving document images using natural-language descriptions rather than another image or a keyword, is easy to grasp.

Scope: The paper introduces the NL-DIR dataset and benchmark for retrieving document images from a gallery using fine-grained natural-language queries, and benchmarks existing contrastive vision-language models and OCR-free visual document understanding models on it.

What This Paper Is About

Existing document image retrieval (DIR) methods either match a query image against document images, which limits retrieval to the same coarse semantic category such as newspapers or receipts, or use OCR to convert documents to text and then do text matching, which discards visual information such as layout, visual elements, and display style. The authors argue that real users instead provide natural-language queries with fine-grained semantics, and that no dedicated dataset existed to train or evaluate models for this setting. Their goal is to build such a dataset and benchmark, and to propose a retrieval approach that works on real-world documents while remaining efficient in time and space.

Key Contributions

1. The authors release NL-DIR, described as the first benchmark for fine-grained document image retrieval in natural scenes, consisting of 41,795 document images paired with 205K queries (also stated as 41K images and 205K queries), with queries generated by large language models and filtered through scoring models plus manual verification.

2. They conduct a comprehensive analysis of popular cross-modal retrieval models and document understanding models on this task, evaluating mainstream contrastive VLMs and OCR-free generative VDU models in both zero-shot and fine-tuning settings.

3. They propose a two-stage retrieval method consisting of a recall stage and a re-ranking stage, which achieves strong retrieval performance while ensuring efficient use of time and space, and they compare it against OCR-dependent text retrieval and large vision-language model (LVLM) based DIR methods on both accuracy and efficiency.

4. They report three key insights: the choice of pre-training tasks and datasets significantly impacts retrieval performance, with image-text contrastive pre-training performing best; fine-grained interaction is an effective way to capture semantic information in document images; and OCR-free models show advantages when queries contain visual information about non-text elements, particularly in low-quality document images.

Main Findings

• Zero-shot recall: SigLIP-So400m achieves the best zero-shot result among contrastive VLMs with Recall@10 of 61.18, Recall@1 of 36.17, and MRR@10 of 43.78, attributed to pre-training on a vast corpus of image-OCR pairs. DFN reaches Recall@10 of 51.24 and InternVL-14B-224px reaches 43.45, while CLIP-base reaches 3.99 and BLIP-base 6.02 at Recall@10.

• Generative VDU models lag in zero-shot: Donut, Nougat, Pix2Struct, Vary, and TextMonkey all score at or below Recall@10 of 0.27, while DocOwl1.5 reaches 0.94, UReader 1.19, and Qwen2-VL 1.66. InternVL-14B-224px, which aligns visual and text representations during pre-training, attains a noteworthy Recall@10 of 43.45, considerably superior to Qwen2-VL.

• Fine-tuning aligned encoders helps modestly: Aligning VDU visual encoders with CLIP or BLIP text encoders via LoRA and an extra alignment layer produces best results with BLIP text and TextMonkey visual (Recall@1 5.03, Recall@10 23.33, MRR@10 9.60) and with CLIP text and TextMonkey visual (4.46 / 21.99 / 8.92). BLIP's text representation is reported as more suitable for the DIR task than CLIP's.

• SigLIP fine-tuning is the strongest recall result: Fine-tuning SigLIP's text module with LoRA raises Recall@10 from 61.18 to 79.40 (Recall@1 54.32, MRR@10 62.39). Fine-tuning the image module (SigLIP-Image-LoRA) gives the best recall-stage performance at Recall@1 69.33, Recall@10 89.72, MRR@10 76.29, and the best model's Recall@100 reaches 97.52.

• Re-ranking with BLIP-ITM and Pix2Struct is best: The configuration using BLIP-ITM text, Pix2Struct visual, and extra cross-attention achieves Recall@1 81.03, Recall@10 94.17, and MRR@10 85.68 after re-ranking the top 100 results.

• Concurrent LVLM methods are competitive in zero-shot: DSE scores 69.43 / 87.57 / 75.61 and ColPali scores 79.65 / 91.64 / 83.79, with ColPali achieving impressive results despite less training data (12.7K fine-tuning samples) through fine-grained interactions in the same representation space. The OCR-IR pipeline scores 52.83 / 71.63 / 58.85.

• Concrete queries are easier than abstract ones: On a concrete set of 3,100 queries (length greater than 18 words and more than 10 overlapping words with OCR), MRR@10 after re-ranking is 96.66, versus 85.68 on the original test set and 75.73 on an abstract set of 3,100 queries (length shorter than 14 and fewer than 6 overlapping words). Fine-grained re-ranking improves all query sets, most for detailed intents.

• Visually rich subsets: On a chart subset (61 images, 305 queries), recall reaches Recall@1 89.20 / Recall@10 99.02 / MRR@10 93.13, and re-ranking gives 89.70 / 98.65 / 92.91, versus OCR-IR at 74.96 / 92.39 / 80.57. On an advertisement subset (163 images, 815 queries), recall gives 87.64 / 99.01 / 92.17 and re-ranking 80.66 / 97.05 / 85.55, versus OCR-IR at 66.88 / 85.90 / 72.40.

• Efficiency: Averaged over 100 sampled images on a 4090 GPU with batch size 4, SigLIP encoding takes 0.07 seconds per image, text-based methods 2.56 seconds, DSE 0.62 seconds, and ColPali 0.65 seconds. Embedding size is 4KB for SigLIP, comparable to BGE at 3KB and DSE at 6KB, while ColPali requires 256KB due to multi-vector embeddings. Re-ranking visual embedding generation takes 0.2 seconds per image.

• OCR-free advantage on hard cases: Qualitative analysis shows the OCR-free model outperforms OCR-dependent methods on poor image quality such as handwritten text, out-of-vocabulary problems (a query mentions "Whitalter" while OCR produced "Whéaaker"), and non-text visual elements such as humorous cartoon-strip ads.

• Configuration findings: One additional cross-attention layer is optimal, since more layers hinder convergence. Removing pairwise loss causes a slight performance decrease, while removing pointwise loss causes much slower convergence and a more pronounced drop. Attempts to fine-tune DSE on the training data were not satisfactory, likely due to the size and diversity of fine-tuning data and batch size limits, and using TextMonkey or Pix2Struct-Large for re-ranking did not improve results while significantly increasing encoding time.

Methodology in Plain English

The authors first collected real document images from the Industry Documents Library: roughly 50k images from OCR-IDL plus additional images from DocVQA, giving 60k initial documents after removing duplicates, which became 41,795 images. OCR annotations came from Microsoft OCR for DocVQA and Amazon Textract for OCR-IDL. ChatGPT was used to generate ten candidate queries per image, guided by prompts and by layout information simulated with spaces and line breaks. Those queries were then scored by ChatGPT and Qwen-VL-Plus (with rationales requested to reduce bias) and by the image-text alignment models CLIP and BLIP, weighted 3:3:2:2, sorted, and passed to experienced researchers for manual verification, with rules to remove low-quality images and queries that were weakly relevant, strongly layout-dependent, or overly general. The dataset is split 8:1:1 into training, validation, and test sets with identical category distribution, and 10% of the data forms the benchmark.

For retrieval, they use a two-stage pipeline. In the recall stage, visual and text representations are extracted (mean pooling for zero-shot; for generative models the EOS token or a compressed last-layer feature is used, following concurrent works), scored by dot product, and used to fetch the top 100 candidates. Fine-tuning aligns text encoders (CLIP/BLIP with LoRA and an extra alignment layer, InfoNCE loss; SigLIP with LoRA on both encoders and Sigmoid loss) with frozen visual encoders. In the re-ranking stage, cross-attention modules (BLIP-ITM's or Pix2Struct's, or additional ones) perform fine-grained image-text interaction on the top 100, trained with a hard-negative mining strategy that targets the top 10 hard negatives from the recall stage, using a combination of pointwise and pairwise loss. Evaluation uses Recall@k and MRR@k with k set to 1 and 10.

Why This Matters

Impact on research: The paper defines a new retrieval task, natural-language-based document image retrieval, and supplies the first dedicated benchmark for it, with 41,795 images, 205K queries, 247 document categories, and released multi-class labels and multi-page OCR results to support document classification, multi-page retrieval, and other tasks. It shows that existing single-modality pipelines lose information, and that pre-training choices, fine-grained interaction, and OCR-free representations each matter in distinct ways.

Real-world applications:

• Searching large archives of scanned or photographed documents, such as the Industry Documents Library, using a written description rather than a reference image or exact keywords.

• Finding specific pages in a personal or corporate document collection by describing content, layout, or visual elements such as charts and ads.

• Supporting retrieval when OCR fails, for instance on handwritten text or unusual vocabulary, by relying on visual understanding instead.

• Serving as a component in downstream document workflows such as classification, multi-page retrieval, question answering, and information extraction.

Industry relevance: The efficiency analysis matters for deployment: single-vector embeddings are much cheaper to store than ColPali's 256KB multi-vector embeddings, contrastive recall avoids the OCR step that dominates text-based pipelines at 2.56 seconds per image, and the pipeline leaves room for optimization such as parallel visual encoding of the top 100 candidates.

Future Directions

• Improving zero-shot retrieval of generative VDU models, whose Recall@10 on NL-DIR stays between 0.21 and 1.66 for most models and only reaches 43.45 for InternVL-14B-224px.

• Closing the gap between the recall stage and the re-ranking stage on visually rich documents, where re-ranking actually weakens results on the advertisement subset (MRR@10 drops from 92.17 to 85.55).

• Addressing the reported bad cases: fine-tuning DSE did not transfer well, larger visual encoders such as TextMonkey and Pix2Struct-Large did not improve re-ranking while increasing encoding time, and LoRA fine-tuning of the text encoder during re-ranking produced unpromising results.

• Extending the benchmark beyond what is reported, since the paper states that datasets and codes will be publicly available at huggingface.co/datasets/nianbing/NL-DIR and that multi-class labels and multi-page OCR results will be released for other tasks.

Target Audience

Researchers and practitioners in visual document understanding, cross-modal retrieval, and information retrieval who are interested in text-to-document-image search; engineers building document search or archiving systems; and graduate students looking for a benchmark with defined metrics, baseline numbers, and released data to build on.

Authors’ abstract

Document image retrieval (DIR) aims to retrieve document images from a gallery according to a given query. Existing DIR methods are primarily based on image queries that retrieve documents within the same coarse semantic category, e.g., newspapers or receipts. However, these methods struggle to effectively retrieve document images in real-world scenarios where textual queries with fine-grained semantics are usually provided. To bridge this gap, we introduce a new Natural Language-based Document Image Retrieval (NL-DIR) benchmark with corresponding evaluation metrics. In this work, natural language descriptions serve as semantically rich queries for the DIR task. The NL-DIR dataset contains 41K authentic document images, each paired with five high-quality, fine-grained semantic queries generated and evaluated through large language models in conjunction with manual verification. We perform zero-shot and fine-tuning evaluations of existing mainstream contrastive vision-language models and OCR-free visual document understanding (VDU) models. A two-stage retrieval method is further investigated for performance improvement while achieving both time and space efficiency. We hope the proposed NL-DIR benchmark can bring new opportunities and facilitate research for the VDU community. Datasets and codes will be publicly available at huggingface.co/datasets/nianbing/NL-DIR.