Spatially-Grounded Document Retrieval via Patch-to-Region Relevance Propagation

AI summary

Overview

Research area: Computer vision and information retrieval, specifically late-interaction multimodal document retrieval and retrieval-augmented generation (RAG) context selection.

Technical level: Intermediate to Advanced. The paper assumes familiarity with vision transformer patch grids, late-interaction retrieval (ColBERT-style MaxSim), and OCR bounding boxes, and it includes formal definitions, theorems, and complexity analysis.

Scope: The paper proposes and evaluates a training-free, inference-time architecture that converts page-level patch similarity scores from ColPali-family models into region-level relevance scores over OCR-extracted text blocks, evaluated on BBox-DocVQA.

What This Paper Is About

Late-interaction visual retrievers such as ColPali embed document pages as images and score relevance at the patch level, but they return results at page granularity, so a RAG system knows which page holds the answer but not where on the page it sits. OCR pipelines provide the opposite: precise text with bounding box coordinates but no semantic ranking of which region answers a query. The paper unifies the two by treating ColPali's per-patch similarity scores as spatial relevance filters over OCR bounding boxes, returning ranked text regions instead of whole pages, without any additional training.

Key Contributions

1. Coordinate mapping formalism. A formal mathematical correspondence between vision transformer patch grids and OCR bounding boxes, including the scaling of OCR coordinates into the model's input coordinate space when the original page resolution differs from the model input resolution.

2. Relevance propagation via interpretability maps. A method that repurposes ColPali's late-interaction mechanism to produce per-query-token similarity heatmaps, then propagates these scores to OCR regions through IoU-weighted patch-region intersection, with alternative max and mean aggregation strategies also defined.

3. Two-stage retrieval architecture. Stage 1 mean-pools patch embeddings for approximate nearest-neighbor candidate page retrieval; Stage 2 computes full patch-level similarity on the candidates and reranks regions.

4. Theoretical analysis and open implementation. Bounds on area efficiency as a function of patch resolution, a context reduction bound, computational complexity analysis, empirical validation on BBox-DocVQA, and release of the open-source Snappy system.

Main Findings

• Within-page localization accuracy: For within-page localization given correct page retrieval, ColQwen3-4B with percentile-50 thresholding (P50) achieves a 59.7% hit rate at IoU@0.5, 84.4% at IoU@0.25, and 35.8% at IoU@0.7, with a mean IoU of 0.569. Random region selection is reported at approximately 6.7%.

• Diminishing returns from scale within an architecture family: ColQwen3-8B scores 59.8% at IoU@0.5 with a mean IoU of 0.569 — nearly identical to ColQwen3-4B (59.7%, 0.569) — indicating that doubling parameters within the same architecture yields negligible localization gains. Both report 1,619 samples.

• Smaller model underperforms but is more token-efficient: ColModernVBERT (250M parameters) reaches 45.5% at IoU@0.5, 72.2% at IoU@0.25, 27.9% at IoU@0.7, and a mean IoU of 0.480, but achieves the highest token savings of the three (38.0% versus all OCR regions, 58.5% versus full image tokens).

• Token savings: ColQwen3-4B P50 reduces context tokens by 28.8% versus returning all OCR regions and 52.3% versus full-page image tokens. ColQwen3-8B P50 reduces tokens by 26.3% and 50.7% respectively. Total token counts across the 1,619 samples are 4,003,039 for full image, 2,678,723 for all OCR regions, 1,908,329 for ColQwen3-4B, 1,974,263 for ColQwen3-8B, and 1,661,684 for ColModernVBERT.

• Category variation: ColQwen3-4B performs best on computer science (mean IoU 0.697, IoU@0.5 of 75.5%) and electrical engineering (0.656, 78.1%), and worst on economics (0.513, 46.7%) and mathematics (0.382, 28.7%). The paper attributes lower accuracy to denser tabular content and smaller region sizes where patch quantization effects are more pronounced.

• Category-specific model gaps: The gap between ColQwen3-4B and ColModernVBERT is largest for electrical engineering (Δ mean IoU 0.205; 35.2 percentage points at IoU@0.5) and computer science (Δ 0.170; 26.0 pp), and smallest for mathematics (Δ 0.012; 1.6 pp). The overall gap is 0.089 mean IoU and 14.2 pp at IoU@0.5.

• Failure modes: Of the 40.3% of samples (652 of 1,619) where the top-ranked region does not reach IoU@0.5, 487 samples had no selected region achieving adequate overlap. Among these, 414 (85.0%) are OCR ceiling failures where no OCR region overlaps the ground truth sufficiently, and 73 (15.0%) are selection errors. Selection error rates by category are q-bio 9.7%, physics and eess 6.1%, cs 2.3%, and econ 0.9%. Of samples failing at IoU@0.5, 48% still achieve IoU at or above 0.25, and the mean IoU of failed samples is 0.250.

• Theoretical bounds: The area efficiency bound gives a maximum efficiency of ≤73% for a 200×50 pixel paragraph region, ≤60% for a 100×30 table cell, and ≤46% for a 50×20 label at a patch size of 14 pixels. The context reduction factor relative to page-level retrieval is bounded below by M/k, which for M=15 regions per page and k=3 returned regions is at least 5×.

• Two-stage efficiency: For typical values (N=100,000 pages, K=100, G=32, n=20 tokens, d=128 dimensions, M=15 regions per page), Stage 1 reduces the search space by 1000× before region-level computation.

• Model-selection regimes: The paper identifies three empirically distinct regimes — architecture-sensitive documents with large well-separated regions, scale-insensitive behavior within an architecture family, and uniformly challenging documents where all models converge to lower performance.

Methodology in Plain English

The authors take a ColPali-family model that already splits each page image into a 32×32 grid of 1,024 patches, each mapped to a 128-dimensional embedding over a 448×448 input, with each patch covering 14×14 pixels. Instead of collapsing those patch scores into a single page score via MaxSim, they keep the spatial heatmap: for each patch, they record the maximum cosine similarity to any query token.

They then render each PDF page as an image and run layout-aware OCR (DeepSeek-OCR with visual grounding in markdown mode) to extract text regions with bounding boxes, chosen because it groups semantically coherent blocks such as paragraphs, table cells, and captions rather than individual words or lines. Because the page resolution usually differs from the model's 448×448 input, they rescale OCR boxes into model coordinates.

Each OCR region is scored by an IoU-weighted average of the patch scores it overlaps, normalized by total IoU so that large regions do not automatically win. Regions are then ranked and the top ones returned. The pipeline runs in two stages: first a fast approximate nearest-neighbor search using mean-pooled page embeddings to get candidate pages, then full patch-level scoring and region reranking on those candidates. Thresholding at the 50th percentile (P50) was chosen based on an ablation: P25 thresholds were too inclusive and selected all regions, while P75 required careful tuning of minimum patch overlap to avoid accuracy degradation. Token counts were estimated using tiktoken's cl100k_base encoding for OCR text and an image token approximation that resizes images within 1568×1568 pixels and computes floor(w×h/750).

Why This Matters

Impact on research: The work shows that interpretability signals already computed inside late-interaction retrievers — patch attention distributions that are normally discarded — can be repurposed as a spatial filter at inference time. It introduces region-level granularity as an evaluation target distinct from the page-level NDCG@5 metrics used by benchmarks such as ViDoRe, and it provides an empirical map of where patch quantization, rather than model capacity, becomes the binding constraint. It also documents a comparison limitation: RegionRAG, the closest prior work, had no public implementation or model weights as of December 2025, so comparison is against shared ColPali-family baselines.

Real-world applications:

• RAG systems that need to feed a language model only the specific passage, table cell, or figure caption that answers a query, rather than an entire page.
• Financial and business document analysis, where an answer such as a revenue figure lives in one table cell rather than across a page.
• Technical and scientific document search across domains such as computer science, physics, and quantitative biology where layouts vary widely.
• Form field and small-label extraction scenarios, though the paper notes these suffer most from patch quantization and may need higher-resolution patch grids or multi-scale approaches.

Industry relevance: The approach requires no additional training and works with any OCR system providing bounding boxes and any ColPali-family model, so it can be layered onto existing retrieval stacks. The 28.8% and 52.3% token reductions translate directly into lower LLM inference cost, reduced latency, and less attention dilution in the context window. The finding that ColQwen3-4B matches ColQwen3-8B for localization means practitioners can use the cheaper model, while ColModernVBERT's 16× parameter efficiency may be acceptable for mathematics-heavy corpora where all models converge to similar accuracy.

Future Directions

• Raising the OCR ceiling. Since 85% of analyzed failures come from OCR segmentation not producing a region that overlaps the ground truth adequately, improving upstream layout segmentation or aligning it with the evaluation's evidence boxes appears to be the largest single lever.
• Higher-resolution or multi-scale patch grids. The area efficiency analysis shows small regions suffer disproportionately; the authors suggest higher-resolution patch grids or multi-scale approaches for fine-grained localization such as form field extraction.
• Explaining the mathematics convergence. The paper reports that mathematics documents show the smallest model gap (Δ 0.012 mean IoU) and the lowest absolute performance across all models, and states that the precise cause — training data composition, difficulty parsing equations and dense notation, or other factors — remains an open question.
• Improving region selection on ambiguous layouts. The 73 selection failures concentrate in categories such as q-bio (9.7%) and physics/eess (6.1%), which the paper links to semantically ambiguous layouts where multiple regions could plausibly contain the answer; the truncated error-analysis implications section points toward further work here.
• Threshold and aggregation tuning. The ablation reports that P25 selects all regions and P75 requires careful tuning of minimum patch overlap, leaving room for more adaptive thresholds and aggregation strategies beyond max, mean, and IoU-weighted.

Target Audience

Researchers and engineers working on retrieval-augmented generation, multimodal document retrieval, and visual document understanding who are already familiar with late-interaction models like ColPali and with OCR pipelines. It is also relevant to practitioners building production RAG systems who are optimizing the tradeoff between context precision and token cost, and to readers interested in treating model interpretability signals as functional components rather than diagnostics. Readers without background in vision transformer patch grids or MaxSim-style scoring will need to consult the cited ColPali and ColBERT work first.

Authors’ abstract

Late-interaction multimodal retrieval models like ColPali achieve state-of-the-art document retrieval by embedding pages as images and computing fine-grained similarity between query tokens and visual patches. However, they operate at page-level granularity, limiting utility for retrieval-augmented generation (RAG) where precise context is paramount. Conversely, OCR-based systems extract structured text with bounding box coordinates but lack semantic grounding for relevance assessment. We propose a hybrid architecture that unifies these paradigms: using ColPali's patch-level similarity scores as spatial relevance filters over OCR-extracted regions. We formalize the coordinate mapping between vision transformer patch grids and OCR bounding boxes, introduce intersection metrics for relevance propagation, and establish theoretical bounds on area efficiency. We evaluate on BBox-DocVQA with ground-truth bounding boxes. For within-page localization (given correct page retrieval), ColQwen3-4B with percentile-50 thresholding achieves 59.7% hit rate at IoU@0.5 (84.4% at IoU@0.25, 35.8% at IoU@0.7), with mean IoU of 0.569, compared to ~6.7% for random region selection. Our approach reduces context tokens by 28.8% compared to returning all OCR regions and by 52.3% compared to full-page image tokens. Our approach operates at inference time without additional training. We release Snappy, an open-source implementation at https://github.com/athrael-soju/Snappy.