DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding

AI summary

Overview

Research area: Computer Vision / Vision-Language Models for document understanding, with a focus on efficient inference on resource-constrained edge devices.

Technical level: Intermediate. The paper assumes familiarity with vision-language model architectures, OCR pipelines, cross-attention, and token-level uncertainty estimation, but its core ideas are explained at an architectural level.

One-sentence scope: DocSLM is a 2B-parameter small vision-language model that compresses each document page into a fixed 576 tokens and processes long documents segment-by-segment with entropy-based abstention, matching or beating 7B–8B models and RAG systems at much lower memory and latency.

What This Paper Is About

Long multimodal documents such as financial reports, technical papers, and slide decks contain dense text, tables, figures, and complex layouts spread over many pages. Existing Large Vision-Language Models (LVLMs) handle these well but frequently exceed 8B parameters and require thousands of input tokens per page, which makes them impractical on edge GPUs. DocSLM's goal is to deliver accurate long-document understanding under strict memory and token-length limits by compressing each page to a fixed token budget and streaming through document segments with a learned uncertainty mechanism.

Key Contributions

1. DocSLM, a compact 2B-parameter Vision-Language Model that uses 82% fewer visual tokens and 75% fewer parameters than existing large LVLMs and retrieval-augmented models.
2. A Hierarchical Multimodal Compression module that jointly encodes visual, textual (OCR), and layout features, achieving a 5.6x reduction in input tokens (3210 to 576) while preserving local and global semantics.
3. A Streaming Abstention mechanism that processes documents segment-by-segment using an entropy-based uncertainty calibrator, maintaining a constant memory footprint so arbitrarily long documents can be processed on constrained hardware.
4. State-of-the-art results for its size class, surpassing DocOwl2-8B by +9.3% under a comparable token budget and outperforming the similarly sized Docopilot-2B by +0.9%, while reducing latency by 3.5x (32.1 ms vs. 113.4 ms) compared to InternVL2-RAG.

Main Findings

• Memory scales flat, not linearly: DocSLM sustains a constant plateau of roughly 14 GB peak GPU memory beyond 10 document pages because of its streaming mechanism, and can process up to 120-page documents from MMLongDocBench. Peak memory is measured with PyTorch profiling tools under identical inference settings across models.
• Main benchmark results (Table 2): DocSLM records 22.7 on MMLDoc, 70.0 ANLS on MP-DocVQA, 47.6 ANLS on DUDE, and 66.2 ANLS on NewsVQA, using 576 tokens per image, 2B parameters, and 32.1 ms latency.
• Comparisons with large models: DocSLM shows a +5.7 pp gain over InternVL2-8B on DUDE and near-parity with 8B Docopilot on MMLDoc despite using 75% fewer parameters. Against 8B RAG models it retains over 95% of their accuracy while running 3.5x faster.
• Comparisons with small models: Against Docopilot-2B (3,133 tokens, 35.9 ms), DocSLM runs faster (32.1 ms) while improving by +0.9 pp on MMLDoc and +26.3 pp on DUDE. Against InternVL2-2B, it gains +12.2 pp on MMLDoc and +47.6 pp on DUDE.
• Why dense-token small models fail: The state-of-the-art small model Docopilot-2B requires about 3,133 tokens per image, nearly saturating the input capacity of edge GPUs (1,440 to 4,320 tokens) and thus accommodating only a single image per inference. DocOwl2 offers the lowest token count per image but degrades due to over-aggressive compression.
• Video generalization: Trained mainly on documents, with only 8.6K video samples versus 6.75M document annotations, DocSLM reaches 66.2 ANLS on NewsVideoQA, surpassing Idefics3, LLaVA-Next, and SV-RAG by +6.0, +9.5, and +5.2 pp respectively, using 5.4x fewer visual tokens and 75% fewer parameters.
• Module ablation on MP-DocVQA (Table 3): OCR baseline 50.2 ANLS at 3210 tokens and 29.2 GB; OCR-free baseline 38.5 at 2880 tokens and 27.9 GB; visual compression alone 22.7 at 576 tokens and 23.5 GB (the running text describes this step as decreasing accuracy to 36.1, which differs from the table value); OCR compression without visual compression 68.3 at 2880 tokens; combined visual plus OCR compression 67.4 at 576 tokens; adding Streaming Abstention reaches 70.0 ANLS at 14.2 GB.
• Curriculum ablation (Table 4): ANLS rises from 38.5 (instruction tuning, 3.0K steps, 1.00M data) to 50.5 (Image-OCR Alignment, 9.0K steps, 3.00M) to 61.8 (Image Compression, 2.4K steps, 2.00M) to 66.7 (Document Compression, 3.0K steps, 0.58M) to 70.0 (Streaming Abstention, 4.4K steps, 0.18M).
• Robustness to document length (Table 5): On MP-DocVQA, the baseline scores 75.3 for single-page, 29.7 for 2–10 pages, and 0.7 for more than 10 pages (38.5 overall). The OCR-augmented baseline scores 78.6, 40.5, and 2.2 (50.2 overall). DocSLM scores 79.6, 70.0, and 61.2 (70.0 overall), with gains of +4.3, +40.3, and +60.5 ANLS respectively.
• Compression design (Table 6): Four layers for both OCR and visual branches gives 70.2 ANLS; a 2-layer configuration gives 70.0 with 38.6M fewer parameters and is adopted as default. An asymmetric 4/2 setup drops to 57.3, and 2/4 reaches 70.1.
• Tuning and OCR quality (Table 6): Progressive fine-tuning from the compressor (66.5) to a linear adapter (68.6) to the SigLIP vision encoder (70.0) improves accuracy. PaddleOCR yields 70.0 versus 69.1 with Tesseract, showing OCR extraction quality remains critical under aggressive token compression.
• Latency context: InternVL2-RAG exhibits a token generation latency of 113.4 ms, described as 3.5x slower than compact non-RAG models; RAG methods such as InternVL2+RAG and M3DocRAG often exceed 110 ms per sample. VisRAG's reported latency is 288.3 ms.

Methodology in Plain English

The approach has two halves, one for each page and one for the whole document.

Per-page compression. Each page image is split into a grid of spatial crops (4 to 16 crops depending on aspect ratio and resolution) plus a downsampled global crop. A shared vision encoder (SigLIP2) extracts patch-level features. Separately, a lightweight OCR module (PaddleOCR) produces word-and-bounding-box pairs, which are assigned to crops by bounding-box overlap (intersection over union above a threshold). Compression then happens in two stages. First, local OCR compression uses cross-attention so each visual patch attends only to its paired OCR tokens, fusing text semantics into the visual features without lengthening the sequence. Second, global visual compression lets global visual features attend selectively to those compressed local regions, and the regional outputs are concatenated into a final page representation. The result is a fixed 576 tokens per page regardless of how much OCR text the page contains.

Streaming across pages. Because memory still grows with page count, the document is divided into T segments of equal length. Each segment is encoded independently and fed to a Qwen2.5-1.5B language model along with the query to produce a prediction. Before activations and KV caches are released, the model computes token-level entropy as the segment's uncertainty score. All valid segment predictions are collected, and the final document-level answer is the one with the lowest uncertainty. Because only prediction text and an uncertainty scalar are retained across segments, memory stays constant no matter how long the document is.

Training. The model follows a five-stage curriculum. Three pretraining stages (image-OCR alignment, image compression, document compression) progressively unlock the OCR compressor, vision encoder, and SLM, using datasets including DocStruct4M, MP-DocStruct1M, DocDownstream 1.0, DocDownstream 2.0, DocGenome12K, and MP-DocReason51K. Two finetuning stages handle instruction following and then streaming abstention with negative-pair supervision: for each segment containing evidence for a query, an unrelated negative segment is created where the query cannot be answered, and the model learns to emit an explicit "Not Answerable" token sequence. Training uses next-token prediction loss, FSDP across 8 nodes with 4 NVIDIA A100 (80 GB) GPUs each (32 total), batch size 1024 for pretraining and 256 for finetuning, learning rates of 1e-4 and 2e-5, AdamW, and a cosine schedule. Evaluation uses Average Normalized Levenshtein Similarity (ANLS) with a threshold of 0.5, except MMLongDocBench which reports binary Accuracy.

Why This Matters

Impact on research. The paper reframes long-document multimodal understanding as a memory-allocation problem at two levels simultaneously: page-level token compression and document-level streaming. Its ablation shows that a baseline collapsing to 0.7 ANLS past 10 pages can reach 61.2 ANLS with the same general approach, suggesting constant-memory inference is a viable substitute for ever-larger context windows and retrieval pipelines.

Real-world applications.

• On-device analysis of financial and technical reports where documents must not leave the local machine.
• Industrial document question answering over scanned pages containing tables, diagrams, and handwritten text, as in MP-DocVQA.
• Medical, legal, and technical report review, the domain mix covered by DUDE.
• Text-rich broadcast news and video frame understanding, enabled by the NewsVideoQA generalization results.
• Slide-deck querying: the supplementary material describes an interactive on-device system for uploading PPTX files, browsing slide thumbnails, and asking natural-language questions, running on a Windows laptop with a Qualcomm Snapdragon X Elite (X1E80100) and 16 GB memory.

Industry relevance. The results target the practical bottleneck for edge deployment. A 2B model at 576 tokens per image and roughly 14 GB peak memory is positioned against systems that require 3,133 to 16,384 tokens per image and 7B–16B parameters, and the paper explicitly notes RAG pipelines add retrieval latency and fragment context. Eliminating the need to load separate models for documents and video is framed as a further edge benefit.

Future Directions

• Extending DocSLM to additional modalities such as audio, working toward an omnimodal foundation model for edge deployment.
• Pursuing balanced document and video training, since the current model saw only 8.6K video samples against 6.75M document annotations yet already generalizes well.
• Reducing dependence on OCR extraction quality, given that swapping PaddleOCR for Tesseract costs 0.9 ANLS under aggressive compression (70.0 vs. 69.1).
• Revisiting aggregation strategies: the authors explored hierarchical aggregation that reuses high-confidence predictions as SLM input, which gave slight accuracy gains but incurred extra computation and latency, making it unsuitable for edge deployment. Whether better confidence calibration or smarter aggregation can close that gap remains open.
• Understanding the compression depth trade-off, since an asymmetric 4-layer OCR / 2-layer visual configuration collapses to 57.3 ANLS while 2/2 reaches 70.0.

Target Audience

Researchers and engineers working on efficient vision-language models, document AI, and on-device or edge inference. It is also relevant to practitioners building document question-answering systems over multi-page reports who are weighing retrieval-augmented pipelines against compact end-to-end models, and to readers interested in uncertainty-based abstention and streaming inference for long-context multimodal inputs.

Authors’ abstract

Large Vision-Language Models (LVLMs) have demonstrated strong multimodal reasoning capabilities on long and complex documents. However, their high memory footprint makes them impractical for deployment on resource-constrained edge devices. We present DocSLM, an efficient Small Vision-Language Model designed for long-document understanding under constrained memory resources. DocSLM incorporates a Hierarchical Multimodal Compressor that jointly encodes visual, textual, and layout information from each page into a fixed-length sequence, greatly reducing memory consumption while preserving both local and global semantics. To enable scalable processing over arbitrarily long inputs, we introduce a Streaming Abstention mechanism that operates on document segments sequentially and filters low-confidence responses using an entropy-based uncertainty calibrator. Across multiple long multimodal document benchmarks, DocSLM matches or surpasses state-of-the-art methods while using 82\% fewer visual tokens, 75\% fewer parameters, and 71\% lower latency, delivering reliable multimodal document understanding on lightweight edge devices. Code and Model are available in https://github.com/Tanveer81/DocSLM.git.