HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding

AI summary

Overview

Research area: Efficient inference for vision-language models (VLMs) applied to document parsing — specifically, accelerating autoregressive decoding of long, structured page-level outputs.

Technical level: Advanced. The paper assumes familiarity with autoregressive decoding, vision-language models, speculative decoding, KV caches, and attention masking.

Scope: The paper introduces Hierarchical Speculative Decoding (HSD), a training-free, two-stage local-to-global verification framework that speeds up end-to-end document-parsing VLMs while preserving page-level coherence.

What This Paper Is About

End-to-end document parsers built on VLMs produce accurate, globally coherent page markup, but they must autoregressively generate very long token sequences, so latency grows roughly linearly with output length. Hybrid approaches restore speed by segmenting a page and decoding regions in parallel, yet decoding each region independently discards full-page context (reading order, cross-column links) and propagates layout errors. HSD's goal is to obtain region-level parallelism and page-level coherence without any additional training, by verifying cheap pipeline-generated drafts in two stages: first per region, then across the whole page.

Key Contributions

1. Hierarchical Speculative Decoding (HSD) — a two-stage paradigm for end-to-end document parsing that performs region-wise parallel verification first, then restores global coherence through page-level verification.
2. Decoupled Speculative Verification (DSV) — a mechanism for further acceleration that resolves prefix–draft misalignment via a draft–target matching process and verifies multiple candidate continuations efficiently using prefix-tree batching.
3. A training-free, plug-and-play method demonstrated across several parsers and benchmarks, reported to achieve up to 7.04x speedup with near-lossless accuracy.
4. Empirical characterization across document types, languages, and target models, including comparisons against prior speculative decoding baselines and against pipeline, hybrid, and end-to-end parsing paradigms.

Main Findings

• Headline speedups with HunyuanOCR: end-to-end speedups of 2.78x on OmniDocBench v1.5, 2.46x on olmOCR-Bench, and 3.29x on Ocean-OCR-Bench; up to 7.04x on long-document parsing tasks.
• Broad generality across models: consistent speedups are reported for dots.ocr (3B), HunyuanOCR (0.9B), Qwen2.5-VL-3B, Qwen2.5-VL-7B, Qwen3-VL-2B, and Qwen3-VL-8B across all three benchmarks.
• Document-type dependence: larger gains appear for long documents with multiple semantic blocks (e.g., Newspaper and Academic Papers in OmniDocBench v1.5), while harder inputs such as Old Scans in olmOCR-Bench produce lower-quality drafts and therefore less acceleration. In Table 1, HunyuanOCR's per-type end-to-end speedup reaches 7.04x on Financial Report and 4.30x on Notes.
• Language coverage: Ocean-OCR-Bench (100 English and 100 Chinese document images) shows bilingual gains, e.g., HunyuanOCR at 3.96x for English and 2.86x for Chinese.
• Strong performance against speculative decoding baselines: under the same Qwen2.5-VL-3B target model, the best baseline (ViSpec) reaches 1.72–1.90x, whereas HSD reaches 2.12–2.72x without additional training.
• Stage 2 is essential: using Stage 1 alone degrades accuracy sharply — dots.ocr drops from 88.41 to 70.47 on OmniDocBench v1.5 — while adding Stage 2 restores accuracy to 88.81 on OmniDocBench v1.5 and 92.56 on Ocean-OCR-Bench.
• Ablation of framework design (dots.ocr, OmniDocBench v1.5): a fast pipeline drafter alone gives 2.09x end-to-end speedup; adding the hierarchical two-stage design raises this from 2.09x to 2.42x and improves AAL from 2.49 to 3.98.
• Stackable with other acceleration: combined with visual token compression (DeepSeek-OCR), HSD adds further speedups — 1.56x end-to-end on OmniDocBench v1.5, 1.41x on olmOCR-Bench, and 1.91x on Ocean-OCR-Bench over VTC alone.
• Position relative to parsing paradigms (OmniDocBench v1.5, Table 5): pipelines are fast but lower quality, hybrids trade quality and latency, and end-to-end parsers achieve the strongest accuracy but are markedly slower. As reported, HunyuanOCR goes from 94.10 accuracy at 30.47 s/sample to 94.02 accuracy at 10.96 s/sample with HSD, reducing end-to-end parser latency by 2.42–2.78x with essentially unchanged quality.
• Not reported in the available content: comparisons are run on NVIDIA A100 GPUs using the same Hugging Face Transformers stack with FlexAttention; memory consumption, energy cost, and throughput in pages per second are not reported.

Methodology in Plain English

A lightweight document-parsing pipeline runs once per page and does two things: it predicts a layout that splits the page into semantic regions (paragraphs, tables, figures), and it produces a rough draft of the text for each region. These drafts are cheap "guesses" at what the accurate VLM parser would output.

The accurate VLM then acts as the judge in two verification passes. In the first pass, each cropped region plus its draft is fed to the model, and the model accepts or corrects the draft tokens in parallel across regions — this is fast because no full page is decoded. However, region-only verification can keep layout mistakes made by the pipeline and lacks whole-page context, so a second pass takes the assembled region outputs as a page-level draft and verifies it against the full page image, fixing remaining structural errors such as wrong reading order in relatively few decoding steps.

Because the drafts are generated only once and are not refreshed as the VLM generates, they can be misaligned with what the model has already accepted. The paper's Decoupled Speculative Verification handles this by sliding a short reference window (n = 3 tokens) from the accepted output across the draft to find matching positions, extracting the suffixes that follow each match as candidate continuations. When several candidates exist, they are merged into a prefix tree so shared prefixes are verified once, and a tree-ancestry attention mask lets all candidate paths be processed together in a single forward pass. A token is accepted when its log-probability is within a threshold (τ = 0.75) of the model's most probable token; otherwise the traversal stops and the model's own token is used. The packed verification sequence is capped at 128 tokens, and PP-StructureV3 is used by default for layout analysis and draft generation.

Why This Matters

Impact on research. The paper argues the tension between region-level parallelism and page-level global coherence is not inherent, and shows a training-free way to get both. It also demonstrates that generic speculative decoding techniques transfer poorly to document parsing: the best prior baseline reaches only 1.72–1.90x under the same target model, whereas HSD reaches 2.12–2.72x, framing document parsing as a distinct challenge for speculation research.

Real-world applications:

• Large-scale document indexing and retrieval, where millions of pages must be converted to structured text.
• Workflow automation and data governance pipelines that ingest heterogeneous scanned or digital documents.
• Large-scale corpus construction for training and analytics, where parsing throughput is a bottleneck.
• Intelligent document analysis in multilingual settings (the evaluation covers English and Chinese page images).

Industry relevance. The method is training-free and plug-and-play, requires no modification of the underlying VLM, and stacks with existing visual token compression (demonstrated with DeepSeek-OCR). For deployments already serving end-to-end parsers, that combination of properties directly reduces inference cost and user-perceived latency without retraining or re-validating model weights.

Future Directions

• Improving draft quality on hard inputs. Gains shrink on degraded or handwritten scans such as Old Scans in olmOCR-Bench because drafts are weaker; better draft generation for these inputs is an open problem.
• Reducing dependence on the external pipeline. HSD relies on a separate layout/drafting pipeline (PP-StructureV3 by default), so its errors and interfaces matter; whether this stage can be simplified or internalized is unexplored.
• Extending beyond document parsing. The hierarchical draft-verify idea is motivated by the structured layout of documents, and testing it on other long structured multimodal generation tasks is a natural next step.
• Deeper stacking with other efficiency techniques. The paper shows compatibility with visual token compression; combining HSD with additional accelerations (quantization, other drafting schemes) and quantifying end-to-end cost trade-offs remains open.

Target Audience

Researchers and engineers working on efficient multimodal inference, speculative decoding, and document AI; practitioners deploying VLM-based document parsers who need lower latency without retraining; and readers interested in how hierarchical, drafting-based verification can be adapted to long structured outputs.

Authors’ abstract

Document parsing is a fundamental task in multimodal understanding, supporting a wide range of downstream applications such as information extraction and intelligent document analysis. Benefiting from strong semantic modeling and robust generalization, VLM-based end-to-end approaches have emerged as the mainstream paradigm in recent years. However, these models often suffer from substantial inference latency, as they must autoregressively generate long, full-page sequences when processing long-form documents. While recent hybrid methods mitigate this issue via region-level parallel decoding with VLMs, independent region decoding loses full-page context and might weaken global coherence. To address this issue, we propose Hierarchical Speculative Decoding (HSD), a two-stage local-to-global framework for document parsing. HSD first employs a lightweight pipeline drafter to predict region partitions and generate coarse drafts for each region. The first stage verifies the generated region-level drafts in parallel for efficiency, while the second stage further performs page-level verification on these refined outputs to preserve full-page coherence. Experimental results show that HSD achieves a near-lossless 2.7x speedup with HunyuanOCR on OmniDocBench v1.5 and up to 7.04x speedup on long-document parsing tasks, demonstrating the effectiveness of the proposed method. The code is available at https://github.com/whlscut/HSD.