DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splitting

AI summary

Overview

Research area: Visual document understanding / intelligent document processing — specifically multi-page document packet classification and splitting (grouping, classifying, and re-ordering pages within concatenated document packets).

Technical level: Intermediate. The paper is readable by a general NLP/ML audience; it introduces a formal task definition and a set of clustering/ranking metrics (Rand Index, V-measure, Kendall's Tau) but the dataset construction and experimental protocol are explained in plain operational terms.

Scope: This paper introduces DocSplit, a benchmark of five datasets of increasing difficulty plus a new evaluation framework, and reports how five vision-language models perform on document packet splitting.

What This Paper Is About

Real-world document processing often starts with a packet of pages stitched together from several different documents, sometimes with pages out of order, duplicated, interleaved, or with no clear separators. The paper's goal is to formalize the task of splitting such a packet into its constituent documents — identifying boundaries, classifying each document's type, and restoring the correct page order — and to provide the first benchmark dataset and evaluation metrics for measuring how well models do it.

Key Contributions

1. First comprehensive benchmark for document packet splitting. The authors introduce DocSplit, described as the first comprehensive benchmark dataset for the document packet splitting task, with five datasets of varying complexity covering diverse document types, layouts, and multimodal settings.
2. A formal task definition. The DocSplit task is defined as transforming an input sequence of N pages P = {p₁, p₂, ..., pₙ} into document boundaries B = {(s₁, e₁, t₁), ..., (s_k, e_k, t_k)} (start index, end index, document type) plus a correct page ordering O = {o₁, o₂, ..., oₙ} within each document, decomposing the problem into document clustering and page ordering.
3. Five benchmark datasets. DocSplit-Mono-Seq, DocSplit-Mono-Rand, DocSplit-Poly-Seq, DocSplit-Poly-Int, and DocSplit-Poly-Rand, progressively introducing shuffled pages, interleaved documents, and full randomization. Released at https://huggingface.co/datasets/amazon/doc_split.
4. A novel evaluation framework. Metrics for boundary detection (Rand Index, V-measure), page ordering (Kendall's Tau), and a combined Packet score, evaluated against classical exact-match accuracy metrics on 10 designed edge cases.

Main Findings

• Qwen 3 attains the highest Packet scores. Qwen 3 VL 235B achieves the highest Packet scores across benchmarks, ranging from 0.9238 to 0.9492, with clustering above 0.86 and ordering above 0.97.
• A competitive second tier. Claude Sonnet 4.5, Claude Haiku 4.5, and DeepSeek V3.1 form a competitive second tier with Packet scores above 0.90.
• Gemma 3 27B is the weakest. Gemma 3 shows the weakest performance, with clustering as low as 0.5584; despite ordering scores above 0.99, its poor clustering reveals boundary detection limitations.
• Boundary detection, not ordering, is the bottleneck. Ordering scores remain high across models (above 0.97), indicating reliable sequence determination once clustering succeeds, while clustering shows much greater variance (0.56 to 0.90) and is the primary differentiator.
• Difficulty increases as packets get more heterogeneous and scrambled. DocSplit-Mono-Seq yields the highest scores (Packet above 0.93). DocSplit-Mono-Rand causes 1–3% clustering degradation while ordering stays above 0.99. DocSplit-Poly-Int and DocSplit-Poly-Rand are the most challenging, with classical metrics degrading 20–30% for weaker models.
• Classical metrics fail in specific ways. The paper reports four limitations: binary evaluation with no partial credit (DeepSeek reaches Clustering of 0.85 but Page+Split drops to 0.51 on DocSplit-Poly-Rand); a cascading penalty effect where one boundary error propagates incorrect group IDs to all subsequent pages; poor utility signals (Gemma 3 on DocSplit-Poly-Rand gets Clustering of 0.77 yet Page+Split of only 0.17); and metric redundancy (Gemma 3 shows identical Page+Split and Page+Split+Order values across benchmarks, for example 0.31 on DocSplit-Mono-Seq).
• Edge-case analysis shows the proposed metrics are more informative. On the Misclassification Only case, classical metrics report complete failure (0% on all metrics) while the proposed metrics report S_packet = 0.80 and S_clustering = 0.59. For Split Groups versus Merged Groups, the proposed metrics give S_clustering = 0.69 versus 0.20, but classical Page+Split gives 0% for both. Reverse Order versus Wrong Ordering are distinguished by S_ordering = −1.0 versus −0.67, while classical Page+Split+Order gives 0% for both. For Partial Misclass, classical Page+Split drops to 50% while the proposed S_packet = 0.89.
• Score range of the proposed framework. With default parameters α = β = 0.5 and w = 0.5, the theoretical range of S_packet spans from −0.5 to 1, where 1.0 is best and −0.5 is worst.
• Model-count discrepancy. Section 7 states that five benchmark variations were evaluated using Claude Sonnet 4.5, Claude Haiku 4.5, DeepSeek, Gemma, and Qwen; Table 2 reports those five models, while the conclusion refers to "seven vision language models." The paper does not resolve this discrepancy.

Methodology in Plain English

The authors start from RVL-CDIP-MP, a multi-page extension of the RVL-CDIP document collection. After removing 3 corrupted files from the original 998 documents, they keep 995 valid documents comprising 5,231 pages across 13 categories (form, scientific publication, handwritten, resume, letter, language, specification, questionnaire, memo, news article, email, invoice, and budget); category sizes range from 183 documents (resume) to 7 documents (language). Each page is converted to an image and its text extracted with a hybrid OCR approach: AWS Textract (LAYOUT, TABLES, FORMS, and SIGNATURES enabled) for most categories, and DeepSeek OCR (deepseek-ai/DeepSeek-OCR) for the multilingual "language" category. Documents are split category-wise and stratified into 55% training, 20% validation, and 25% testing, so no document appears in more than one split.

From this pool they build five packet-level benchmarks: single-category documents concatenated in order (Mono-Seq); the same with pages shuffled (Mono-Rand); multi-category documents concatenated in order (Poly-Seq); multi-category documents with pages interleaved round-robin (Poly-Int); and multi-category documents with all pages fully randomized (Poly-Rand). Multi-category packets are built toward a target page count of 5–20 pages.

Evaluation is done two ways. The proposed framework scores clustering with a weighted combination of V-measure and Rand Index (S_clustering = w·V + (1−w)·RI), scores ordering with average Kendall's Tau over multi-page documents, and combines them into S_packet = α·S_clustering + β·S_ordering. The classical comparison uses three progressively stricter exact-match accuracies: Page Level Accuracy (document type only), Page+Split Accuracy (type and group ID), and Page+Split+Order Accuracy (type, group ID, and page ordinal). Models were run with temperature 0.0, top-p 0.1, top-k 5, and a maximum output of 4,096 tokens to accommodate structured JSON responses with boundary predictions, type classifications, and page ordinal assignments. A separate appendix stress-tests the metrics on 10 hand-designed edge cases, each with 5 pages (3 "invoice" pages and 2 "form" pages).

Why This Matters

Impact on research: The paper opens a task the authors argue is largely unaddressed — it states that existing benchmarks such as RVL-CDIP focus exclusively on single-page classification and that recent multimodal LLMs have not been systematically evaluated on document packet splitting. It supplies both source datasets and a scoring framework, and explicitly frames boundary detection as the open problem.

Real-world applications (as described in the paper):

• Healthcare and insurance claims: packets mixing prescription records, laboratory results, physician notes, and insurance forms from disparate systems, with page duplication, combined patient records, and omitted documentation.
• Mortgage and property transactions: packets containing deeds, mortgages, liens, and tax records from multiple sources, with pages scanned randomly, duplicated, or omitted.
• Transportation and logistics: proof-of-delivery packets containing rate confirmations and bills of lading photographed by drivers at delivery sites, causing page shuffling and poor image quality.
• Legal document processing: bundles of multiple contracts or agreements of the same type, where boundaries must be inferred from subtle cues such as signature pages, headers, or semantic shifts.

Industry relevance: The authors are affiliated with Amazon Web Services, and the benchmark targets document-intensive commercial domains where, per the paper, extraction accuracy directly affects operational efficiency and decision-making quality. The evaluation framework is described as model-agnostic, designed to assess capabilities rather than promote specific commercial systems.

Future Directions

• Improve boundary detection. The paper identifies boundary detection as the primary bottleneck (0.56 to 0.90 clustering scores) and points to cross-page consistency modeling and hierarchical document representations as directions for future research.
• Extend beyond the current data distribution. The ethical considerations section acknowledges limited generalizability to low-resource language documents and non-Western formats, since the source RVL-CDIP collection is predominantly English-language material from North American business and governmental sources.
• Adapt the metric weights to specific use cases. The framework exposes parameters (w, α, β) to emphasize clustering or ordering by task priority, and the paper notes that either the clustering or ordering metrics can be substituted by other relevant metrics.
• Determine whether frontier models close the gap. The existing results cover five models and five benchmarks; the paper does not report how other models or task-specific training would perform, leaving the value of domain-specific models as an open question.

Target Audience

Researchers and practitioners working on visual document understanding, intelligent document processing, and multimodal LLM evaluation, especially those building document ingestion pipelines for healthcare, insurance, finance, mortgage, legal, and logistics workflows. It is also relevant to dataset and benchmark designers interested in clustering- and ranking-based evaluation metrics as alternatives to exact-match accuracy.

Authors’ abstract

Document understanding in real-world applications often requires processing heterogeneous, multi-page document packets containing multiple documents stitched together. Despite recent advances in visual document understanding, the fundamental task of document packet splitting, which involves separating a document packet into individual units, remains largely unaddressed. We present the first comprehensive benchmark dataset, DocSplit, along with novel evaluation metrics for assessing the document packet splitting capabilities of large language models. DocSplit comprises five datasets of varying complexity, covering diverse document types, layouts, and multimodal settings. We formalize the DocSplit task, which requires models to identify document boundaries, classify document types, and maintain correct page ordering within a document packet. The benchmark addresses real-world challenges, including out-of-order pages, interleaved documents, and documents lacking clear demarcations. We conduct extensive experiments evaluating multimodal LLMs on our datasets, revealing significant performance gaps in current models' ability to handle complex document splitting tasks. The DocSplit benchmark datasets and proposed novel evaluation metrics provide a systematic framework for advancing document understanding capabilities essential for legal, financial, healthcare, and other document-intensive domains. We release the datasets to facilitate future research in document packet processing.