Neurosymbolic Information Extraction from Transactional Documents

AI summary

Overview

Research area: Natural Language Processing and Document AI — specifically information extraction (IE) from transactional documents (invoices and receipts), combining large language models with symbolic rule-based validation ("neurosymbolics").

Technical level: Intermediate. Readers should be comfortable with LLM prompting and fine-tuning concepts, JSON schema-style structured output, and document IE evaluation metrics (F1, edit distance), but the paper is written in an accessible style with a fully explained pipeline.

Scope (one sentence): The paper proposes and evaluates a three-layer symbolic validation framework (syntactic, task, domain) applied to LLM-generated extractions from invoices and receipts, using a new 53-field schema with arithmetic constraints, relabeled datasets, and a knowledge-distillation study.

What This Paper Is About

Large language models can extract structured fields from invoices and receipts without task-specific training, but they lack symbolic reasoning, can confidently produce incorrect information, and require substantial compute. The paper's goal is to recover some of the interpretability and data-efficiency of rule-based systems while keeping the generalization of neural models, by defining a strict transactional-document schema with arithmetic relationships between fields and using it to filter LLM predictions. The filtered, high-quality predictions are then used as labels for distilling smaller models.

Key Contributions

1. A comprehensive schema for transactional documents consisting of 53 fields organized into three main objects in one-to-many relationships (global, line items, sub line items), with well-defined arithmetic relationships between fields, including fields linked to nested fields by summation.
2. Two relabeled datasets: the existing CORD and SROIE datasets were reannotated to fit the new schema and published as CORD_TD and SROIE_TD, with constraint-satisfaction rates of 96.5% (CORD) and 95.5% (SROIE).
3. A neurosymbolic methodology that combines LLMs with three levels of validation — syntactic, task-level, and domain-level — to filter candidate extractions and produce high-quality labels for downstream knowledge distillation.
4. Experimental evidence across zero-shot, fine-tuned upper-bound, sampling, and distillation settings showing that the validation layers select subsets of predictions with improved F1, nTED, and constraint validity.

Main Findings

• Each validation layer raises F1, at the cost of retention. In zero-shot experiments, every additional layer of symbolic validation increased F1 across all models and datasets. However, the domain filter left Ministral-8B with only 25.0% (CORD_TD) and 11.7% (SROIE_TD) of documents, versus 44.0% and 19.9% for Pixtral-12B.
• Filtering happens only at the task and domain levels. The models were capable of generating syntactically correct outputs, so the syntactic filter removed essentially no documents in most configurations (one exception: 99.1% remaining for Ministral-8B on SROIE_TD).
• Task filtering improves F1 but barely moves document accuracy. The paper reports that the task filter removes a varying number of documents with a corresponding F1 increase, yet its impact on document accuracy remains marginal.
• Valid documents do not imply perfect F1. After domain filtering, all remaining documents are valid and the "Valid" column reads 100.0, but F1 does not reach 100%, because models that extract only a limited number of fields leave the rest to programmatic inference — and the more fields a model predicts, the greater the chance of a constraint violation.
• SROIE_TD is harder than CORD_TD. Base zero-shot F1 for Pixtral-12B was 69.0 on CORD_TD versus 40.5 on SROIE_TD; Ministral-8B scored 60.0 versus 39.4. The authors attribute this to more intricate tax structures and unrelated surrounding text in SROIE receipts, whereas CORD content outside line items and global fields was blurred out, yielding cleaner OCR.
• A strong upper bound exists. A Ministral-8B fine-tuned on ground-truth labels reached base F1 of 95.7 (CORD_TD) and 80.6 (SROIE_TD), rising to 98.1 and 90.6 after domain filtering. The task and domain layers improved the selected subset's F1 by approximately 10 points for SROIE and 2.4 for CORD, while filtering out far fewer samples than in the zero-shot setting.
• Sampling trades F1 for recall. Drawing multiple samples at temperature 1.0 reduced base F1 by 11.7 points on CORD (69.0 at 1 sample to 57.3 at 16) and 23.8 points on SROIE (40.5 to 16.7). The retained-document percentage nearly doubled for CORD at the domain level (44.0% to 83.0%), and the SROIE domain filter's retained percentage increased by 157% (19.9% to 51.2%). The authors conclude sampling should not be expected to raise F1 at any validation level but can be used consistently to improve recall.
• Domain-filtered labels are better distillation targets. Fine-tuning on domain-filtered Pixtral-12B predictions outperformed fine-tuning on unfiltered predictions. On CORD_TD, Ministral-8B improved F1 from 69.7 to 77.4 — higher than the 74.0 F1 measured on the filtered predictions used for training; Llama-3.1-8B went from 69.2 to 72.6 and Qwen-2.5-14B from 70.4 to 72.4. On SROIE_TD, improvements ranged from 2.1 to 2.5 F1 points (Ministral-8B 41.2 to 43.7, Llama-3.1-8B 42.1 to 44.2, Qwen-2.5-14B 43.9 to 46.2). Document accuracy on SROIE_TD remained 0.0 in all configurations.
• Number formatting is the main parsing bottleneck. Periods and commas used interchangeably, compounded by OCR noise on those characters, are the leading cause of constraint non-satisfaction. The authors nonetheless report achieving a parsing rate that allows more than 95% of documents to be correctly parsed.

Methodology in Plain English

An LLM is given a document — the OCR-extracted text and, for models that support it, the document image — plus a predefined extraction schema and task instructions. It must emit JSON matching the schema, with all keys present even when values are null. The prompt includes guidelines such as extracting only values present verbatim, never inferring information, using each OCR value at most once, and assuming amounts are gross when no tax indication is given. No few-shot prompting was used.

Because generated text may not parse, the predictions pass through three filters. The syntactic filter parses the output as JSON against a schema specification, tolerating the extra explanatory text or backticks models tend to add by parsing only the text between the first and last opening braces. The task filter checks that every extracted value appears verbatim as a contiguous span in the OCR text, guarding against hallucination. The domain filter converts string values to decimals (the "implicit" output, versus the raw "explicit" output), iteratively infers any fields that can be derived from known values, and then checks arithmetic constraints among totals, taxes, and line item amounts. Default values (typically 0, or 1 for quantities) are assumed where none is provided. A document is valid when every constraint that can be evaluated — meaning all fields in its equation have values — holds, with a relative tolerance of 0.5% to absorb implicit rounding. Additional requirements include at least one line item and a tax rate between 0 and 1.

Evaluation uses the implicit micro F1 score, the normalized Tree Edit Distance (nTED), the percentage of extractions satisfying domain constraints, full document accuracy, and the percentage of documents remaining after each filter, with metrics computed strictly on the remaining subset. Models were run in 16-bit floating point with vLLM on a single NVIDIA H100 NVL using greedy top-1 decoding unless stated. The main models are Ministral 8B Instruct 2410 (text-only) and Pixtral 12B 2409 (text and images); the distillation experiments also use Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct, fine-tuned with LoRA at rank and alpha 128, 5% dropout, learning rate 5e-5, 10% warmup, cosine schedule, 4 epochs, and 4 evaluations per epoch on a random 20% validation split of the training set, keeping the checkpoint with lowest validation loss. The relabeled datasets were produced by mapping existing labels to the new schema and correcting constraint violations and missing fields, done by a single annotator over a few days.

Why This Matters

The work shows that a relatively small amount of domain knowledge — arithmetic relations already known about invoices — can be turned into a validation signal that measurably improves extraction quality and, importantly, produces training labels good enough to improve smaller models offline. This matters for research because earlier zero- and few-shot LLM document IE has lagged fine-tuned performance, and the authors state that to their knowledge previous studies had not investigated domain-specific knowledge for generating high-quality labels for distillation. It also offers a reusable benchmark: a strict, published schema plus two relabeled datasets.

Real-world applications:

• Insurance claims processing: the paper explicitly motivates the travel-insurance case, where invoices accompany a claim and expected structure (itemized expenses, travel dates, provider names) supplies the prior knowledge used for validation.
• Accounts payable and expense automation: verifying that subtotals, taxes, and totals on incoming invoices are arithmetically coherent before payment or booking.
• Receipt digitization and expense reporting: extracting line items and totals from photographed receipts for reimbursement or accounting systems.
• Document data pipelines more broadly: generating verified labels to distill cheaper, faster extraction models when annotation budgets are limited.

Industry relevance is direct: the authors are affiliated with Shift Technology and La Rochelle University's L3i laboratory, the code for the schema, inference, and constraints is published, and the distillation setup uses LoRA fine-tuning of small (8B–14B) models, which is a practical cost profile for production deployment.

Future Directions

• More domain-invariant constraints. The authors note the methodology relies heavily on domain-specific schemas and constraints, limiting initial adaptability to other document types, and suggest developing constraints that transfer without the strict schema. The task filter is already domain-agnostic in principle, applying to visual IE generally.
• Automatic generation of domain-specific schemas and constraints. Since domain constraints add significant value, the authors propose investigating how to generate them automatically.
• Better numeric parsing. Handling interchangeably used periods and commas, potentially with another language-model call, is named as a target for future work given the parsing-related constraint failures.
• OCR error correction. OCR errors were not addressed; the authors note they would not significantly affect findings but could be managed with OCR error-correction techniques. A related open question is that a model correcting OCR from the image is not permitted to deviate from the OCR text under the task filter.
• Schema generalization. The most limiting factor identified is that the schema allows only a single tax rate besides non-taxable amounts, which did not cause problems on the annotated datasets but might on others.

Target Audience

Researchers and practitioners in document understanding, document AI, and NLP information extraction, especially those working with invoices, receipts, and other transactional documents. It is also useful for applied machine-learning engineers who need to turn LLM extractions into reliable training labels or production pipelines, since the validation layers, the published schema and code, and the LoRA distillation setup are directly reusable. Readers interested in neurosymbolic AI more broadly will find it a concrete case study of applying symbolic constraints to neural predictions in an open-ended domain where a clean "win/loss" signal, as in games, does not exist. A background in LLM generation and structured output is helpful but not required to follow the core argument.

Authors’ abstract

This paper presents a neurosymbolic framework for information extraction from documents, evaluated on transactional documents. We introduce a schema-based approach that integrates symbolic validation methods to enable more effective zero-shot output and knowledge distillation. The methodology uses language models to generate candidate extractions, which are then filtered through syntactic-, task-, and domain-level validation to ensure adherence to domain-specific arithmetic constraints. Our contributions include a comprehensive schema for transactional documents, relabeled datasets, and an approach for generating high-quality labels for knowledge distillation. Experimental results demonstrate significant improvements in $F_1$-scores and accuracy, highlighting the effectiveness of neurosymbolic validation in transactional document processing.