LAVA: Logic-Aware Validation and Augmentation Framework for Large-Scale Financial Document Auditing

AI summary

Overview

Research area: Applied AI for document intelligence — specifically validation and auditing of semi-structured financial documents (payroll, tax, mortgage, compliance) using multimodal large language models (MLLMs).

Technical level: Intermediate. The architecture is modular and conceptually accessible, but the evaluation metrics, ablation design, and enterprise framing assume some familiarity with document AI pipelines and LLM prompting strategies.

Scope: The paper introduces LAVA, a four-stage, backbone-agnostic pipeline for verifiable financial document validation, evaluated on roughly 1,000 real Canadian mortgage-application documents against a library of several dozen expert-curated rules.

What This Paper Is About

Financial institutions process millions of documents daily — bank statements, tax forms, invoices, paystubs — that vary in layout, arrive as noisy scans or non-standard PDFs, and must satisfy embedded business rules. Current pipelines built on general-purpose MLLMs struggle here: they hallucinate, produce untraceable reasoning, adapt poorly across regulatory schemas, and consume escalating token budgets at enterprise scale. The paper's goal is a modular, auditable framework that combines layout-preserving extraction, domain-aware metadata augmentation, and explicit symbolic/arithmetic verification to deliver accurate, reproducible, and cost-efficient validation under compliance-critical conditions.

Key Contributions

1. Task and system formalization. The authors formalize financial document validation as a multi-document reasoning task — described as largely absent from existing benchmarks — and instantiate it in LAVA, a modular framework for auditable validation of semi-structured financial documents. The task is defined as a human-in-the-loop copilot setting where inputs are a document set (scanned PDFs or images) and a natural-language validation intent, and outputs include supporting documents, retrieved rules, a binary Pass/Fail label, and an explanation trace.

2. Controllable hybrid reasoning with auditability. They design a bifurcated validation framework that routes each rule to either an Arithmetic Processor (formula generation plus deterministic external execution, with a checker-LLM fallback loop) or a Symbolic Reasoner (direct delegation to a general-purpose model for semantic and contextual rules), unifying factual and contextual templates with symbolic and arithmetic tasks.

3. Comprehensive evaluation framework. They propose task-specific metrics — Factual Hallucination Rate, Numerical Infidelity Rate, and Edge Case Handling error rate, plus Token Cost — with explicit emphasis on suppressing false positives, since the paper identifies false positives as a critical industrial concern causing costly investigations, delays, and loss of trust.

4. Large-scale real-world benchmark. They evaluate on a proprietary dataset sampled from Canadian mortgage application documents: around 1,000 scanned PDFs or images spanning multiple document types, with ground-truth outcomes manually annotated by domain experts and complemented by manual audits from business collaborators.

Main Findings

• LAVA achieves the lowest failure rates across all metrics and rule categories. In Table 1, LAVA records the lowest score in every reported cell of Factual Hallucination Rate, Numerical Infidelity Rate, and Edge Case Error Rate across the five rule categories, compared with VLM + Field-Level OCR, LLM + Field-Level OCR, and LLM + Enhanced OCR.

• Largest gains appear in multi-step logic and constrained arithmetic reasoning. The paper reports that LAVA reduces hallucination and numerical errors by over 10% compared to the best baseline in multi-step logic reasoning (C3) and constrained arithmetic consistency checking (C5). For example, Factual Hallucination Rate on C5 falls from 0.30 (LLM + Field-Level OCR), 0.28 (VLM + Field-Level OCR), and 0.15 (LLM + Enhanced OCR) to 0.05 for LAVA, and on C3 from 0.33 / 0.31 / 0.30 to 0.18.

• Near-zero numerical infidelity in three of four applicable categories. LAVA records approximately 0% Numerical Infidelity Rate in three of the four categories where it is reported (0.01 on C2, 0.00 on C4, 0.00 on C5). C1 is marked N/A for all pipelines.

• Substantial edge-case improvement. On Edge Case Error Rate, LAVA scores 0.02 (C2), 0.17 (C3), 0.11 (C4), and 0.08 (C5), versus baselines ranging up to 0.92 on C3 and 0.89 on C5. The paper notes that these edge cases represent about 10%–25% of document–rule pairs.

• Structured knowledge extraction is the most critical module. In the ablation (Table 2, percentage of responses failing to exactly match ground truth), removing Knowledge Extraction raises C2, C3, and C5 failure rates to 0.65–0.67 — over twice LAVA's error in multi-step logic (C3, LAVA at 0.28) and nearly tenfold in constrained arithmetic consistency checking (C5, LAVA at 0.07).

• Downgrading representation quality degrades performance progressively. Markdown KE (C2 0.25, C3 0.54, C5 0.55) and plain-text KE (C2 0.63, C3 0.58, C5 0.48) both fall short of full LAVA. The paper reports that Markdown underperforms plain text specifically on C5 due to weaker preservation of columnar alignment in tables.

• Information Augmentation primarily supports logical reasoning. Removing IA raises C3 error from 0.28 to 0.45, which the authors attribute to metadata and domain cues helping models resolve field semantics and link conditions across steps.

• The Arithmetic Processor drives numerical fidelity. Removing it leaves C1–C3 unchanged but drives C4/C5 error to 0.10/0.56, reflecting the limits of unconstrained LLM reasoning in arithmetic tasks.

• Token efficiency. LAVA cuts input tokens by 25%–45% versus the VLM and LLM baselines using Field-Level OCR. Across all ablations, full LAVA adds under 3k tokens per rule (approximately $0.009 with Claude 3.7 or approximately $0.006 with GPT-4.1).

• Transfer to public data. On a sampled subset of DocVQA documents with five manually defined rules each, LAVA also achieved strong performance. No numerical results are reported for this experiment; the paper states only that the framework "readily transfers to public data."

• Baseline configurations underperform. The paper attributes the weaker performance of VQA and QA-style baselines to their limited handling of layout variability and domain-specific rules in a compliance-oriented validation task.

Methodology in Plain English

The authors build LAVA as two parallel pipelines connected by bidirectional constraints, deliberately avoiding a single opaque end-to-end model.

Document-processing track. Documents are first normalized (correcting OCR errors, rotation, and skew) and each one is classified into a predefined document type using a lightweight image-based classifier (TinyViT). Document-level metadata such as date and coverage period is pulled out using template-guided NER with regex patterns and rule-based heuristics. The second stage converts each document into an HTML-like structured markup built from layout and OCR signals — using LayoutParser, LayoutLMv3, AWS Textract, and Tesseract OCR — augmented with proximity-based grouping of fragmented tokens and preservation of visual regions (charts, stamps, signatures) as image patches. Headers, footers, boilerplate, and placeholders are then pruned to cut token usage and sharpen model attention.

Rule-grounding track. Rules are retrieved from a predefined library using a sentence encoder (Sentence-BERT) for semantic relevance, with temporal constraints parsed from the validation intent by a lightweight LLM. Each retrieved rule is then classified as symbolic or arithmetic by a lightweight LLM query, avoiding brittle heuristics, and dispatched to the matching engine. Document types extracted from retrieved rules feed back to prune irrelevant documents, and document types from the document side feed back to eliminate inapplicable rules — a closed-loop filter that reduces reasoning load on both sides.

Validation. The Arithmetic Processor never lets the LLM compute. The model generates only a Python-executable formula, which a deterministic external engine evaluates; a secondary "checker" LLM audits alignment between rule and formula, and on mismatch the formula is regenerated conditioned on the previous incorrect version as a negative example, with a maximum retry count of 2. The Symbolic Reasoner handles everything else by direct delegation to a general-purpose model. Prompts are constructed dynamically with template logic (e.g., Jinja) in a zero-shot, step-wise meta-prompting style, avoiding few-shot examples that would fail to generalize across heterogeneous documents.

Experimental setup. All baselines, LAVA, and LAVA's ablation variants use Claude 3.7 Sonnet as the validation component, with a maximum response length of 5120 tokens and reasoning enabled with a budget of 1024 tokens. Only off-the-shelf tools requiring no custom training or dataset-specific fine-tuning are used, specifically to make results portable and reproducible. Evaluation is conducted at the rule level rather than the intent level, with document type metadata provided to all pipelines so that document and validation retrieval are excluded from the comparison.

Why This Matters

Impact on research. The paper argues that enterprise-grade validation is a distinct and emerging frontier of document intelligence, not merely an extension of perception or QA. Existing benchmarks are dominated by perceptual and question answering tasks and rarely probe symbolic rule enforcement, cross-field consistency, or multi-step logical coherence. By formalizing validation as a multi-document reasoning task with task-specific metrics focused on false-positive suppression, the work defines a new evaluation target for the field.

Real-world applications:

• Payroll auditing, including verification of regular pay, year-to-date amounts, CPP/QPP contributions, and employment insurance values against Canada Revenue Agency guidelines.
• Tax compliance checks on forms such as T4 documents, including Social Insurance Number presence and deduction verification.
• Loan and mortgage underwriting, where income thresholds and cross-document consistency determine approval decisions.
• Fraud detection and general compliance review, where the audit trail and rule attributions support investigator workflows.

Industry relevance. The design explicitly targets production constraints: modular decoupling isolates failure points for targeted validation, modules can be debugged, updated, or replaced without systemic disruption, and the traceable Pass/Fail output with explanation traces supports human accountability. The reported 25%–45% input token reduction and under-3k-token overhead per rule address the cost and latency pressures of high-volume, time-critical workflows. The work was conducted at BMO Financial Group, and ground truth was validated by domain experts and business collaborators.

Future Directions

• Handling noisier, less-structured documents. The conclusion states an aim to move beyond the semi-structured formats tested toward documents with less inherent structure.
• Learning rule generalization across formats and domains. The authors want the framework to generalize validation rules rather than relying on a fixed, curated rule library and predefined document types.
• Removing dependency on provided document-type metadata. Experiments provide document type labels to all pipelines for fair comparison and exclude document and validation retrieval from evaluation; production deployment would require validating that retrieval stage end to end.
• Bias auditing for production deployment. The ethical considerations section states that any production deployment would necessitate rigorous, ongoing audits for demographic bias to ensure fair and equitable outcomes.

Target Audience

This paper is most useful to applied AI and machine learning engineers building document-processing pipelines in regulated industries; financial technology practitioners in compliance, audit, underwriting, and fraud roles who need traceable automation; and researchers working on document understanding, LLM verification, and hallucination control who want a task formulation and metric suite oriented toward validation rather than perception or QA. It is also relevant to technical decision-makers evaluating whether to replace monolithic MLLM prompting with modular, auditable architectures in high-stakes workflows.

Authors’ abstract

Financial document validation in production, such as payroll auditing, tax compliance, and loan underwriting, demands exceptional accuracy, consistency, and reproducibility under strict enterprise constraints. In practice, documents arrive with heterogeneous layouts and formats, semantically rich and context-dependent content, and embedded business rules that current pipelines struggle to process reliably. We introduce LAVA (Logic-Aware Validation and Augmentation), a modular, backbone-agnostic pipeline built on multimodal large language models, that integrates a four-stage design: document-rule retrieval, layout-preserving information extraction, auxiliary metadata enrichment, and auditable symbolic/arithmetic verification. LAVA supports robust rule grounding, fine-grained error attribution, and consistent, traceable end-to-end execution, capabilities essential for high-stakes deployment. Evaluated on a large real-world benchmark with diverse financial documents and dozens of expert-curated validation rules, LAVA outperforms baselines in hallucination control and edge-case handling while maintaining efficient token usage, demonstrating practicality for high-volume, time-critical validation.