VERDICT: Visual Extraction-Result Decoder with Integrated Confidence and Traceable Evidence

AI summary

Overview

Research area: Computer vision / document AI — schema-guided field extraction, visual evidence grounding, and confidence calibration for Vision–Language Models (VLMs).

Technical level: Advanced. The paper assumes familiarity with AUROC, expected calibration error (ECE), selective prediction (precision–coverage), LoRA fine-tuning, and VLM internals.

Scope (1 sentence): VERDICT is a plug-in framework that attaches to an unmodified black-box document extractor, uses a small open-weight VLM to localize the evidence region for every extracted field in a single pass, and trains a calibrator that outputs two separate confidences — one for the value and one for the evidence box.

What This Paper Is About

Enterprises extract structured fields from business documents using frontier multimodal models, and accuracy keeps improving — but two things have not: knowing where on the page each extracted value came from, and knowing how much to trust that value and that location. Existing systems either produce evidence without scoring it (grounding work) or score values without producing evidence (confidence work), and none do both. The paper builds VERDICT to supply both, retrofitted onto an extractor it is not allowed to modify.

Key Contributions

1. A plug-in localization and calibration framework. It attaches to a fixed external extractor, localizes every field of a schema in a single forward pass (rather than one query per field), and estimates separate value and evidence confidence from VLM features plus optional external OCR agreement.

2. Two calibrated targets instead of one, read from a model that emits evidence rather than an answer. Internal states are probed for confidence, and a second probability is calibrated about whether the evidence box itself is correct. Because the VLM is open-weight, its hidden states are readable; the paper reports internal features are worth 0.184 and 0.225 AUROC over output log-probabilities on the two targets in distribution.

3. A mechanism for calibration failure under distribution shift. The paper shows the features that calibrate in distribution are not the ones that calibrate out of distribution, and argues the signal is displaced rather than destroyed (a linear model fitted on target labels recovers fine-tuned evidence confidence at 0.911 AUROC, while the in-distribution-fitted model reads 0.493).

4. A cost-of-recovery analysis and a deployment decision table. Roughly 120 labelled target fields largely restore evidence confidence; the paper also shows why a scalar metric (AUROC/AUPRC under low base rate) can report the least usable setting as the best, so precision is never reported without coverage.

Main Findings

• Fine-tuning the localizer substantially improves evidence, and expands the operating range. Localization recall rises from 0.421 to 0.852 in distribution, from 0.019 to 0.586 on FormNLU, and from 0.421 to 0.699 on VRDU. The frozen model localizes about one field in fifty on FormNLU.

• The frozen setting often collapses to near-zero coverage. At a 0.80 confidence threshold the frozen grounder accepts 29% of fields in distribution, 8% on VRDU, and none on FormNLU; at 0.50 it accepts 43%, 25% and 1%. The fine-tuned grounder accepts 92%, 58% and 77% at 0.50 with better precision in every case, and 77% in distribution at 0.80 with 0.941 precision.

• Internal states beat output log-probabilities in distribution. On the frozen backbone, internals lift value confidence from 0.730 to 0.914 AUROC and evidence confidence from 0.681 to 0.906. The untrained output-probability floor falls below random ordering for evidence in three of six settings.

• The advantage does not survive distribution shift. After fine-tuning, the internals difference (M3 − M1) turns negative on both OOD datasets, and external OCR agreement becomes the dominant signal: M2 reaches 0.778 and 0.765 AUROC versus M4's 0.636 and 0.739.

• Internals and external agreement help different targets. On the frozen in-distribution backbone, external agreement lifts evidence by 0.167 but value by only 0.043; after fine-tuning it helps value more (0.113 rather than 0.043), because checking whether the value lies in the proposed box only matters once boxes are usually right.

• Value confidence does not require correct localization. Among the frozen grounder's 8,854 mislocalized fields, value confidence still reaches 0.918 AUROC — almost the same as on correctly localized fields.

• A base-rate trap inflates one OOD cell. Frozen FormNLU evidence has a 0.019 positive rate, so M4 scores 0.980 AUROC and 0.006 ECE while auto-accepting only 1% of fields.

• Roughly 120 labelled target fields recover most of the lost calibration. Evidence confidence goes from 0.641 to 0.905 on FormNLU and from 0.675 to 0.829 on VRDU — around 80% of the full-label gain. Fitting from scratch is worse at every budget.

• The calibrator's failure axis is the document domain, not the extractor. Swapping Gemini 3 Flash for the weaker Claude Haiku 4.5 leaves evidence confidence almost unchanged; value confidence follows extractor accuracy, but the cost shows up as coverage rather than ranking.

• OCR mapping is a genuine competitor where text is verbatim. The OCR baseline beats the frozen grounder everywhere (by 14 points in distribution, 66 on FormNLU); fine-tuning reverses this in two of three settings, leading by 29 points in distribution, but loses on FormNLU (0.586 against 0.686).

• Smaller backbone reproduces the pattern. Qwen3-VL-2B tracks closely in distribution (value 0.931 against 0.940; location 0.868 against 0.867), though the paper explicitly declines to draw a scale conclusion because depth and size both differ.

Methodology in Plain English

The system is built around three pieces that are deliberately kept separate.

The extractor stays untouched. A fixed black-box extractor (Gemini 3 Flash, called through its API) reads the document and returns a value for each field in the schema, or null where it judges the field absent. Nothing in VERDICT changes this component.

A small open-weight VLM acts as the "grounder." The extracted values are handed to Qwen3-VL-4B-Instruct as a JSON object keyed by field name, and it returns an object of the same shape containing bounding boxes (coordinates in [0, 1000]) for each value, or null where it cannot locate one. Putting the whole schema into one prompt keeps cost at a single pass regardless of schema size and gives each field's coordinates their own token positions inside one generated sequence. The model is used in two conditions — frozen weights, and LoRA fine-tuned for evidence localization (rank 16, α = 16, one epoch) — and every result is reported for both.

Features are harvested from that same pass, in four groups, with each field getting its own copy: output log-probabilities over its coordinate tokens (6 scalars, what a closed API would expose); hidden states at two layers (last layer where visual features are injected, and one near the end); visual encoder features pooled over the predicted region; and external agreement (OCR fuzzy-match against the extracted value plus a count of how many places in the document contain that value). The OCR check runs after localization — the region already exists, and OCR tests it.

A small MLP calibrator with a shared trunk and two independent linear heads outputs the two probabilities. Two sizes are used: MLP A (6 → 64 → 32) for scalar features only, and MLP B (each high-dimensional group projected to 256, then → 512 → 256 → 128) once a high-dimensional group enters. Training minimizes binary cross-entropy summed over both heads with equal weight; class-frequency weighting was tested and rejected. Four configurations are compared: M1 (log-probabilities and null flags), M2 (adds OCR agreement), M3 (adds internals and region features), M4 (all features).

Labels and evaluation. The value label is normalized string match against ground truth; the evidence label is IoU ≥ 0.4 against the true box. Data is trained on DocILE (constrained schema, 33 fields) plus KVP10k (open, page-text field names) so neither model can memorize field names, and evaluated in distribution plus on two unrelated form datasets, Form-NLU (12 fields) and VRDU registration forms (5 fields). Baselines are OCR string mapping and the untrained output probability. Everything is trained with five seeds, and a difference counts as a finding only if it exceeds seed-to-seed spread.

Why This Matters

Impact on research. The paper names a structural gap: localization work produces evidence but never scores it, and confidence work scores values but never produces evidence, because the two literatures do not meet. VERDICT is the first system to calibrate a second target about a region it produced itself, which makes "the value is right" and "the evidence is right" separately auditable. The finding that in-distribution calibration features are displaced rather than destroyed under shift is a concrete, testable claim about transfer, and the base-rate warning about AUROC/AUPRC on low-positive-rate OOD sets applies well beyond this paper.

Real-world applications:

• Accounts-payable and invoice processing, where a field accepted without review must be checkable afterwards by an auditor.
• KYC, onboarding and registration-form intake (the VRDU foreign-agent filing setting), where documents must be routed to humans selectively under cost pressure.
• Claims and substantial-holder notice processing (the Form-NLU setting), where a wrong region is as damaging as a wrong value.
• Any human-in-the-loop review UI that needs to highlight the source region while showing a calibrated risk score for routing.

Industry relevance. The framework is explicitly designed for retrofit: it attaches to an extractor chosen for extraction quality and leaves it alone, and it degrades gracefully along a decision table — fine-tune the grounder if you can, use log-probabilities plus OCR if you have no target labels, warm-start with ~120 labelled fields if you have a few. The demonstration that the calibrator's failure axis is the document domain rather than the extractor is directly actionable for teams swapping vendors.

Future Directions

• Test other localizer architectures. The paper states its study is limited to one localizer model family, and other VLM-based localizer architectures remain untested; the Qwen3-VL-2B comparison is explicitly not a controlled scale study because depth and size both change.

• Predict field existence rather than reading it. Field existence is currently taken from the extractor's abstention rather than predicted, and the authors note value fabrication is too rare in their data to study.

• Evaluate document-level aggregation. Results are field-level, while deployment decisions are per document; the natural rule — accept a document only when the fields it depends on clear both thresholds — is described but not evaluated.

• Make shift-robust calibration cheaper. Recovery still requires labelled target fields and thresholds do not transfer across datasets; a calibration method that detects or adapts to the feature displacement without target labels would remove a recurring deployment cost.

Target Audience

Document AI and information-extraction engineers building production pipelines who need auditable, routable outputs; ML researchers working on VLM reliability, grounding, or selective prediction; and applied scientists and technical product owners evaluating whether a confidence score is trustworthy enough to gate auto-acceptance. Readers without grounding in calibration metrics (AUROC, ECE, precision–coverage) will find the tables dense, but the deployment section and the central argument are accessible.

Authors’ abstract

Enterprises run schema-guided structured field extraction on documents with frontier multimodal models, and accuracy keeps improving as new models and systems arrive. Two things have not kept pace: the evidence for each extracted value, meaning the location of the document it was read from, and how far to trust the value and that evidence. Neither is a component of most extraction systems today, so we build them as a framework that attaches to an existing extractor and leaves it untouched. We present VERDICT, a plug-in framework that uses a small Vision--Language Model (VLM) to ground/localize the evidence once per document over a fixed external extractor's output, then calibrates value and evidence confidence. Fine-tuning the VLM for evidence localization substantially improves visual evidence quality and, more importantly, expands VERDICT's selective-prediction operating range. At a strict confidence threshold $0.80$, VERDICT with the fine-tuned VLM gives usable precision--coverage trade-offs for both value and evidence confidence across the in-distribution and out of distribution datasets, whereas the frozen setting often collapses to little or no coverage. Running an open-weight VLM for localization rather than a closed API makes its internal states available to the calibrator, and they are worth more than output log-probabilities: in-distribution AUROC rises from $0.730$ to $0.914$ for value confidence and from $0.681$ to $0.906$ for evidence confidence. This advantage does not survive a distribution shift. We show that the signal is displaced rather than destroyed, and that labeling roughly $120$ fields from the target dataset takes evidence confidence from $0.641$ to $0.905$ on one dataset and from $0.675$ to $0.829$ on the other.