FCMBench: The First Large-scale Financial Credit Multimodal Benchmark for Real-world Applications

AI summary

Overview

Research area: Multimodal AI evaluation — specifically, benchmarking vision-language models (VLMs) on financial credit-review documents.

Technical level: Intermediate. Readers need some familiarity with multimodal benchmarks, VQA-style evaluation, and precision/recall/F1 metrics, but no deep financial or model-training expertise.

Scope: The paper introduces FCMBench, a self-constructed, privacy-compliant multimodal benchmark of 26 certificate types, 5,198 images and 13,806 VQA samples, and reports an evaluation of 28 state-of-the-art VLMs across perception, reasoning, and robustness settings.

What This Paper Is About

Credit reviewers make loan decisions by looking at images that borrowers upload — ID cards, income certificates, bank statements, property documents — checking whether documents are complete, whether the information on them agrees across documents, and whether the numbers are economically sensible. The paper argues that existing benchmarks do not cover this workflow: general benchmarks (e.g., MME), document benchmarks (e.g., OCRBench, ChartMuseum, WildDoc) and even financial benchmarks (e.g., CFBenchmark-MM, FinMME) either lack credit specificity or rely on limited, fragmentary, privacy-constrained credit data.

The goal is to build a benchmark that mirrors real credit review end to end, without using any real customer data. To do this, the authors manually designed synthetic certificate templates, physically fabricated the certificates, photographed them under controlled real-world conditions, and released the resulting tasks for evaluation.

Key Contributions

1. A credit-specific multimodal benchmark. The paper claims FCMBench is the first large-scale multimodal benchmark dedicated to financial credit. All physical certificates and images were self-created and were not publicly available before, resolving the privacy-driven unavailability of shareable real credit data.

2. An application-oriented evaluation system. The benchmark organises evaluation around 3 perception tasks (Image Quality Evaluation, Document Type Recognition, Key Information Extraction), 4 reasoning tasks (Consistency Checking, Validity Checking, Numerical Calculation, Rationality Review), and 10 robustness challenges, mapped explicitly onto real credit review procedures.

3. A privacy-compliant construction pipeline. Personal information (persona settings, institution and location names, logos, seals) is fully fictionalised. Most logos, emblems and portraits are replaced with AIGC-synthesised alternatives, and a fictional institutional ecosystem (streets, government agencies, banking entities) was built. The 15 English-language driver's licences taken from the synthetic IDNet dataset are the only exception noted.

4. A large multi-model evaluation. 28 state-of-the-art VLMs from 14 AI companies and research institutes were evaluated under a single set of prompts and test samples, providing a comparative picture of commercial versus open-source capability on credit tasks.

Main Findings

• Benchmark scale and composition: FCMBench-V1.1 covers 26 certificate types, 5,198 privacy-compliant images, and 13,806 paired VQA samples. V1.1 extends V1.0 by adding 9 English certificates and optimising some instructions from the previous version.

• Top commercial model: Gemini 3 Pro achieves the best F1 (%) score among commercial models at 65.16, with Gemini 3 Flash close behind at 64.94.

• Top open-source model: Kimi-K2.5 achieves the best score among open-source baselines at 60.58, followed by Qwen3.5-397B-A17B at 53.38.

• Benchmark is discriminative: Model F1 scores span roughly 25–65, with a mean of 44.8 ± 10.3. The authors interpret this wide dispersion as evidence that FCMBench is non-trivial and separates model capabilities.

• Two-stage OCR pipelines struggle: The DeepSeek(OCR + V3.2) pipeline underperforms end-to-end VLMs with an overall average of 34.01, which the authors read as fragility in two-stage approaches.

• Perception versus reasoning gap: Most top-tier models excel at Document Type Recognition — Gemini 3 Flash reaches 95.32 — but performance drops sharply on Key Information Extraction and Image Quality Evaluation, where even the best models struggle to exceed 50. KIE scores for the strongest models are clustered (Gemini 3 Pro/Flash at 43.36/43.99, Kimi-K2.5 at 45.28), while reasoning dimensions show substantially larger variance.

• Robustness degrades strong models: All models show consistent performance drops under robustness challenges relative to Normal Captures. The authors report relative F1 (each model's Normal Captures score scaled to 1.0) and conclude that high-performing models are not inherently robust, with a model performing strongly overall still able to fail systematically on partial crops, multi-document compositions, or severe blur.

• Progress over time: Best-in-class performance climbed from an F1 of roughly 40 to over 60 since early 2025.

• Scaling still matters: Both dense and Mixture-of-Experts (MoE) models show a positive correlation between total parameter count and average F1, with MoE models generally competitive at larger scales.

• Language differences: Separate Chinese and English results show top Chinese open-source models closing the gap with commercial models more in Chinese prompts than in English. For example, Gemini 3 Flash scores 65.15 on Chinese and 69.75 on English, while Kimi-K2.5 scores 64.68 on Chinese and 59.34 on English.

• Token cost is not performance: Analysis of total output tokens indicates higher token usage does not necessarily correspond to better performance; some models reach competitive scores with substantially lower token consumption.

• Human upper bound: In a human study on 100 sampled test instances (approximately 10 per robustness category), humans achieve consistently higher scores across most challenge types, but human scores also degrade under the most severe artifacts, indicating some errors stem from intrinsic information loss rather than model limitations alone.

Methodology in Plain English

The team started by interviewing over 20 senior credit reviewers from commercial banks and financial credit companies to identify which document types and decisions actually matter in the review chain — loan review, income verification, and asset evaluation. This produced 26 certificate categories in Chinese and English, spanning simple cards through complex charts and tables, with both single-image and multi-image inputs.

To avoid using real personal data, they built a synthetic document pipeline. A pool of fictional identities with diverse demographic and economic attributes was created, and each identity received a complete application profile spanning multiple document types, so names, IDs, addresses and financial attributes stay consistent across certificates. Logos, institutional emblems and portraits were replaced with AIGC-synthesised alternatives, and a fictional institutional ecosystem of simulated streets, government agencies and banking entities was constructed. The templates were then physically fabricated — as high-fidelity card replicas or printed on standard A4 paper — and a team of 11 participants photographed the physical documents under ecologically valid conditions, including the ten robustness challenges listed in Table 2 (off-axis viewpoints, uneven illumination, specular reflections, out-of-focus, small ROIs, secondary captures, cluttered background, overlaid watermarks, cropped captures, and multi-doc images).

For scoring, the authors deliberately chose exact-match rather than semantic equivalence, on the grounds that fields such as ID numbers, addresses, bank account identifiers and dates are legally atomic in credit review, so paraphrases are not actionable. Outputs are flattened into order-invariant key-value sets with tailored normalisation (hierarchical key prefixes for dictionaries, tuples for list-of-lists, string and number normalisation), with Numerical Calculation tasks tolerating numerical differences under 2. Set-based precision, recall and F1 are computed and reported on a 0–100 scale, aggregated at instance, subtask, and task level, with task-level scores as a macro-average over subtasks. Robustness is reported as a relative F1 ratio — each artifact's F1 normalised by the F1 on Normal Captures.

Why This Matters

Research impact: The paper positions credit review as a distinct multimodal capability that general, document-understanding, and even financial benchmarks do not isolate. Because the images are self-created and never publicly available before release, data leakage from web-crawled training corpora is largely avoided, which the authors argue makes the evaluation a fairer measure of genuine capability.

Real-world applications:

• Loan document triage: Automatically rejecting low-quality uploads and flagging missing mandatory document types before a human reviewer sees the file.
• Applicant profile automation: Extracting identity, income, and property fields from certificates to populate internal profiles and application forms.
• Cross-document fraud and consistency screening: Checking whether a listing document has its supporting document, whether documents belong to the same person, and whether a receipt reconciles with a bank statement.
• Income and risk sanity checks: Comparing an income certificate against a tax certificate, loan application form or bank statement, and computing derived indicators for collateral coverage.

Industry relevance: The paper argues that open-sourcing FCMBench benefits both sides of the market — financial institutions gain a standard for comparing credit AI models, while academic and fintech researchers gain high-quality data for in-depth study — breaking down the data barriers that currently prevent shared progress. It also argues robustness should be a first-class pre-deployment requirement, recommending capture-side guidelines, automated quality gates, and targeted fine-tuning on dominant failure modes.

Future Directions

• Broaden data forms: FCMBench currently covers only physical certificate photos. The authors note the absence of screenshots, scanned copies and digital certificates, and state that certificate and template categories need further expansion.

• Add non-image modalities: Introducing audio and video is described as a meaningful direction, extending beyond static document images.

• Improve robustness handling: The paper notes that severe acquisition artifacts can make required fields partially unreadable even for humans, and that ground truths were pre-generated without accounting for such artifacts — an open question about how to score fairly when image quality rather than model capability limits accuracy.

• Continue updating the benchmark: The authors state FCMBench will be continuously updated and expanded to match the complexity and practical demands of modern credit-risk workflows.

Target Audience

Researchers and engineers working on multimodal document understanding and VLM evaluation; fintech and banking teams evaluating or deploying models for credit review, KYC-adjacent document checking, or loan automation; benchmark designers interested in privacy-compliant synthetic data pipelines and robustness stress testing; and compliance or risk stakeholders who need to understand where current vision-language models still fail under realistic capture conditions.

Authors’ abstract

FCMBench is the first large-scale and privacy-compliant multimodal benchmark for real-world financial credit applications, covering tasks and robustness challenges from domain specific workflows and constraints. The current version of FCMBench covers 26 certificate types, with 5198 privacy-compliant images and 13806 paired VQA samples. It evaluates models on Perception and Reasoning tasks under real-world Robustness interferences, including 3 foundational perception tasks, 4 credit-specific reasoning tasks demanding decision-oriented visual evidence interpretation, and 10 real-world challenges for rigorous robustness stress testing. Moreover, FCMBench offers privacy-compliant realism with minimal leakage risk through in-house scenario-aware captures of manually synthesized templates, without any publicly released images. We conduct extensive evaluations of 28 state-of-the-art vision-language models spanning 14 AI companies and research institutes. Among them, Gemini 3 Pro achieves the best F1 score as a commercial model (65.16), Kimi-K2.5 achieves the best score as an open-source baseline (60.58). The mean and the std. of all tested models is 44.8 and 10.3 respectively, indicating that FCMBench is non-trivial and provides strong resolution for separating modern vision-language model capabilities. Robustness evaluations reveal that even top-performing models experience notable performance degradation under the designed challenges. We have open-sourced this benchmark to advance AI research in the credit domain and provide a domain-specific task for real-world AI applications.