FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computation

AI summary

Overview

Research area: Multimodal large language model (MLLM) evaluation, document understanding, and numerical reasoning in the financial domain (Computer Vision / NLP).

Technical level: Intermediate. The paper is a benchmark-and-evaluation study; the concepts (multimodal inputs, retrieval-augmented generation, program-of-thought prompting) are accessible to readers with basic familiarity with LLMs, and the results are reported as accuracies and error breakdowns rather than requiring deep architectural knowledge.

Scope: FinMMDocR is a 1,200-question bilingual (Chinese/English) benchmark that tests whether multimodal models can reason numerically over long, visually rich financial documents while inferring the real-world scenarios those questions assume.

What This Paper Is About

Existing financial question-answering benchmarks are mostly text-only, use short or single-image inputs, state all conditions explicitly, and tolerate generous rounding errors. Real financial analysts, by contrast, must infer assumptions from a market context, dig through hundred-page research reports containing tables and candlestick charts, and produce numerically exact answers. FinMMDocR is built to close that gap: it packages 1,200 expert-annotated numerical reasoning problems together with 837 real financial documents, evidence page annotations, golden Python solutions, and exact answers, then measures how far current models fall short.

Key Contributions

1. Scenario awareness at scale. 57.9% of questions require implicit assumptions rather than given conditions, drawn from 12 financial scenario categories (e.g., Portfolio Management). Questions carry 1.9 scenarios on average, and 66.2% are scenario-driven, forcing models to reason about context the way an analyst would.

2. A deep, bilingual document corpus. 837 Chinese and English documents across 9 financial categories (e.g., Company Research, Financial Engineering, Futures & Options), averaging 50.8 pages and 38.8k tokens each, with professional visual elements modelled as rendered page images rather than extracted text.

3. Multi-step computation with strict grading. Problems demand 11 reasoning steps on average (5.3 extraction steps — 1.0 textual and 4.3 visual — plus 5.7 calculation steps), with 65.0% requiring cross-page evidence (2.4 evidence pages on average). Evaluation enforces a 0.2% error tolerance covering units, percentages, and decimals, plus golden Python solutions for every item.

4. Broad empirical evaluation. 11 proprietary and open-source MLLMs on image input, 15 LLMs on OCR-extracted text input, plus 6 embedding models and 5 Agentic RAG frameworks, with fine-grained analysis by scenario count, document length, evidence position, and step count, and a 100-case error analysis.

Main Findings

• No model reaches expert level. The strongest MLLM, OpenAI o4-mini-high, scores 58.0%; no configuration in either the image or text setting exceeds 60%. Open-source models lag badly, with the best open-source text model (DeepSeek-R1) at 40.0% and the best open-source MLLM (Qwen2.5-VL 72B) at 12.9%.

• Reasoning-enhanced models consistently win. Across both input settings, the top proprietary performers are all reasoning-enhanced variants, and DeepSeek-R1 leads its open-source group, suggesting reinforcement-learning-style reasoning transfers to this task.

• Vision is harder than text for most current models. Most MLLMs score below the OCR+LLM text pipelines on the same questions, indicating bottlenecks in handling long multimodal inputs. OpenAI o4-mini-high is the exception, beating its own text counterpart. Accuracy gaps between MLLMs (nearly 30 points) are far wider than between OCR+LLM models (under 12 points), implying visual understanding varies much more than language understanding.

• Scenario complexity degrades accuracy. Every model loses accuracy as the number of scenarios in a question grows, since more scenarios require more assumptions and associations.

• Extraction, not arithmetic, is the bottleneck. Under Program-of-Thought prompting, errors trace back primarily to locating and extracting the right figures. In the error analysis of 100 o4-mini-high failures, 78 involved document understanding errors, 44 knowledge reasoning errors, 33 scenario awareness errors, and only 5 pure numerical calculation errors.

• Document length tolerance separates the leaders. OpenAI o4-mini-high and Gemini 2.5 Pro Preview hold performance steady as document length grows, while other models drop sharply; average evidence position correlates with document length, making retrieval harder in longer files.

• Visual retrieval beats text retrieval, but complex agents underperform simple ones. Visually oriented RAG methods outperform text-only approaches, with ColQwen2.5 best. Agentic RAG frameworks (M3DocRAG, ViDoRAG, MDocAgent, SimpleDoc, VRAG-RL) consume more tokens and time yet score lower. ViDoRAG achieves broader retrieval coverage on hard questions but introduces numerical errors such as invalid significant figures; VRAG-RL, built on a 7B base model, performs poorly.

Methodology in Plain English

The benchmark was assembled from two sources. First, the authors took 600 English questions from the existing DocMath-Eval CompLong set — all 300 from its testmini split plus 300 selected for diversity and complexity — completed the missing solutions, answers, and evidence pages, retrieved the corresponding documents, rendered every page as an image, and removed the original text so that models must read the document visually. Second, they wrote 600 entirely new Chinese questions grounded in 385 authorized Chinese research reports, constructing realistic financial scenarios and knowledge-intensive problems with the help of two strong MLLMs.

Quality control was human-centred. Two leading models (Gemini 2.5 Pro Preview and Claude 3.7 Sonnet) produced candidate annotations, which a team of 15 finance master's students and 2 CFA-certified experts cross-reviewed, selected from, and refined, with a third expert arbitrating disagreements. Of 759 generated samples, 159 were discarded; of the surviving 600, 494 required edits, mostly to evidence pages (451).

For evaluation, models receive either document page images or Tesseract-OCR text and answer using Program-of-Thought prompts, which ask the model to write executable Python rather than do arithmetic in its head. Answers are graded correct only if they fall within 0.2% of the reference value. Image-input experiments vary how many pages are merged into a single input (300, 50, 15, or none) and at what resolution (full, 3840-pixel, or 1920-pixel long edge); text-input experiments vary context cut-offs. Success rates were then sliced by scenario count, document length, evidence position, and extraction versus calculation step counts, and a sample of 100 failures was manually categorised.

Why This Matters

Impact on research. The paper supplies a harder, more realistic target than prior financial QA benchmarks, which mostly test extraction plus simple arithmetic under explicit conditions. By showing that nearly four in five failures stem from document understanding rather than calculation, it redirects effort toward retrieval and visual grounding instead of further arithmetic training. Its RAG comparison also provides evidence that elaborate multi-agent pipelines can cost more and deliver less than a single strong visual retriever.

Real-world applications:

• Equity and industry research automation — drafting or checking analyst conclusions from full research reports containing charts, tables, and multi-page data trails.
• Risk and portfolio analysis — questions that require inferring assumptions about market conditions before computing exposure, returns, or hedging figures.
• Financial reporting and compliance review — extracting and reconciling figures scattered across long Chinese and English filings.
• Document AI assistants for analysts — retrieval systems that must locate intermediate variables never stated in the question itself.

Industry relevance. The work was produced with Hithink RoyalFlush Information Network, a financial data provider, which signals direct commercial interest in automating document-heavy analyst workflows. Benchmark results also give model developers a concrete diagnostic: long-document visual grounding, not language fluency or arithmetic, is the frontier.

Future Directions

• Fix the extraction bottleneck. Since document understanding errors dominate failures, better page-localisation, table and chart parsing, and cross-page evidence chaining are the highest-leverage improvements.
• Design retrieval that knows what a question needs. Current agents miss pages holding intermediate variables that the question never names; closing that gap may require reasoning-aware retrieval rather than semantic similarity alone.
• Scale reinforcement learning for visual retrieval. VRAG-RL's weak showing is attributed to its 7B base model, leaving open whether larger RL-trained visual agents change the picture.
• Make agentic pipelines worth their cost. Iterative agents currently add latency and token consumption without accuracy gains, so measuring and reducing error propagation through multi-stage pipelines is an open engineering problem.
• Close the human–model gap and widen coverage. Comparing against expert human performance, and extending the scenario and document categories, would sharpen the benchmark's diagnostic value.

Target Audience

Researchers building and evaluating multimodal LLMs; benchmark designers in document understanding and numerical reasoning; financial NLP and document-AI engineers; and practitioners at financial data or investment firms assessing whether current models can support analyst-grade workflows.

Authors’ abstract

We introduce FinMMDocR, a novel bilingual multimodal benchmark for evaluating multimodal large language models (MLLMs) on real-world financial numerical reasoning. Compared to existing benchmarks, our work delivers three major advancements. (1) Scenario Awareness: 57.9% of 1,200 expert-annotated problems incorporate 12 types of implicit financial scenarios (e.g., Portfolio Management), challenging models to perform expert-level reasoning based on assumptions; (2) Document Understanding: 837 Chinese/English documents spanning 9 types (e.g., Company Research) average 50.8 pages with rich visual elements, significantly surpassing existing benchmarks in both breadth and depth of financial documents; (3) Multi-Step Computation: Problems demand 11-step reasoning on average (5.3 extraction + 5.7 calculation steps), with 65.0% requiring cross-page evidence (2.4 pages average). The best-performing MLLM achieves only 58.0% accuracy, and different retrieval-augmented generation (RAG) methods show significant performance variations on this task. We expect FinMMDocR to drive improvements in MLLMs and reasoning-enhanced methods on complex multimodal reasoning tasks in real-world scenarios.