V-FiLLM: Verified Financial LLM Reasoning Benchmark

AI summary

Overview

Research area: Benchmark construction and evaluation for large language models performing financial reasoning over structured tabular data.

Technical level: Intermediate. The paper is readable without deep mathematical background, but it assumes familiarity with LLM evaluation, LoRA fine-tuning, and terms such as chain-of-thought and adversarial perturbation.

Scope: The paper introduces V-FiLLM, a deterministic pipeline that generates verified financial question-answer benchmarks from executable computation trees grounded in synthetic spreadsheets, and uses it to measure how model accuracy depends on reasoning depth, question breadth, financial concept complexity, context size, multi-turn decomposition, and spreadsheet noise.

What This Paper Is About

Existing financial QA benchmarks are mostly built by human annotation or model-assisted extraction from real documents, which mixes genuine reasoning failures with source-level noise such as formatting irregularities, missing values, and ambiguous phrasing, and makes difficulty hard to control. The authors build a generator that samples executable arithmetic trees over synthetic financial tables, evaluates those trees symbolically to produce ground truth, and renders them into natural-language questions, so every answer is correct by construction and no model sits in the labeling loop. The goal is to give researchers a scalable benchmark that isolates compositional financial reasoning and lets difficulty be dialed up or down explicitly.

Key Contributions

1. A deterministic, annotation-free benchmark generation pipeline. Synthetic financial spreadsheets are decomposed into typed atoms, composed into typed expression trees, and rendered into natural-language questions paired with answers obtained by symbolically evaluating the tree. Because no model generates the labels, items can be produced at arbitrary scale without annotation cost and without inheriting a generator's error rate.

2. Four independently controllable difficulty axes plus derived financial concepts. Computation depth, expression breadth, financial concept complexity, and context size can be varied separately. The pipeline also supports named derived concepts (gross profit, operating income, pretax income, income tax expense, net income, current assets, long-term assets, long-term liabilities) stored as special atoms with hidden executable formula trees, plus a rejection step that prevents an arbitrary arithmetic expression from being confused with a protected named concept.

3. A structurally derived multi-turn benchmark and a suite of adversarial perturbations. Trees with depth d(T) ≥ 1 are converted into ordered dialogue turns, one per parenthesized sub-expression, with per-turn targets read directly from tree nodes and no extra annotation. Four spreadsheet perturbation families (missing values, garbage values, OCR look-alikes, cross-sheet contamination) plus question-level distractor text are applied only to question-irrelevant cells, so ground truth is invariant by construction.

4. A LoRA fine-tuning recipe on verified chain-of-thought traces. Phase 1 generates chain-of-thought traces and keeps only those whose final answer matches the ground-truth value from the expression tree; Phase 2 fine-tunes with LoRA adapters. The authors report improvements on their held-out benchmark and a proof-of-concept transfer result on FinQA.

Main Findings

• Accuracy collapses with reasoning depth. On 100 questions per depth level, Gemma-31B falls from 85.0% at 6 steps to 55.0% at 8 steps on simplified statements, while DeepSeek-v4-Flash falls from 84.0% to 26.0%. After 6 steps DeepSeek-v4-Flash degrades much faster than Gemma-31B, which flips their ranking relative to the easier questions. The abstract states accuracy falls by up to 51% as reasoning depth increases.

• Real filings are harder than simplified statements at the same depth. In the 10-Q setting only depths 1–4 are available. Both models score almost perfectly at 1 or 2 steps but drop to about 60–72% at 3 or 4 steps. At steps 3 and 4, the 10-Q filings are about 20 points harder than the simplified versions, which the authors attribute to messy layouts, footnotes, and mixed units.

• Model rankings are stable across contexts, but the gap widens on simplified statements. On 250 mixed-depth questions, Gemma-31B scores highest on 10-Q filings (98.4%) and simplified statements (97.6%). Llama-3.3-70B and Qwen3.5-9B degrade to 80.8% and 55.2% respectively on simplified statements. Gemma-31B matches or outperforms GPT-OSS-120B and Qwen3.7-Plus across all conditions despite a much smaller parameter count, which the authors read as evidence that specialized training data and robust table parsing matter more than scale.

• Multi-turn decomposition rescues the weakest models. Qwen3.5-9B improves from 55.2% to 86.8% and Llama-3.3-70B rises from 80.8% to 86.8% when questions are restructured into sequential sub-queries. Models already near the performance ceiling shift negligibly.

• Adversarial perturbations hurt, and unit/scale shift is the most damaging single perturbation. Average drops across the six perturbation types are 27.3 points for Gemma-31B and 29.6 points for DeepSeek-v4-Flash on 10-Q filings, and 35.7 and 47.0 points on simplified statements. Unit/scale shift crashes Gemma-31B to 3.0% and DeepSeek-v4-Flash to 22.0% on 10-Q filings, but is far less harmful on simplified statements (57.0% and 38.0%). Combined perturbations drop scores to 26.0% and 17.0% on real documents, shrinking the gap between the two models to 9 points. Adding useless extra information is the mildest test, lowering both models by less than 20 points on 10-Q filings.

• OCR-style corruption is more challenging than missing or irrelevant data. The paper's conclusion highlights OCR-style character corruption as a greater challenge than missing or irrelevant content, underscoring the importance of robust numerical extraction.

• LoRA fine-tuning on verified chain-of-thought traces improves results. The held-out benchmark (n = 90) baseline is 81.1% (73/90) with no LoRA. The best configuration, r = 8, α = 8, dropout 0.10, reaches 85.6% (77/90), tied at dropout 0.15. Accuracy is most sensitive to the α/r ratio, and α/r ≠ 1 erases the gain over baseline. On FinQA, the fine-tuned model answers 32/100 questions correctly versus 27/100 for the baseline, which the abstract describes as outperforming the base model by 5 percentage points.

• Training data verification. Phase 1 retained 608 verified examples from a pool of 688, an 88% retention rate. LoRA adapters used 0.50% trainable parameters.

Methodology in Plain English

The authors do not collect questions from real filings or ask people to write answers. Instead they build the data backwards from arithmetic.

First they simulate two kinds of financial tables. One imitates a typical 10-Q filing; the other is a regularized sheet with consistent column names, units, and time structure, which strips out formatting noise so that purely compositional reasoning can be measured. The regularized sheet holds company-year observations for 15 synthetic companies from 2020 to 2025, with each row carrying metadata (name, ticker, sector, country, exchange, credit rating) and normalized variables such as revenue, costs, expenses, assets, liabilities, equity, cash, receivables, inventories, investments, current liabilities, capital expenditures, dividends, shares outstanding, stock price, and employees. The number of companies is adjustable; the authors use 6 in the prompt to balance report scale against manageable context length.

Each spreadsheet cell becomes a typed "atom" storing a number plus metadata: financial concept, company, fiscal year, unit, and quantity type. The generator then samples a typed binary expression tree whose leaves are atoms and whose internal nodes are operations (addition, subtraction, multiplication, ratios, growth rates, minimum, maximum, average). The sampler is type-aware, so it never builds nonsense such as multiplying a year by a monetary amount. Depth is a direct proxy for the number of reasoning steps: a single lookup is depth 0, a maximum over two companies' assets is depth 1, and deeper trees require composing intermediate results. Balanced trees can be used instead of linear chains to control breadth, with a breadth-to-depth growth factor of 2^i for depth i.

Some concepts, such as gross profit and operating income, are stored as special atoms with hidden formula trees, so their values cannot be looked up and the model must know the finance. A rejection step prevents a sampled tree from silently reconstructing a protected concept's canonical formula without naming it, keeping raw arithmetic separate from financial abstraction.

A binding step then pins the abstract tree to real cells, respecting constraints such as company, fiscal year, and concept. Same-period operations bind to compatible company-year contexts; growth operations pick two distinct years for the same company and concept; aggregations pick a set of reporting years. The bound tree is rendered bottom-up into English using operator-specific templates, with redundant phrasing suppressed when subexpressions share context. Ground truth comes from evaluating the tree directly, so answers are automatically verified. Stored metadata includes the question, numeric answer, full symbolic expression, template expression, requested and realized depths, derived-concept sampling probability, and the list of leaf cells.

Lightweight linguistic augmentation then varies surface form, introducing minor typographical errors, capitalization changes, and informal alternatives such as "What is the value of ..." versus "How would you calculate ...", while preserving the computation and answer. The authors motivate this with prior findings that prompt formatting alone can swing performance by up to 76 accuracy points and that instruction paraphrases change model rankings.

For the multi-turn extension, trees with depth d(T) ≥ 1 are converted so that each parenthesized sub-expression becomes one turn, ordered by computational dependency, with the final turn restating the original question in full so the variant stays comparable to its one-shot counterpart. Per-turn targets are read straight from tree nodes.

For robustness testing, cells that do not appear in the answer's expression tree are perturbed in four ways: emptied (missing values), overwritten with extreme or error values from {-10^7, 10^12, -1, "ERROR"} (garbage values), rewritten with visually similar characters such as 0→O, 1→l, 5→S (OCR look-alikes), and swapped with rows from other companies (cross-sheet contamination). Irrelevant context is also appended at the question level. Because only question-irrelevant cells are touched, any accuracy drop reflects model failure rather than label shift.

Finally, the authors fine-tune Qwen3.5-4B in two phases: generating chain-of-thought traces and keeping only those matching ground truth, then training LoRA adapters on the verified traces. LoRA is applied to the attention projections (q_proj, k_proj, v_proj, o_proj) and feed-forward projections (gate_proj, up_proj, down_proj), keeping original weights frozen.

Why This Matters

Impact on research. The benchmark removes the confound between source-document noise and genuine reasoning failure. Because every item carries its computation trace, researchers can analyze failures by depth, operator type, number of required cells, use of derived concepts, and temporal structure. The multi-turn conversion additionally exposes intermediate computations as first-class evaluation targets, enabling per-depth diagnostics that single-question correctness cannot identify, and the perturbations probe robustness without label shift.

Real-world applications:

• Automated financial statement analysis over filings, where the paper shows real 10-Q style documents are roughly 20 points harder than clean simplified statements at the same depth.
• Table QA systems deployed over noisy, partially missing, or heterogeneously sourced financial data, where the authors report average drops of 27.3 to 47.0 points across perturbation types.
• OCR-dependent processing of scanned statements, where visually similar character substitutions (0→O, 1→l, 5→S) are identified as a more serious challenge than missing or irrelevant data.
• Multi-document consolidation, where cross-sheet contamination produces well-formed but semantically inconsistent tables, and where unit/scale shift on 10-Q filings collapses Gemma-31B to 3.0%.

Industry relevance. The finding that Gemma-31B matches or outperforms larger models such as GPT-OSS-120B and Qwen3.7-Plus suggests that specialized training data and robust table parsing matter more than parameter count for this task. The LoRA results point to targeted, low-cost adaptation as a

Authors’ abstract

While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored. We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction. Trees are evaluated symbolically to obtain ground truth and rendered into natural-language questions, removing any model from the labeling loop, so items can be generated at arbitrary scale without annotation cost and without inheriting a generator's error rate. V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size. By evaluating on open-source models, we find that accuracy falls up to 51% as reasoning depth increases, and up to 47% points under adversarial numerical perturbations, highlighting remaining challenges in robust financial reasoning over tables. We further show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems and outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s), suggesting that targeted, low-cost adaptation is a promising direction for compositional reasoning in financial QA.