BankMathBench: A Benchmark for Numerical Reasoning in Banking Scenarios

AI summary

Overview

Research area: Natural Language Processing — domain-specific benchmark construction and numerical reasoning evaluation for large language models (LLMs) in retail banking.

Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with LLM evaluation terminology (zero-shot prompting, supervised fine-tuning, LoRA/QLoRA, tool-augmented reasoning, pass@1 accuracy).

Scope: This paper introduces BankMathBench, a 13,839-problem Korean-and-English benchmark covering deposits, savings, and loans across three difficulty levels, and reports zero-shot, supervised fine-tuning (SFT), and tool-augmented SFT results for a range of open-source and closed-source LLMs.

What This Paper Is About

LLM-based chatbots are increasingly used for digital banking customer service, but they make systematic errors on everyday banking calculations such as maturity payout estimation, product comparison under different interest rates, and interest calculation under early repayment. Existing benchmarks do not capture this gap: mathematical datasets focus on educational arithmetic and algebra, while financial benchmarks target financial statements, securities analysis, or certification exams rather than routine deposit, savings, and loan queries. The paper's goal is to build a benchmark that reflects realistic banking tasks and to show that fine-tuning on it improves LLM numerical reasoning in this domain.

Key Contributions

1. BankMathBench, described by the authors as the first comprehensive benchmark for numerical reasoning in everyday banking, organized into three difficulty levels—basic (single-product reasoning), intermediate (multi-product comparison), and advanced (multi-condition scenarios)—spanning deposits, savings, and loans.

2. A fully automated, parameterized data generation pipeline (question generation, solution generation with automatic verification, and reasoning generation) that produces question–answer–reasoning triplets and allows key variables such as interest rates, tax policies, and currency units to be adjusted for portability across national financial contexts.

3. A rigorous validation and filtering process, combining two-tiered automatic consistency checking of solutions against each other plus manual review by two banking professionals with over three years of retail banking experience.

4. An empirical demonstration that fine-tuning open-source LLMs on BankMathBench substantially improves both formula generation and numerical reasoning, and that combining the fine-tuned models' formula generation with external calculator tool calling yields large additional gains.

Main Findings

• Closed and specialized models fail on real banking tasks. In zero-shot evaluation (Table 1), AdaptLLM scored 0.4 (Basic), 0.0 (Interm.), 0.0 (Adv.); Fin-o1 scored 3.2, 3.3, 0.3; DeepSeek-Math-Instruct scored 50.4, 1.4, 0.1; Qwen2.5-Math-Instruct scored 61.2, 5.0, 2.5; Claude Sonnet-3.5 scored 66.7, 9.9, 10.0; Gemini-2.5-flash scored 71.2, 11.8, 10.3; and GPT-4o scored 67.8, 14.4, 6.3.

• Accuracy collapses as difficulty rises for open-source models. On the Korean dataset, open-source models averaged 18.0% (basic), 0.8% (intermediate), and 0.5% (advanced) in zero-shot evaluation.

• Fine-tuning produces consistent gains. Qwen3-8B improved over zero-shot by 42.0 percentage points in Korean and 44.6 percentage points in English on the basic-level dataset, while DeepSeek-Math-Instruct-7B improved by 29.3 and 14.5 percentage points.

• Tool-augmented fine-tuning yields the largest gains over SFT. Relative to SFT, tool-augmented fine-tuning increased performance by 21.3 (basic), 71.1 (intermediate), and 62.8 (advanced) percentage points on Korean, and by 24.1, 72.7, and 63.4 percentage points on English. The abstract reports average accuracy increases with tool-augmented fine-tuning of 57.6%p (basic), 75.1%p (intermediate), and 62.9%p (advanced) over zero-shot baselines.

• The intermediate level benefits most from external tools. Because intermediate questions require generating multiple formulas per product, SFT improved formula generation but models still struggled to aggregate multiple computed results into one correct outcome; tool calling relieved that bottleneck.

• Fine-tuning stabilizes performance across model sizes. On the Korean basic-level dataset, zero-shot results had high variance (μ = 18.0, σ = 15.8), which dropped markedly after fine-tuning (μ = 54.4, σ = 9.5), a pattern also observed on the English dataset.

• Numerical precision improves even without tools. Error analysis of Qwen3-8B on the advanced dataset showed zero-shot absolute error of 2.65 (Korean) and 2.65 (English) with error ratios of 100.00 and accuracies of 2.75 and 0.50; after SFT the absolute errors were 2.05 and 2.12, error ratios 1.28 and 1.89, and accuracies 1.75 and 1.75; with tool-augmented SFT the absolute errors and error ratios fell to 0.00 and accuracy rose to 79.95 (Korean) and 73.94 (English).

• Scale helps but is not decisive. Larger 7B/8B backbones generally performed best, yet DeepSeek-R1-Distill-Qwen-1.5B performed comparably to math-specialized backbones such as DeepSeek-Math-7B and Qwen-Math-7B, and Kanana-nano-Base-2.1B matched or surpassed Kanana-1.5-Instruct-8B. LLaMA-3.2-Instruct-1B, which lacks official Korean language support, performed very poorly but improved considerably when scaled to 3B.

• Language specialization matters more at higher difficulty. Korean-specialized models such as the Kanana series achieved superior results on the Korean dataset, while other multilingual models were relatively consistent across languages.

Methodology in Plain English

The authors built the dataset with a three-stage automated pipeline and then had humans check it.

Question generation. Questions were generated with GPT-4o using prompt templates organized into four fields (role, requirements, example, output format). Difficulty was stratified by formula complexity. Basic questions cover a single real product and specify all conditions—amount, interest rate, duration, payment frequency, and interest calculation method—so the answer is computable. Intermediate questions involve two products chosen from realistic pairings (deposit vs. deposit, deposit vs. savings, savings vs. savings, loan vs. loan), with unrealistic pairings such as deposit vs. loan excluded, and each states the comparison criterion explicitly. Advanced questions add multi-condition features such as taxation, early withdrawal options, preferential rates, variable interest periods, and prepayment fees, with feasibility guaranteed by capping early withdrawal rates at the nominal annual rate.

Solution generation and filtering. Basic questions were solved by GPT-4o using two fully computable representations (mathematical expressions and LaTeX). Intermediate and advanced questions were solved by GPT-4o and o1-mini producing executable Python code. Only pairs whose results agreed across the two solutions were kept, giving large-scale automatic verification.

Reasoning generation. GPT-4o generated stepwise reasoning from the question, solution, and answer, with reasoning in <think> tags, final answers in <answer> tags, computable expressions in <calc> tags, and their results in <result> tags so that external tools can be invoked on the expressions.

Data augmentation. Numerical augmentation introduced realistic irregular values because models perform well on round numbers (e.g., 2.5 or 10) but struggle with values like 2.93 or 3.1. Random masking of basic-level questions removed certain numeric conditions so models must infer them; augmented questions account for approximately 47% of the intermediate dataset.

Evaluation setup. The dataset was split 8:2 into training and test sets, with evaluation by exact-match accuracy (pass@1) and normalization of equivalent numerical or currency expressions. SFT used 4-bit quantized LoRA (QLoRA) with rank r = 16, scaling factor α = 16, learning rate 2 × 10⁻⁴, and batch size 4 for 10 epochs on four NVIDIA RTX 3090 GPUs. In tool-augmented SFT, equations inside <calc> tags were automatically routed to an external calculator and the results inserted into <result> tags. Models evaluated include LLaMA-3, DeepSeek-R1-Distill, Qwen2.5-Instruct, Phi-3.5-Mini-Instruct, and Kanana, plus Qwen2.5-Math, DeepSeek-Math, and Qwen3, all up to 8 billion parameters, alongside Claude Sonnet-3.5, Gemini-2.5-Flash, and GPT-4o.

Why This Matters

For research, BankMathBench targets a gap the authors argue prior work overlooked: mathematical benchmarks (e.g., GSM8K, MATH) and financial benchmarks (e.g., FinQA, TAT-QA, FinanceMath, FinReasoning, Fin-R1, XFinBench, AlphaFin) do not cover everyday deposit, savings, and loan reasoning. The paper's results show that math-specialized and financial LLMs transfer poorly to these tasks, which motivates domain-specific benchmark design rather than relying on general math training.

Real-world applications:

• Digital banking chatbots that answer customer questions about maturity payouts, product comparisons, and loan interest accurately.
• Product comparison support, helping customers weigh a deposit against a savings product using explicit comparison criteria such as interest rates or total payouts.
• Loan servicing calculations, including variable-rate loans, equal principal versus equal principal-and-interest repayment, and prepayment fees.
• Cross-national deployment, since the parameterized pipeline lets institutions adjust interest rates, tax policies, and currency units for different financial contexts.

For industry, the paper frames accuracy in these computations as directly tied to institutional trustworthiness, where reliability and factual correctness are paramount. The finding that lightweight or domain-adapted models can rival larger ones suggests practical deployment options, and the large tool-augmented gains point to pairing fine-tuned LLMs with calculator backends.

Future Directions

• Broadening product coverage. The authors state that funds and insurance are not yet included, which may introduce product-specific biases in fine-tuned models, and plan to extend the dataset via the automated pipeline.

• Direct evaluation of reasoning quality. Performance was assessed mainly through numerical correctness and error rates; the authors note that reasoning quality was not directly evaluated and propose explicit assessment of reasoning processes.

• Reducing the reliance on tool calling. SFT alone left exact computation accuracy at 1.75% on the advanced dataset even as relative error ratios fell to 1.28% (Korean) and 1.89% (English), raising the question of how much internal numerical consistency can be improved without external tools.

• Extending tool-integrated learning. The authors call for further exploration of tool-integrated learning approaches for more robust numerical reasoning, and for developing banking-specialized reasoning models.

Target Audience

This paper is most useful to NLP and machine learning researchers working on numerical reasoning, domain-specific benchmarks, and tool-augmented LLM training; to financial-industry technologists and data scientists building or evaluating banking chatbots and customer-service assistants; and to practitioners who need a concrete reference for how well current LLMs perform on everyday banking calculations in Korean and English.

Authors’ abstract

Large language models (LLMs)-based chatbots are increasingly being adopted in the financial domain, particularly in digital banking, to handle customer inquiries about products such as deposits, savings, and loans. However, these models still exhibit low accuracy in core banking computations-including total payout estimation, comparison of products with varying interest rates, and interest calculation under early repayment conditions. Such tasks require multi-step numerical reasoning and contextual understanding of banking products, yet existing LLMs often make systematic errors-misinterpreting product types, applying conditions incorrectly, or failing basic calculations involving exponents and geometric progressions. However, such errors have rarely been captured by existing benchmarks. Mathematical datasets focus on fundamental math problems, whereas financial benchmarks primarily target financial documents, leaving everyday banking scenarios underexplored. To address this limitation, we propose BankMathBench, a domain-specific dataset that reflects realistic banking tasks. BankMathBench is organized in three levels of difficulty-basic, intermediate, and advanced-corresponding to single-product reasoning, multi-product comparison, and multi-condition scenarios, respectively. When trained on BankMathBench, open-source LLMs exhibited notable improvements in both formula generation and numerical reasoning accuracy, demonstrating the dataset's effectiveness in enhancing domain-specific reasoning. With tool-augmented fine-tuning, the models achieved average accuracy increases of 57.6%p (basic), 75.1%p (intermediate), and 62.9%p (advanced), representing significant gains over zero-shot baselines. These findings highlight BankMathBench as a reliable benchmark for evaluating and advancing LLMs' numerical reasoning in real-world banking scenarios.