FiMI: A Domain-Specific Language Model for Indian Finance Ecosystem

AI summary

Overview

Research area: Domain-specific large language model (LLM) adaptation for the Indian financial services sector, with a focus on real-time digital payments infrastructure.

Technical level: Advanced. The paper covers a multi-stage training pipeline (continuous pre-training, instruction fine-tuning, supervised fine-tuning), data curation at trillion-token scale, distributional drift metrics (KL divergence), and perplexity-based evaluation.

Scope: The paper describes the construction, training data, evaluation methodology, and headline results of FiMI (Finance Model for India), a domain-specialized model family built by the National Payments Corporation of India (NPCI) on top of the Mistral Small 24B architecture.

What This Paper Is About

General-purpose LLMs handle broad reasoning well but perform inconsistently on India-specific financial operations, where terms like RRN (Retrieval Reference Number) and UMN (Unique Mandate Number) carry precise operational meaning. The authors report that prompt engineering, chain-of-thought guidance, and role-based prompting alone produced unstable, unpredictable results for NPCI's agentic use cases, forcing constant prompt rewrites whenever workflows changed. The goal of FiMI is to build a model family that internalizes Indian payments terminology, dispute lifecycles, mandate management, and compliance context directly into its weights, while retaining general reasoning ability comparable to models of similar size.

Key Contributions

1. Two released model variants built on Mistral Small 24B. FiMI Base (continuous pre-training) and FiMI Instruct (instruction fine-tuning plus domain-specific supervised fine-tuning), both trained on approximately 68 Billion tokens overall.

2. A documented financial data curation pipeline. The authors describe sourcing across Dolma, FineWeb, NeMo/Nemotron, CommonPile, Ai4Bharat Sangraha, FineMath, StarCoder and ProofPile-2, filtering roughly 30 Trillion tokens down to 2.2 Trillion tokens and then selecting 68 Billion high-quality tokens, with PII anonymization, deduplication, three-level quality classification, and a BERT-style domain classifier.

3. Custom Indian-finance evaluation benchmarks. A HellaSwag-style General MCQ Finance Dataset (2,471 finalized questions), a Finance Reasoning Dataset (around 300 questions, also called CPT Eval), and an MMLU-standard-styled MMLU Finance Validation dataset (5,300 questions after filtering), generated with the Bring Your Own Benchmark (BYOB) framework and human-in-the-loop review.

4. A staged, tool-calling-oriented alignment strategy. Instruction fine-tuning on open corpora, followed by supervised fine-tuning on synthetic multi-turn tool-driven conversations modeling transaction disputes and mandate lifecycle management (view, pause, unpause, revoke), with deployment in NPCI's UPI Help.

Main Findings

• Finance reasoning gain: FiMI Base achieves a 20% improvement over the Mistral Small 24B Base model on a finance reasoning benchmark.
• Tool-calling gain: FiMI Instruct outperforms the Mistral Small 24B Instruct model by 87% on domain-specific tool-calling.
• General capability preserved: FiMI achieves these domain gains while maintaining comparable performance to models of similar size on general benchmarks.
• Foundation model selection rationale: Among four candidates compared, Mistral Small 24B scored 79 on MMLU and 82.9 on IFEval; Llama 3 8B scored 69.4 and 80.4; Mistral Nemo 12B scored 68.0 and 62.9; Mistral 7B scored 60 and 60. Mistral Small 24B was judged best on agentic consistency, multilingual capability, and tool-calling stability, under an Apache 2.0 license.
• Prompt engineering was insufficient: The authors report that the general-purpose model could perform reasonable tool calls but failed to consistently interpret domain-specific terminology, and responses varied across repeated runs.
• Training data composition: The final CPT mixture is dominated by general English filtering (Nemotron-CC-v2 at 810.00 Billion filtered tokens, Dolma v1.7 at 608.50, FineWeb at 337.40, Common Pile v0.1 at 90.00), with much smaller Indian finance contributions (economic and monetary system 0.24, investment schemes 0.11, insurance services 0.15, finance news articles 0.20 Billion tokens).
• Synthetic QA augmentation: 1.8 million question–answer samples (approximately 0.87 Billion tokens) were generated from the 95% training split using on-prem hosted models such as Gemma 3 27B, and introduced in the final stage of CPT.
• Training infrastructure: 8 nodes (64 H100 GPUs), bfloat16 precision, 8192-token context window, peak efficiency (MFU) of approximately 40%, using DeepSpeed ZeRO-3 and FlashAttention-2.
• Reported training schedule: Learning rate 3×10⁻⁶ in Phase C1, 1×10⁻⁵ in Phase C2, and 1×10⁻⁵ in Phase C3, with durations of 3 days, 6.66 days, and 5 hours respectively. The batch size row in Table 6 is not fully reported in the provided content.
• Detailed evaluation result tables for the general benchmarks are not reported in the provided paper content, which ends partway through the training recipe section.

Methodology in Plain English

The team started by testing whether a general-purpose LLM could be made to work through prompt engineering alone. It could not reliably handle NPCI-specific acronyms and workflows, so they decided to adapt a model directly.

They compared four open models and chose Mistral Small 24B because it combined strong general reasoning, decent Hindi and Hinglish ability, stable tool calling, and a permissive license.

Training happened in three phases. First, continuous pre-training: the base model was fed a curated mix of general English, math, code, Hindi, and Indian finance text, drawn from a corpus reduced from about 30 Trillion tokens to 2.2 Trillion filtered tokens and then to 68 Billion selected tokens. Some of this included synthetic question–answer pairs generated from the data to reinforce finance concepts without inheriting raw-text noise.

Second, instruction fine-tuning on broad open-source instruction corpora to keep the model good at following multi-step natural-language instructions and working across English, Hindi, and Hinglish.

Third, domain-specific supervised fine-tuning on a synthetic dataset of multi-turn, tool-driven conversations reflecting real production workflows, such as raising a transaction dispute or managing an autopay mandate.

For evaluation, they used standard benchmarks (ARC Challenge, GSM8K, Humaneval Instruct, MMLU, MMLU Pro, Trivia QA) via the lm-eval-harness framework with few-shot and chain-of-thought prompting, plus their own Indian-finance benchmarks scored by accuracy, conditional log-likelihood, DeepEval with a judge model, KL divergence against the base model, and perplexity. KL divergence values of 1 or above were treated as a warning sign of overfitting. The learning rate was annealed across three curriculum phases to internalize finance knowledge without forgetting general reasoning.

Why This Matters

Research impact: The paper is a detailed case study of domain adaptation at a national infrastructure operator, showing that targeted continued pre-training plus tool-calling supervision can deliver large gains on domain tasks (87% on tool-calling) without obvious sacrifice on general benchmarks. It also contributes Indian finance evaluation datasets, an area with relatively few public benchmarks.

Real-world applications:

• Customer support in UPI Help, handling natural-language questions about transactions and issues in English, Hindi, and Hinglish.
• Transaction dispute workflows, where the model reasons over UPI-specific references and process timelines.
• Mandate lifecycle management, letting users view, pause, unpause, or revoke automatic payments through conversation.
• Compliance and operations assistance for banks, Payment Service Providers (PSPs), and fintechs operating under Indian regulatory structures.

Industry relevance: For fintechs and payment operators, the paper argues that prompt engineering on a general model is not a reliable production strategy in regulated, terminology-dense domains, and that a smaller domain-aligned model can beat a larger generalist on the workflows that matter. The Apache 2.0 licensing of the Mistral Small 24B foundation is presented as an operational enabler.

Future Directions

• Publishing the full results tables for the general benchmarks (ARC Challenge, GSM8K, Humaneval Instruct, MMLU, MMLU Pro, Trivia QA), which are not reported in the provided content, to quantify exactly how much general capability is retained.
• Expanding the Indian finance domain data, which currently contributes very small token counts relative to general English (for example, 0.24 Billion filtered tokens for the economic and monetary system versus 810.00 Billion for Nemotron-CC-v2), and augmenting the topic-modelling leaf nodes that were flagged as undersized.
• Extending beyond the three supported languages (English, Hindi, Hinglish) to other Indic languages, given that Indic-language coverage was a stated selection criterion.
• Reporting finer-grained operational metrics for the deployed UPI Help system, such as tool-call accuracy and consistency across repeated runs, which were the specific failures that motivated moving away from context engineering.

Target Audience

This paper is most useful to applied ML engineers and research scientists working on domain adaptation and tool-calling agents; to engineering and product leaders at banks, Payment Service Providers, fintechs, and payment infrastructure operators evaluating build-versus-prompt decisions; and to researchers interested in Indian-language and Indian-finance benchmarks. Readers without a background in LLM training pipelines will find the data and evaluation sections demanding, though the motivation and results sections are accessible.

Authors’ abstract

We present FiMI (Finance Model for India), a domain-specialized financial language model developed by National Payments Corporation of India (NPCI) for Indian digital payment systems. We develop two model variants: FiMI Base and FiMI Instruct. FiMI adapts the Mistral Small 24B architecture through a multi-stage training pipeline, beginning with continuous pre-training on 68 Billion tokens of curated financial, multilingual (English, Hindi, Hinglish), and synthetic data. This is followed by instruction fine-tuning and domain-specific supervised fine-tuning focused on multi-turn, tool-driven conversations that model real-world workflows, such as transaction disputes and mandate lifecycle management. Evaluations reveal that FiMI Base achieves a 20\% improvement over the Mistral Small 24B Base model on finance reasoning benchmark, while FiMI Instruct outperforms the Mistral Small 24B Instruct model by 87\% on domain-specific tool-calling. Moreover, FiMI achieves these significant domain gains while maintaining comparable performance to models of similar size on general benchmarks.