Financial Instruction Following Evaluation (FIFE)

AI summary

Overview

• Research area: Machine Learning, specifically evaluation of language models for instruction following in financial analysis.
• Technical level: Intermediate.
• Scope: The paper introduces FIFE, a high-difficulty benchmark of 88 human-authored financial prompts with verifiable constraints, and evaluates 53 language models to measure how well they follow complex, interdependent financial instructions.

What This Paper Is About

Language models often fail when instructions are complex, interdependent, and require precision, which is especially risky in finance. The paper asks how well current models can follow difficult financial analysis instructions and introduces FIFE to measure that capability with fine-grained, verifiable scoring. It then compares proprietary, open-weight, and open-source models to identify performance gaps and support future reinforcement learning research in finance.

Key Contributions

1. FIFE benchmark: A high-difficulty benchmark of 88 human-authored prompts designed to test language model instruction following for financial analysis tasks.
2. Verifiable constraint system: A verification framework with chainable, verifiable constraints that provides fine-grained reward signals for evaluating model compliance.
3. Large zero-shot evaluation: An assessment of 53 models across proprietary, open-weight, and open-source categories under strict and loose scoring criteria.
4. Open-source release: Public release of the dataset and code to encourage reinforcement learning and evaluation research in the financial domain.

Main Findings

• Top open-weight model outperforms leading proprietary system: The best open-weight model scores 76.1 strict / 79.5 loose, surpassing the leading proprietary system at 65.9 strict / 70.5 loose.
• Open-source models lag significantly: The best open-source models score 45.5 strict / 48.9 loose, far behind the top open-weight and proprietary systems.
• Clear performance hierarchy among top models: The leading open-weight model ranks above the leading proprietary model, while the best open-source models trail both.
• No model achieves perfect compliance: Even the strongest models fail to satisfy all of FIFE's complex, interdependent requirements.
• Strict scoring remains harder: Models consistently score lower under strict compliance than under loose compliance, showing that exact instruction adherence is difficult.

Methodology in Plain English

The researchers created 88 finance-focused prompts that require models to follow multiple connected instructions, such as analyzing data while meeting formatting, calculation, and constraint requirements. They built an automatic verification system that checks whether each constraint is satisfied and can chain checks together, producing detailed reward signals rather than one overall score. They then tested 53 language models in a zero-shot setting, without extra examples or fine-tuning, and compared proprietary, open-weight, and open-source models using strict and loose scoring. The dataset and code were released for other researchers to use.

Why This Matters

Impact on research: FIFE provides a difficult, finance-specific benchmark and verifiable reward signals that can support reinforcement learning, model evaluation, and high-stakes instruction-following research. It shows that strong general models still struggle with interdependent instructions and precision.

Real-world applications:

• Automating financial analysis and report generation from complex, multi-step instructions.
• Checking regulatory and compliance documents against multiple precise constraints.
• Assisting investment research by following strict data analysis, calculation, and formatting requirements.
• Improving risk assessment workflows where models must adhere to detailed disclosure and calculation rules.

Industry relevance: Financial institutions need reliable, precise AI systems. FIFE helps identify which models are trustworthy for high-stakes financial tasks and provides a foundation for training models to better follow finance-specific instructions.

Future Directions

• Use FIFE's verifiable constraints as reward signals to fine-tune or reinforce models for financial instruction following.
• Expand the benchmark with more prompts, financial subdomains, languages, and real-world document types.
• Investigate why top open-weight models outperform leading proprietary systems on this task and whether the pattern holds across other financial tasks.
• Develop more advanced constraint chaining, partial-credit scoring, and human-in-the-loop evaluation to capture nuanced financial reasoning.

Target Audience

AI and NLP researchers working on evaluation, instruction following, and reinforcement learning; financial AI practitioners and model developers; compliance, risk, and regulatory technology teams exploring language model adoption; and anyone interested in benchmarks for high-stakes, precision-critical language tasks.

Authors’ abstract

Language Models (LMs) struggle with complex, interdependent instructions, particularly in high-stakes domains like finance where precision is critical. We introduce FIFE, a novel, high-difficulty benchmark designed to assess LM instruction-following capabilities for financial analysis tasks. FIFE comprises 88 human-authored prompts and employs a verification system with chainable, verifiable constraints for fine-grained reward signals. We evaluate 53 models (proprietary, open-weight, open-source) in a zero-shot setting. Our key findings reveal a clear performance hierarchy: the top open-weight model (76.1 strict / 79.5 loose) surpasses the leading proprietary system (65.9 strict / 70.5 loose), while the best open-source models lag significantly (45.5 strict / 48.9 loose). However, even top-performing models struggle with FIFE's complex requirements, failing to achieve perfect compliance. We release our dataset and code as an open-source resource to promote research in Reinforcement Learning for the financial domain.