FinHopper: A Progressive Multi-Hop Benchmark for Complex Financial Agents with Step-wise Evaluation

AI summary

Overview

Research area: Financial AI agents — benchmark design for multi-hop retrieval, tool use, and step-level evaluation of LLM agents answering investment questions.

Technical level: Intermediate. The paper is readable without deep finance knowledge, but it assumes familiarity with LLM agents, tool-augmented (ReAct-style) search, and standard benchmark metrics.

Scope (one sentence): The paper introduces FinHopper, a 200-question financial benchmark with controlled reasoning depth of 1 to 6 hops, gold intermediate answers, and a process-aware scoring scheme, and uses it to evaluate 20 LLM-agent configurations against human experts.

What This Paper Is About

Financial analysts rarely answer a question by looking up one fact: they identify a company from a comparison, use that company to find an event date, and use that date to retrieve a market indicator. FinHopper formalizes this as answer-conditioned multi-hop search, where each intermediate answer (an entity, date, sector, event, or indicator) becomes a hard constraint on the next query, so one small error derails the whole trajectory while still looking plausible.

Existing financial benchmarks mostly emphasize numerical reasoning over provided documents (FinQA, TAT-QA, ConvFinQA) or judge open financial search by final-answer correctness alone (FinSearchComp). The paper's goal is a benchmark that combines reproducible public-search conditions, strictly chained dependent questions, and step-level supervision so that failures can be localized rather than only counted.

Key Contributions

1. A dependency-preserving financial search benchmark. FinHopper contains 200 questions with controlled depth from 1 to 6 hops across stocks, funds, and macroeconomic facts, spanning 14 financial domains, in matched Chinese and English versions. Its three difficulty levels are L1 (1–2 hops), L2 (3–4 hops), and L3 (5–6 hops). Every K-hop item carries K step-level gold answers, for 572 step-level checking points in total.

2. A scalable, verified construction framework. The authors propose state-transition-based synthesis: a library of 21 atomic financial QA patterns over five answer states (Entity, Time, Sector, Event, Indicator) is composed into chains by a random walk over an answer-state transition graph, so that only valid answer types can condition the next hop. Chains are filtered by in-synthesis verification (single-hop re-derivation and chain-merge critics) and post-synthesis verification.

3. A diagnostic evaluation protocol. A two-stage score gives full credit for a correct final answer and partial credit (capped at 0.5 via α = 0.5) for recovering gold intermediate steps when the final answer is wrong, on top of micro-average and macro-average aggregation across hop depths. This exposes partial reasoning progress that final-answer-only metrics hide.

4. A multi-stage quality-control stack. Post-synthesis checks combine an ensemble of five frontier LLMs under ReAct (a sample is accepted only if at least 3 agents agree, with expert arbitration on disagreement) and human verification by 16 financial experts using public search engines rather than the internal database, plus double-blind review of 30% of items sampled from each complexity level, with a reported 95% pass rate.

Main Findings

• Public search is necessary but not sufficient. Without search tools, all evaluated models fail almost completely, scoring below 8.1%; with tools, scores rise substantially but remain far from human performance.

• The best model still trails experts substantially. Claude-4.5-Sonnet-thinking achieves the top Micro Average of 67.3%, versus 81.7% for the 20 human financial experts under the same public-search setting. Human Macro Average is 78.7%, and human latency is reported as 324 (Time column, seconds per query).

• The gap widens with depth. On 6-hop questions, humans retain 64.4% while the best model reaches only 31.1%. Humans score 94.0% on 1-hop items, compared with 81.3% for Claude-4.5-Sonnet-thinking and 88.5% for DeepSeek-V3.2-no-thinking.

• Each model has an "agentic capability boundary." Most models show a cliff-like decline at a specific hop depth. DeepSeek-V3.2 performs well on simple tasks but its score drops by half after 4-hop; Gemini-3-Pro stays competitive but hits its limit at 6-hop. DeepSeek-V3.2-thinking falls from 86.3% at 1-hop to 9.3% at 6-hop.

• Thinking mode helps mainly on long chains. The gain from reasoning-enhanced inference grows with depth, which the authors attribute to more reliable planning and state tracking rather than more memorized financial facts. On 1–2 hop questions the extra reasoning budget often yields limited benefit, and it costs latency (GPT-5.2-thinking at 121s vs. GPT-5.2 at 108s).

• Step-level scoring reveals hidden competence. On 6-hop tasks, DeepSeek-V3.2 scores only 1.0 on the final-answer score s_final but 8.3 on the process score s_process, showing it can recover intermediate steps even when the final answer fails.

• Results are stable under repetition. Mean and standard deviation over repeated evaluations show fluctuations within 0.7–1.8 points, supporting the main ranking trends despite the moderate benchmark size.

• Failures cluster in four categories. Sampling trajectories and annotating the first major error yields decomposition and planning failures (38%), tool misuse (31%), hallucination propagation (17%), and information extraction failures (14%).

• English results follow the same trend. On four representative thinking models, Gemini-3-Pro-thinking reaches a Micro Average of 65.2% and Macro Average of 59.8%; GPT-5.2-thinking 61.7% / 55.0%; Kimi-k2-thinking 57.3% / 54.2%; GLM-4.7-thinking 48.8% / 42.3%.

Methodology in Plain English

The authors build the benchmark backwards from how financial search actually works. They start by defining the answer types a hop can produce — Entity, Time, Sector, Event, Indicator — and noting that not every answer type can meaningfully constrain the next question (an indicator value, for example, is a terminal answer and cannot drive a further hop). They encode the legal moves as a directed state transition graph with normalized transition probabilities, then sample a target depth K and a valid sequence of answer states from it.

Each sampled state points to one of 21 hand-designed single-hop question patterns, such as ranking several entities by an indicator on a given date, or extracting an event date. A generator LLM with access to internal financial database tools instantiates a concrete question, and a verifier that sees only the question text must re-extract the tool parameters, re-run the tools, and confirm the answer matches.

Verified single-hop questions are then merged into longer chains by answer substitution: the earlier answer is embedded as a required constraint inside the new question, while the released question hides all intermediate answers. Two critics review each merge — one on the question (checking that intermediate answers are not leaked, that the original intent survives, that entities, dates, and metrics are unambiguous, that no out-of-scope entities are introduced, and that premises are logically coherent) and one on the answer (checking that it follows strictly from the merged question and is complete and evidence-faithful).

After synthesis, two independent checks gate release: an ensemble of five frontier LLMs under ReAct with majority agreement (at least 3) and expert arbitration for disagreements, and verification by 16 financial experts through public search engines. Once released, the benchmark is evaluated by running models as ReAct agents with public search under a maximum iteration budget of 42, with the internal database never exposed.

Scoring is hierarchical: a normalized final-answer match earns 1.0; otherwise the response earns up to 0.5 depending on how many gold intermediate answers its output entails.

Why This Matters

Impact on research. FinHopper reframes financial agent evaluation from "did the model get the number right" to "where exactly did the reasoning chain break." The step-level gold answers and process score give other researchers a diagnostic instrument rather than a single leaderboard number, and the state-transition synthesis recipe is a reusable way to generate dependency-preserving multi-hop data instead of hand-writing questions.

Real-world applications (potential).

• Automating equity research evidence collection, where an analyst must chain company identification, reporting periods, event dates, and market indicators.
• Fund and ETF due diligence, tracing holdings, managers, sectors, and performance metrics across linked queries.
• Macroeconomic fact assembly, where definitions and reporting windows must match exactly or the answer is silently wrong.
• Compliance and disclosure monitoring, where the correct announcement date determines which subsequent filing or figure is relevant.

Industry relevance. The measured gap — 67.3% for the best agent versus 81.7% for human experts, and 31.1% versus 64.4% on 6-hop items — tells practitioners that current agents are usable for shallow lookups but not yet reliable for the multi-step evidence chains that real investment workflows require. The failure breakdown, dominated by planning (38%) and tool misuse (31%), points at where engineering effort would pay off, and the latency figures (up to 324 seconds for humans, 121 seconds for GPT-5.2-thinking) make the cost trade-off of thinking mode explicit.

Future Directions

• Extend to higher complexity and broader data. The authors plan to incorporate broader heterogeneous data sources and synthesize reasoning chains of greater complexity than the current 1–6 hops.
• Test generalization beyond the Chinese market. The main large-scale evaluation is on Chinese questions with data predominantly from the Chinese market, leaving global-market generalization unverified.
• Improve query naturalness. Auto-generated questions, despite human verification, lack the ambiguity and informality of real user questions — a naturalness gap that could make benchmarks easier than deployment.
• Move beyond factual reasoning. The benchmark excludes subjective tasks such as sentiment analysis and multi-modal interpretation, both of which matter for real financial analysis and would require new construction and scoring machinery.

Target Audience

Researchers and engineers building LLM agents for finance or tool-augmented retrieval; benchmark designers interested in process-level rather than outcome-only evaluation; and quantitative or AI teams at financial firms assessing whether current agents can support multi-step research workflows. It is also useful for evaluation specialists who want a concrete model of how to verify synthesized multi-hop data through ensemble checking and expert public-search review.

Authors’ abstract

Financial agents are expected to answer factual investment queries through open search, but financial multi-hop retrieval differs from general multi-hop QA in a crucial way: each intermediate answer is an exact typed constraint, such as an entity, date, event, or numerical indicator, that determines the next query. This strict answer-conditioned dependency makes small intermediate errors cascade into plausible but wrong search trajectories. We introduce FinHopper, a benchmark for complex financial agents with controlled reasoning depth from 1 to 6 hops, covering stocks, funds, and macroeconomic facts, with gold intermediate answers for process-level evaluation. To construct FinHopper at scale, we design a state-transition-based synthesis framework that composes verified atomic financial QA patterns into dependency-preserving chains, followed by targeted in-synthesis verification, model-ensemble checking, and public-search validation by financial experts. Experiments with 20 LLM-agent configurations show that public search tools are necessary but insufficient: performance drops sharply as hop depth increases, the best model remains far below human experts, and process-aware scoring reveals partial reasoning progress hidden by final-answer-only evaluation. The benchmark is available at https://github.com/justajustin/FinHopper/.