FinForesight: A Diagnostic Backtesting Benchmark for Deep Research Systems in Financial Forecasting

AI summary

Overview

Research area: Evaluation of AI “deep research” systems (LLM agents that search, retrieve and reason over evidence) applied to financial forecasting; sits at the intersection of NLP benchmark design, agent evaluation, and financial NLP.

Technical level: Intermediate. The framework and motivation are accessible, but the scoring formulas (Spearman correlation, Brier skill, clipped utility) and the temporal-regime design assume some familiarity with forecasting evaluation and LLM agents.

Scope (one sentence): The paper introduces FinForesight, a time-controlled backtesting benchmark spanning the Chinese, U.S., and Hong Kong markets that tests whether controlled historical (“post-cutoff, resolved”) backtests can stand in for live forecasting evaluation of deep research systems, and diagnoses where apparent forecasting skill comes from.

What This Paper Is About

Deep research systems look promising for financial forecasting, but measuring whether they can actually forecast is hard: historical backtests may reward outcomes the model already memorized during pretraining, or evidence retrieved after the forecast date may leak the answer, while live forecasting gives trustworthy scores only after events resolve. The authors build a benchmark that keeps retrieved evidence restricted to what was public at the forecast date and then compares three temporal regimes — pretraining-era events (T1), post-cutoff resolved events (T2), and unresolved live events (T3) — to see whether the fast, immediately scorable T2 backtests track true live performance.

Key Contributions

1. A multi-market, multi-task benchmark. FinForesight covers the Chinese, U.S., and Hong Kong markets with four outcome-based forecasting tasks — Valuation (VAL), Return Ranking (RANK), Event Direction (EVT), and Fundamentals (FUND) — plus two capability diagnostic tasks, Counterfactual (CF) and Calculation (CALC), whose scores are reported separately from the forecasting score. The suite contains 1,119 records: VAL (244), RANK (383), EVT (138), FUND (294), CF (30), and CALC (30).

2. A time-controlled, harness-compatible evaluation framework. Instances are generated relative to each model’s declared knowledge cutoff, and retrieved evidence is audited against the forecast date. The framework supports multiple deep research harnesses and three retrieval policies: search without temporal checks, point-in-time (PIT) control, and PIT plus an LLM-based leakage filter.

3. A three-regime temporal comparison design. The benchmark uses forecast dates before 2024-06-01 for T1 and after 2025-07-31 for T2 (subject to each model’s cutoff), and compares T1 vs. T2 to probe historical-knowledge effects and T2 vs. T3 to test backtest-to-live agreement.

4. Attribution and robustness diagnostics. Beyond aggregate scores, the paper analyzes how scores respond to retrieval policy, and how predictions respond to prompt rewrites and numerical perturbations, using metrics including Prompt Change Rate, Perturb Change Rate, and Score Delta.

Main Findings

• Historical backtests flatter models. T1 scores exceed T2 scores in 25 of 27 comparisons across the three markets, including all RANK and FUND comparisons (Table 2), suggesting each of the three analyzed models has memorized some historical outcomes. Pretraining-era backtests can therefore make recall look like prediction.

• Stronger retrieval controls make backtests track live results. In a matched one-week U.S. comparison, the mean absolute T2–T3 gap across nine model–task pairs is 0.1203 with search alone, 0.0618 with PIT control, and 0.0285 with PIT plus the LLM leakage filter — approximately 76% below search alone, with smaller gaps in all nine comparisons.

• Task-specific strengths differ from aggregate rankings. Gemini 3.1 Flash-Lite achieves the highest Overall and EVT scores across all three research harnesses, while Phi-4 consistently leads VAL. In Table 1 the highest reported Overall score is 0.7099 (Gemini 3.1 Flash-Lite with GPT-Researcher).

• The finance-specific harness does not consistently win. FinanceHarness achieves the highest mean Overall score across the six shared models, but no single research harness dominates: DeepSeek V4 Flash and Gemini 3.1 Flash-Lite perform best with GPT-Researcher.

• Reasoning skill does not imply forecasting skill. Claude Sonnet 4.5 achieves perfect CF and CALC scores with Claude Code but shows lower Overall forecasting performance than with FinanceHarness.

• No model leads in all three markets. All six models score highest in the US; DeepSeek V4 Flash narrowly leads in CN (0.6825), while Gemini 3.1 Flash-Lite leads in US (0.7480) and HK (0.6698).

• Predictions shift, but scores stay stable. Across four model–harness configurations, Prompt Change Rates (16.83%–23.67%) exceed Perturb Change Rates (7.00%–9.17%), yet the mean signed Score Delta stays below 0.0026 in magnitude for every configuration.

• Live performance survives the strictest controls. In prediction attribution (Table 4), both models retain meaningful live scores — 0.5805–0.5926 on EVT and 0.7194–0.8563 on FUND — with filtered T2–T3 gaps of only 0.0043 to 0.0267 and input Score Deltas below 0.004 in magnitude.

Methodology in Plain English

The team defined three time zones using two dates: the model’s knowledge cutoff and the date the benchmark was run. Anything resolved before the cutoff is T1 (the model may simply remember it). Anything whose forecast date falls after the cutoff but whose outcome resolves before the run date is T2 (scorable immediately, with no pretraining exposure). Anything still unresolved at the run date is T3 (true live forecasting, but you must wait for the answer).

For every question, the system gets only information that was verifiably public on the forecast date. The benchmark is built from templates across six task families and three markets, drawing on Yahoo Finance prices, filings from SEC EDGAR, CNINFO, the Shanghai and Shenzhen exchanges and HKEXnews, and macroeconomic releases from the U.S. Bureau of Labor Statistics, China’s National Bureau of Statistics, and Hong Kong’s Census and Statistics Department. Source availability is checked independently of label collection, and labels are held out so only the evaluator sees them.

The experiments plug different language models into different agent “harnesses” — finance-specific (FinanceHarness), general research (GPT-Researcher, Open Deep Research), and coding agents (Claude Code, Codex) — giving 22 model–harness configurations over eight models. Each run uses one of three retrieval policies: plain search, point-in-time filtering that rejects pages published after the forecast date, or point-in-time filtering plus a separate LLM (DeepSeek-V4-Flash, April 2026 release) that judges whether retrieved content could have been public at that date. Responses are locked before scoring.

Scoring is task-specific: VAL measures error reduction against a cutoff-price baseline plus whether the entry-timing call was right; RANK uses normalized Spearman correlation plus a top-K hit rate with K = floor(N/3); EVT uses Brier skill against an uninformative 0.5 baseline; FUND uses a per-instance relative-error utility clipped to [-1, 1]. CF and CALC use predefined logical and numerical references. The diagnostics then compare scores across regimes and retrieval settings, and re-run the same questions with reworded prompts and slightly perturbed numbers to see whether the answers and aggregate scores move.

Why This Matters

Impact on research. The paper shows that a benchmark score is only meaningful alongside the conditions under which it was produced. It gives the evaluation community a concrete protocol — cutoff-aware instance generation plus page-level leakage auditing — for separating memorization and retrieval leakage from genuine forecasting, and it quantifies how much of a score gap can be closed by stricter controls.

Real-world applications:

• Model and agent selection for investment research teams: task-level results (VAL vs. EVT vs. FUND) show that picking a system depends on the forecasting objective rather than a single leaderboard number.
• Rapid iteration during system development: resolved T2 events give immediate feedback, and the filtered T2 setup came within 0.0043–0.0267 of live scores in the reported attribution comparisons.
• Cross-market deployment planning: the CN/US/HK breakdown shows leadership changing by market (DeepSeek V4 Flash in CN, Gemini 3.1 Flash-Lite in US and HK), which matters when teams deploy regionally.
• Red-teaming research pipelines: the leakage filter and PIT audit provide an operational check for whether an agent’s “research” is actually looking ahead.

Industry relevance. Any firm building LLM-based research or forecasting workflows faces the same validation problem: live evaluation is slow, and historical evaluation may be contaminated. The paper’s central empirical claim — that PIT plus filtering brings post-cutoff backtests close to live results — is directly useful for teams that need fast iteration without waiting months for events to resolve.

Future Directions

• Whether fresh feedback enables self-improvement. The authors state they will test whether T2 feedback helps autoresearch systems improve T3 forecasts, probing recursive self-improvement (RSI) in finance.
• Rebuilding the horizon for newer models. Because the current evaluation prioritizes temporal comparability, evaluating frontier models with later knowledge cutoffs would require moving the shared T2 window forward and waiting for outcomes to resolve; the configurable generators are meant to support this.
• Tightening point-in-time reconstruction. The authors acknowledge that web pages, metadata, snippets, and archived sources may not perfectly match what an investor or deployed agent could access at the cutoff date.
• Separating market difficulty from benchmark artifacts. CN, US, and HK differ in disclosure rules, language coverage, data vendors, and release timing, so cross-market results are framed as diagnostic stability evidence rather than clean causal estimates.

Target Audience

Researchers and engineers working on LLM agents and deep research systems, benchmark designers concerned with temporal validity and data leakage, and quantitative or AI teams in asset management evaluating whether agentic forecasting tools are worth deploying. It is also relevant to financial NLP researchers who need task-specific, outcome-based evaluation rather than aggregate benchmark scores. Readers without finance background can follow the design, but the scoring sections assume comfort with correlation-, Brier-, and error-based metrics.

Authors’ abstract

Anticipating financial outcomes requires synthesizing evolving evidence and reasoning under uncertainty, making deep research systems promising tools for forecasting. Yet evaluating forecasting ability is challenging because historical backtests may reward memorized outcomes or information retrieved after the forecast date. Live forecasting avoids direct access to realized outcomes, but feedback arrives only after events resolve. To obtain timely feedback and assess historical scores, we introduce FinForesight, a diagnostic backtesting benchmark for financial deep research systems. Across the Chinese, U.S., and Hong Kong markets, the benchmark evaluates four forecasting tasks: valuation, return ranking, event direction, and future financial fundamentals. Two capability diagnostic tasks assess counterfactual reasoning and financial calculation, scored separately from forecasting performance. Our harness-compatible framework restricts retrieved evidence to information available at the forecast date. We compare pretraining-era, resolved post-cutoff, and live evaluations to test whether controlled backtests approximate future performance. Pretraining-era scores exceed post-cutoff scores in most temporal comparisons, and stronger retrieval controls narrow the gap between post-cutoff backtests and live results. Forecasting attribution and robustness analyses reveal how scores respond to retrieval policy and how predictions respond to prompt rewrites and numerical perturbations. The code and benchmark data are publicly available.