Investor's Last Exam: Can AI Agents Do Equity Investment Without Seeing the Future?

AI summary

Overview

Research area: AI agent evaluation / financial AI — specifically, benchmarking autonomous AI agents on professional buy-side equity research.

Technical level: Advanced. The paper assumes familiarity with LLM agent tool-use, rubric-based evaluation, and equity-research concepts (cash-flow reconciliation, forecasting, valuation).

Scope in one sentence: ILE is a benchmark of 86 expert-authored equity-research assignments run against a versioned 3.43-million-document financial database that enforces a strict pre-decision information cutoff, scoring agents on research quality, forecast accuracy, and simulated investment returns.

What This Paper Is About

Existing financial benchmarks test isolated skills — document question answering, agentic search, or forecasting — but none enforce, at the level of the data itself, that an agent can only see information that was genuinely available before its decision date. The paper argues that publication-date filtering is only a retrieval-level heuristic, because pages can keep an early date while carrying later content, so post-cutoff information leaks into evidence, computation, and scoring.

The authors build Investor's Last Exam (ILE) to close that gap: a frozen, versioned financial environment plus expert-authored tasks and rubrics that trace every scored requirement back to a source document, a calculation, or an explicit financial assumption. The goal is to test the full evidence–assumption–judgment chain, not just whether a model can answer a finance question.

Key Contributions

1. A time-frozen research environment. A continuously updated corpus of 3,430,336 document versions (1.52 TB, as of 15 October 2026) across nine source categories, where every immutable version carries an availability timestamp. The cutoff is enforced across the whole toolchain — search, read, download, table operations, and cached results — rather than at the search entry alone, suppressing look-ahead leakage at the mechanism level.

2. An expert-constructed task suite and evaluation framework. Buy-side analysts and fund managers author the questions, and each scored requirement is anchored to evidence via a financial dependency graph that links source-grounded facts, assumptions, intermediate calculations, and investment judgments. Experts provide questions and judgments; agents assist with evidence search; multiple models assist in review; experts finalize rubrics.

3. A multidimensional evaluation spanning research quality, forecasts, and realized outcomes. The taxonomy covers five categories with twenty subtypes plus Integrated Research and Market Trading (86 assignments total), scored on seven rubric dimensions: Result, Evidence, Financial logic, Calculation, Reconciliation, Decision consistency, and Uncertainty. Designated forecasts are additionally settled against subsequent realized disclosures and against a simulated trading sample.

4. A diagnostic, dependency-level error analysis. Rather than reporting only aggregate scores, the paper traces score differences to specific financial dependencies (cash-flow identity, operating-cash quality, accounting scope, forecast assumption validity), and shows that expert rubric refinement can distinguish genuine reasoning failures from ambiguous scoring requirements.

Main Findings

• Top overall scores are moderate, and forecasting lags badly. GPT-6 Astra leads at 68.1 out of 100 and Claude Opus 5.5 follows at 66.9 across eleven evaluated systems. Astra scores 78.9 on Research & Evidence and 77.6 on Evaluate & Decide, but only 52.5 on Forecast & Model, with KPI and cross-company forecasting at 41.1 and 49.6.

• The two leaders have different strength profiles. Opus leads Result and Calculation, while Astra leads Financial logic, Reconciliation, Decision consistency, and Uncertainty; Qwen 3.8 Max leads Evidence. Opus leads both Forecast & Model (52.6) and Monitor & Update (64.2), narrowly ahead of Astra's 52.5 on forecasting.

• Predictive ranking differs from research ranking. On realized forecast-outcome criteria, DeepSeek V4.1 Flash leads at 40.3, ahead of Astra at 37.7 and Opus at 37.6. DeepSeek V4 Pro leads the Market Trading category at 64.6 despite a 59.3 overall score. Aggregate research quality does not order models the same way predictive performance does.

• Simulated returns reorder models again. On a retrospective sample of twenty assignments covering eleven companies and thirteen disclosures (March–August 2026, 10 basis points per leg plus 3% annualized short borrowing), Hunyuan 4 Preview leads at 8.50%, then DeepSeek V4.1 Flash at 7.98% and Astra at 7.18%, versus 1.18% for always long. Kimi K3's 7.15% beats Opus's 6.39% despite a lower research score (58.2 vs 66.9).

• Cost and tool use vary enormously. DeepSeek V4.1 Flash scores 63.0 at USD 0.36 per task, while Opus gains 3.9 points at USD 7.96 — roughly 22 times the cost. Flash averages 73.8 interaction rounds and 113.8 tool calls per task; Kimi K3 averages 26.7 rounds and DeepSeek V4 Pro 34.1, both above the 10.4 and 21.3 reported by FrontierFinance and the 13.5 rounds reported for o3 in Finance Agent Benchmark.

• Comparable scores do not imply comparable information compliance. Astra scores 68.1 on the versioned database and 68.3 on public-web Codex, but 57% of its public-web trajectories contain post-cutoff sources — including the target quarter's realized Microsoft revenue and Tesla deliveries — despite a dated query. In Tesla task O74, a query containing before:2026-03-24 still retrieved a source dated July 2 reporting realized Q2 deliveries.

• Failures concentrate in assumptions and interpretation, not retrieval or arithmetic. All eleven systems recover the eight reported inputs and reconcile both cash-flow identities in AEP N27, yet only five compute both funding shortfalls, three distinguish net fuel adjustments from gross collections, and five reach the supported funding conclusion. In Tesla O80, three systems earn every supporting-analysis point but no forecast-outcome credit.

• Submission reliability is a separate bottleneck. Under strict submission requirements, Step 5 Preview drops from 55.1 to 34.5 and Hunyuan 4 Preview from 60.4 to 47.7. Hunyuan stops without submitting on 17 trajectories, and Opus has three rejected submissions on each of five.

• Rubrics themselves need calibration. Qwen 3.8 27B's initial AWS O15 revenue interval missed the realized outcome yet earned 15.8 points for being narrow; after review replaced separate width credit with a joint accuracy–uncertainty score, its revised interval also missed and earned zero interval credit.

Methodology in Plain English

The team first built a data foundation that cannot "forget." Instead of overwriting documents when companies update them, they keep every historical version and stamp each one with the time it actually became available — not the time it claims to have been published. When an agent runs a task, the tools themselves check that timestamp before returning anything, so the cutoff applies to the files and tables the agent computes with, not just to its search queries.

They then had experienced buy-side investors write realistic research assignments, organized by the deliverable an analyst would actually produce: screening ideas, building evidence, forecasting, making a recommendation, monitoring a thesis, writing an integrated report, and predicting prices or returns. For each task, the expert also builds a dependency map showing which facts come from which document version, which assumptions bridge them, and which calculations produce the answer.

Answers are scored on explicit criteria on a 0–100 scale, with numeric checks handled deterministically and semantic criteria judged by GPT-6 Astra (with a sensitivity check on the judge choice). Each task allows 7,200 seconds and up to three submissions. Two modes are reported: a default answer-based mode that grades the final available answer, and a strict mode that requires successful submission, which isolates formatting and completion failures from reasoning quality.

Finally, they ran eleven systems through the benchmark, and separately re-ran a subset with public-web search instead of the versioned database to see whether a similar score meant a similar information boundary.

Why This Matters

Impact on research. The paper shifts the unit of evaluation from "can a model answer a finance question" to "can a model build a defensible chain from evidence to assumption to judgment under a strict information boundary." That makes it possible to locate where analytical strength fails to become a forecast, rather than only reporting an aggregate number. It also gives a concrete alternative to date-filtered retrieval as a leakage control.

Real-world applications:

• Model procurement by asset managers: choosing between agents on forecast quality and cost-per-task (USD 0.04 to USD 9.11 across the eleven systems) rather than on generic benchmarks.
• Agent development and post-training: using dependency-level error traces in forecast assumptions, accounting scope, and decision consistency as targeted training signal.
• Backtesting discipline for AI-driven strategies: the versioned corpus and simulated-return protocol offer a template for testing agents without look-ahead contamination.
• Audit and compliance review: source-version-linked scoring criteria create an inspectable record of which evidence supported which claim.

Industry relevance. The finding that models reconcile identities correctly but misjudge whether a cash improvement is sustainable, or whether an expense forecast is consistent with a payment forecast, describes exactly the errors that matter in professional equity research. The cost spread and the submission-failure rates are also operational facts that deployment decisions depend on.

Future Directions

• Separating memorization from retrieval. Frontier models may have implicitly learned historical outcomes during pre-training, so the authors propose counterfactual or "cold-start" experiments — a synthetic financial-scenario engine that perturbs real statements and constructs virtual peer companies.
• Decomposing long assignments. Breaking expert-authored tasks into composable micro-task graphs with difficulty-weighted scoring and a multi-model consensus judging protocol.
• Distinguishing reasoning failure from stochastic noise. Forecast-type tasks settled against realized outcomes can penalize sound analysis for exogenous market moves; better evaluation would separate the two.
• Growing into a living benchmark. Continuously expanding task coverage as new market disclosures arrive, while preserving the information boundary.

Target Audience

AI evaluation researchers working on agents and long-horizon tool use; quantitative and fundamental investors evaluating AI for research workflows; financial-data engineers interested in point-in-time and versioned data infrastructure; and benchmark designers outside finance who face the same look-ahead leakage problem in time-sensitive evaluation. Readers need some grounding in equity research concepts (cash-flow reconciliation, forecast bridges, valuation) to get full value from the case analyses.

Authors’ abstract

Financial agents must form forecasts and investment judgments from decision-time evidence, with predictions evaluated against subsequent outcomes. Existing benchmarks assess financial knowledge, research workflows, or forecasting, but evaluating the full research process requires a strictly frozen information set and professional evaluation criteria. We introduce Investor's Last Exam (ILE), an equity-research benchmark led by experienced buy-side investors and fund managers. Its three components are a versioned financial database spanning 3.43 million document versions and nine source categories, supplemented by structured market data, with validated timestamps governing retrieval and computation; a task taxonomy organized around professional analyst deliverables; and an expert-led framework linking scoring criteria to sources, calculations, and assumptions through financial dependency graphs. Evaluation spans 86 assignments and jointly assesses research quality, forecast accuracy, and simulated investment performance. GPT-6 Astra and Claude Opus 5.5 lead overall at 68.1 and 66.9 out of 100, yet even Astra scores only 52.5 on Forecast & Model. Our results show that even state-of-the-art models fail to translate overall research capability into predictive performance. ILE shifts the evaluation unit from isolated capabilities to the evidence--assumption--judgment chain, allowing researchers to locate where analytical advantages fail to translate into forecasts and investment judgments, and providing a traceable, diagnostic foundation for training and evaluating financial agents.