EPSBench: Benchmarking LLM Agents for Earnings Forecasting

AI summary

Overview

Research area: Natural Language Processing / LLM agent evaluation, applied to financial forecasting (quarterly earnings).

Technical level: Intermediate. The paper is readable without deep accounting knowledge, but it assumes familiarity with LLM agents, tool use, prediction intervals, and standard forecast-error metrics.

Scope (one sentence): EPSBench is a benchmark of 149 firm-quarters from 86 US companies that asks LLM agents to estimate company-specific income-statement drivers with cited evidence, converts those estimates into GAAP diluted EPS through a fixed accounting calculator, and scores both the point forecast (against date-matched analyst consensus) and an 80% prediction interval.

What This Paper Is About

Quarterly earnings forecasting requires more than naming a number: an analyst builds up to EPS from below, estimating revenue, then the costs charged against it, then the items beneath them, with each estimate resting on particular evidence. Existing benchmarks such as ForecastBench, FutureX, and FinDeepForecast typically just ask a model for the predicted figure, leaving the route to it buried in whatever the model writes in its trajectory. EPSBench's goal is to force that route into the open — agents must estimate the same company-specific line-item drivers, attach cited evidence to each, and let a shared accounting calculator do the arithmetic — so that a forecast error can be traced to the stage of the income statement where it enters.

Key Contributions

1. Company-specific accounting schemas. For each firm-quarter, the authors build a schema from the company's own historical filings containing a year-ago baseline, forecast drivers, and a directed acyclic graph (DAG) of accounting identities that maps driver values to GAAP diluted EPS. The DAG reproduces reported EPS within one cent on 146 of 149 instances and within four cents on the remaining three.

2. A common evaluation protocol. A fixed information environment (filings, earnings-call transcripts, news, all cut off at the day before release), six tools on a Model Context Protocol server, a shared accounting calculator, and an evaluation that links dated evidence and driver estimates to point-forecast scores, interval calibration, resource use, and format compliance.

3. An empirical analysis locating model differences. The paper shows that the gap between stronger and weaker models concentrates in operating-income estimates on quarters where earnings move far from the year-ago level, and that model-elicited uncertainty is systematically miscalibrated and skewed low.

4. A public release. Schemas, tools, prompts, forecast records, scores, and trajectories are released (transcript text replaced by document identifiers for licensing reasons), along with fetch scripts for public filings and the filtered CC-News index.

Main Findings

• Two tiers of models. Under OpenCode SDK 1.17.9 with the full information environment, Opus 4.6, GPT-5.5, and GPT-5.4 (the "leading models") reach median price-deflated errors (PDE) of 0.113%, 0.131%, and 0.126%, close to analyst consensus at 0.111%, and each beats consensus on 50% of instances. GPT-5-mini, Qwen 3.5, and GLM 4.5 Air (the "trailing models") reach 0.191%, 0.190%, and 0.215% and win on 38%, 32%, and 28%.

• No detectable paired difference for the leading three. Paired bootstrap comparisons show the leading models' intervals all include zero; Opus 4.6 has the lowest median paired difference at −0.002 percentage points of PDE with a 90% interval of [−0.011, +0.006]. The paper explicitly notes that an interval containing zero does not establish equivalence. The other three models have higher paired error than consensus.

• All six beat naive extrapolation. Copying year-ago EPS gives PDE 0.238% and copying last-quarter EPS gives 0.226%, worse than every model.

• Interval coverage falls well short of nominal. Nominal 80% intervals contain reported EPS in only 37% (Qwen 3.5) to 70% (Opus 4.6) of runs that return an interval. Misses are lopsided: reported EPS lands above the upper bound in 20–38% of outcomes for every model, while below the lower bound Opus 4.6 and GPT-5.5 stay at 10% and the other four reach 16–25%.

• Forecasts run low, and the interval follows. Median signed error (y − ŷ)/P ranges from +0.021 to +0.048 pp across models with confidence intervals excluding zero for every model. Interval midpoints sit within 0.004 pp of the point forecast, so a low point pulls the whole interval down. Consensus is also low at +0.028 pp [+0.008, +0.051]; the authors connect this to documented walk-down of analyst forecasts and rewards for meeting or beating expectations.

• Coverage alone does not rank intervals. GPT-5.4 and GPT-5.5 have similar point error, but GPT-5.4's intervals are narrower (0.25 vs 0.33 pp median width), cover fewer outcomes (52% vs 61%), and score worse (4.29 vs 3.08). GPT-5-mini's mean interval score of 19.61 is three to seven times that of every other model (2.87–6.48) because a few narrow, far-off intervals dominate the mean.

• The gap opens on "far" quarters. Splitting into thirds by price-scaled change from year-ago EPS, on the 49 near instances model PDEs range 0.048%–0.082% versus 0.036% for consensus; on the 50 far instances the leading models reach 0.392–0.398% versus 0.599–0.716% for trailing models and 0.409% for consensus.

• Evidence quality tracks performance. On far instances, the leading models cite a number for their most-adjusted driver in 95–96% of runs versus 35–72% for trailing models; they cut year-ago error at least in half on 29–30 of 50 instances versus 12–19; and they move in the wrong direction on only 5–6 instances versus 12.

• The error gap is an operating-income gap. On the 125 instances with an operating-income subtotal, median operating error is 0.089–0.098 pp for leading models versus 0.149–0.166 pp for trailing models, while non-operating increments are similar at 0.007–0.020 pp. A worked example is Boeing's fourth quarter of 2025, where much of the gap comes from a divestiture gain reported within operating income.

• Sources overlap but are not redundant. For GPT-5.5, keeping only news produces no detectable paired change (−0.000 pp [−0.010, +0.009]) while cutting input use to about a quarter; keeping only filings and transcripts raises error by +0.009 pp [+0.002, +0.016]; removing both raises it by +0.033 pp [+0.019, +0.047] and drops the win rate from 50% to 31%. Of the 149 instances, 67 show no harmful removal, 32 are hurt only when both are removed, and 50 depend on a particular source (17 only filings/transcripts, 24 only news, 9 either), with a rank correlation of −0.01 between the two sources' error increases.

• Harness and effort move cost, not accuracy. Across harness and reasoning-effort changes, median paired PDE shifts by only −0.004 to +0.001 pp and all five intervals include zero. Output tokens per run, including reasoning, rise from 9.7K to 40.5K for Opus 4.6 and from 6.2K to 12.6K for GPT-5.5 under OpenCode at maximum effort; the vanilla harness uses 33% fewer input tokens for GPT-5.5. Interval coverage does move: 70% versus 65% across Opus 4.6 configurations, and 54%–61% for GPT-5.5.

• Temporal separation was checked. All benchmark announcements follow the published knowledge cutoffs of four models. GLM 4.5 Air was released before the evaluation window; Qwen 3.5 was released within it and a memorization test found no detectable recall of benchmark outcomes. Context compaction in the harness triggered on 49 of GLM 4.5 Air's 447 runs (its 128K window), and those runs are scored as delivered.

Methodology in Plain English

Building the task. The authors picked large, widely followed US companies across eight sectors for which historical filings, a prior-quarter transcript, and an IBES GAAP consensus were available, yielding 149 firm-quarters from 86 companies with announcements between 2 December 2025 and 24 June 2026. The information cutoff is the earlier of the earnings-call and official-announcement dates minus one day, so the target quarter has always ended but results are not yet public — the same position an analyst is in just before an announcement.

Building the schemas. Because income statements differ across companies (a bank reports net interest income; a technology company may split product and service revenue), each firm-quarter gets its own schema. An LLM extractor (o4-mini) pulls income-statement tables from 10-Ks and 10-Qs; a reasoning model (deepseek-reasoner) reads the four preceding quarterly statements and generates the baseline, driver set, and DAG; GPT-5.4 extracts historical tax and share-count references. The schema is then validated by executing the DAG on the extracted year-ago values.

Running the agents. Agents get six tools on an MCP server covering SEC EDGAR filings, Seeking Alpha transcripts, and CC-News. The server blocks the target-quarter filing and call and rejects any news article dated after the cutoff; shell, file, and web access are disabled in the primary harness. For each driver, the agent writes an evidence line naming its source and submits the value to the shared calculator, which returns the income statement and EPS. The agent keeps that EPS as its point forecast and separately reports p10 and p90 bounds. Each run is a single attempt with no restarts; an unparseable final message counts as a format failure.

Scoring. The primary point metric is price-deflated error, |ŷ − y| / P, where P is the closing share price on the last trading day before release — scaling by price avoids unstable percentage errors when EPS is near zero. Secondary metrics are win rate against consensus, surprise-direction accuracy, interval coverage, the two tail rates, interval width, and a proper interval score scaled by price. Each configuration is run three times per firm-quarter (447 runs), taking the median PDE per instance and then the median across the 149 instances; forecaster comparisons are paired on instances with a bootstrap over 10,000 resamples reporting 5th and 95th percentiles.

Why This Matters

Impact on research. Most agent forecasting benchmarks score only the final number, which makes it impossible to say why one model beat another. EPSBench's driver-and-calculator design turns a scalar score into an auditable record: because the arithmetic is fixed, every EPS score stays attached to the line-item estimates and cited evidence that produced it. That enables the kind of decomposition the paper performs (operating income versus non-operating items) and gives future work a common representation for comparing quantitative assumptions rather than prose.

Real-world applications:

• Equity research and analyst tooling — a structured, evidence-cited earnings estimate that a human analyst can inspect and override line by line.
• Model selection and procurement — the finding that harness and reasoning effort change cost far more than accuracy is directly useful when choosing a configuration under a budget.
• Uncertainty communication — the systematic low bias and 37–70% coverage of nominal 80% intervals is a concrete warning for anyone shipping model-produced confidence bounds.
• Financial NLP evaluation design — the point-in-time tool enforcement and memorization test offer a template for retrospective evaluations that risk training-data contamination.

Industry relevance. The two-tier result — three models statistically indistinguishable from date-matched analyst consensus, three clearly worse — is a practical signal about where current agents stand on a recurring, numerically precise task. The concentration of error in operating income, and the finding that a quarter of instances hinge on one specific source (news or filings, not both), tells practitioners where to invest in retrieval and where model choice actually matters.

Future Directions

• Extending beyond the fixed window. The authors note that the schema-generation model, deepseek-reasoner, is no longer served, so extending EPSBench to new quarters will require another extractor — an open question about how to keep benchmark construction reproducible as model APIs retire.
• Diagnosing the low bias. Median signed error is positive for every model and consensus is also low. Whether this comes from expectation management in the retrieved evidence, from exposure to analyst expectations, or from something in the task framing is unresolved.
• Improving interval calibration. No model approaches 80% coverage, and intervals are nearly symmetric around an already-low point. Whether better intervals require better point forecasts or a separate uncertainty mechanism is not settled by this data.
• Scaling the sample. With 149 firm-quarters, the study can only report that leading models show no detectable difference from consensus — an interval containing zero is not equivalence. A larger sample, or more quarters per company, would be needed to resolve differences of the size observed.

Target Audience

Researchers and engineers working on LLM agents, tool-use evaluation, and financial NLP will get the most from this paper, particularly those designing benchmarks where intermediate reasoning must be auditable rather than inferred from a trajectory. It is also relevant to quantitative finance and equity-research practitioners assessing whether current agents are usable for earnings estimation, and to anyone studying forecast calibration and the gap between nominal and realized prediction-interval coverage. Readers need only modest background in accounting — the schema and calculator abstraction carries most of the domain complexity — but should be comfortable with paired bootstrap comparisons and proper scoring rules.

Authors’ abstract

Quarterly earnings forecasting requires an agent to connect financial evidence to the income-statement line items that produce a company's reported earnings. We introduce EPSBench, a benchmark for studying this ability across 149 firm-quarters from 86 US companies. Agents read a fixed collection of filings, earnings-call transcripts, and news, then estimate company-specific income-statement drivers with cited evidence. A shared accounting calculator converts these estimates into GAAP diluted earnings per share (EPS), providing a common basis for comparing forecast assumptions. We evaluate six models under a common harness, assessing point forecast error relative to date-matched analyst consensus and checking whether their prediction intervals achieve the stated coverage. Three models show no detectable paired difference in point forecast error from consensus, and three have higher error. Across all six models, nominal 80% intervals contain reported EPS in only 37--70% of runs that return an interval, and most misses fall above the upper bound because the forecasts run low. The driver records trace most of the error gap between models to operating-income estimates.