WhatIf-Bench: Benchmarking Intervention-Aware Time Series Forecasting

AI summary

Overview

Research area: Time series forecasting benchmarks; covariate-aware (intervention-aware) forecasting with time series foundation models and large language models.

Technical level: Intermediate — readable for anyone familiar with basic forecasting metrics (error scores, zero-shot evaluation), with the heavy formal notation confined to the appendix.

Scope (one sentence): The paper introduces WhatIf-Bench, a semi-synthetic benchmark of 37,665 tasks built on four mechanistic simulators, to measure whether forecasters can use in-context covariates to predict a target's response to future events — including events that set covariates to values never seen in the history.

What This Paper Is About

Recent time series foundation models can take covariates as part of their input and factor them into a forecast without task-specific training. That capability is what makes "what-if" scenario analysis possible: feed in a hypothetical future for the drivers, and see what the target would do. The paper asks whether models actually do this reliably — specifically, whether they can generalize when a future event sets covariates to values outside anything in the observed history, rather than merely replaying a pattern they have already seen. Existing benchmarks can only test models on covariate values their datasets happen to contain, so they cannot answer this question or pinpoint where a model fails.

Key Contributions

1. A benchmark for intervention-aware forecasting. WhatIf-Bench is semi-synthetic: target series come from mechanistic simulators of four domains — OpenDC (data centers, 22 targets), GridLAB-D (power grids, 11), SUMO (road traffic, 30), and EPANET (water networks, 37) — for 100 target series in total, with controlled covariate-target relationships imposed on top so the effect of any covariate is known exactly.

2. Replay vs. Novel future events. Each task presents a history of "events" (bounded windows in which covariates are moved away from their base setting and the target responds) and asks for a forecast under one further event. A Replay future repeats a past event's covariate values; a Novel future sets covariates beyond the range seen in the history. Both futures are generated over the same history, giving a paired contrast.

3. A systematically controllable difficulty design. Five axes are varied: number of covariates K ∈ {1,…,5}, context/prediction length (4 pairs, from context 1024 / prediction 128 up to roughly context 8,000 / prediction 512–864), event length ∈ {32, 64, 128, 256, 512}, number of historical events (fewest, one or two, up to 64), and future type. This produces 40,000 tasks, of which 37,665 are kept after filtering for tasks whose covariates have a non-negligible effect in every event.

4. A broad zero-shot evaluation. Six covariate-aware foundation models (Chronos-2, t0-gamma, Timestral, TiRex-2, TS-ICL, TimesFM-3.0), one tabular foundation model (TabPFN-3.5, run through the TabPFN-TS harness), two frontier LLMs (Gemini-3.1-Pro, Claude-Opus-4.8) at two reasoning efforts, and three simple regression baselines (XGBoost, Ridge Regression, Linear Regression), scored with skill score against a naive last-value baseline and with win rate.

Main Findings

• Frontier LLMs score highest overall. On the full benchmark, Claude-Opus-4.8 (High) reaches 72.5% wQL skill and Gemini-3.1-Pro (High) 70.0%. The best foundation models follow: t0-gamma 64.5%, TimesFM-3 61.6%, TS-ICL 61.3%, Timestral 59.2%, then TabPFN-3.5 and Chronos-2 at 50.8% each, and TiRex-2 at 46.7%. The untuned regression baselines are below every foundation model: XGBoost 42.9%, Linear Regression 26.2%, Ridge Regression 25.4%.

• Reasoning effort helps the LLMs. Moving from Medium to High raises Claude's skill from 70.3% to 72.5% and Gemini's from 58.2% to 70.0%.

• Every model degrades sharply from Replay to Novel. All models forecast well on Replay futures (48% to 85% skill) but drop to between −7% and 50% on Novel futures. Claude-Opus-4.8 goes from 84.8% to 50.3%, Gemini-3.1-Pro from 84.3% to 42.6%, and TS-ICL falls furthest among foundation models, from 80.4% to 23.6%.

• Regression baselines collapse the most. XGBoost falls from 65.0% on Replay to 6.7% on Novel, and Linear Regression from 49.0% to −6.7%, worse than the naive baseline. Novel futures are therefore where foundation models and LLMs have their largest advantage: TimesFM-3 is only about 10 points above XGBoost on Replay but about 35 points above it on Novel.

• More covariates hurt only on Novel futures. On Replay, skill varies only marginally as K grows. On Novel, skill falls consistently: Claude-Opus-4.8 (High) and Gemini-3.1-Pro (High) both reach 75% at one covariate but fall to 19% and 15% at five. By five covariates the best foundation models catch up — t0-gamma is strongest at 27% and TimesFM-3 (19%) is level with Claude-Opus-4.8 (High), while TS-ICL drops to −12%. Linear Regression goes from 12.1% to −49.8% and Ridge from 12.2% to −49.9%.

• More historical events help, mainly on Novel futures. On Replay, skill is roughly flat for most models, though XGBoost gains about 25 points. On Novel, skill rises sharply: Claude-Opus-4.8 (High) more than doubles from 24% to 58%, Gemini-3.1-Pro (High) from 19% to 42%, and Chronos-2 — the largest gain among foundation models — from 6% to 35%. Linear Regression improves from −27% at one event to 9% at 64 but stays far below the best models.

• Event length and context length follow the same pattern. Skill on Novel futures improves both when events are shorter and when the context is longer, i.e. whenever the context can hold more past events. Large differences between models persist even with the most evidence.

• LLMs are far slower. Most foundation models run in under half a second per series; TabPFN-3.5 is the slowest of them at 21.26 s. The LLMs take roughly 300 seconds per series (Claude-Opus-4.8 High 270.13 s, Gemini-3.1-Pro High 355.28 s) — about four orders of magnitude slower than the fastest foundation models. The LLMs were also only prompted for the median (quantile 0.5) because requesting many quantiles exceeded output length and caused formatting failures, which the authors note disadvantages them under the probabilistic metric.

• The authors' reading of the results. Strong Replay performance does not carry over to Novel futures, and since the same models handle five covariates as well as one on Replay, the difficulty lies not in the number of covariates but in extrapolating their joint effect to unseen values.

Methodology in Plain English

Real-world data only ever shows one version of events, so you cannot observe what a target would have done if its drivers had been set differently. The authors borrow a trick from causal inference: build the data in a simulator where you control everything and therefore know the right answer exactly.

Concretely, they take four established open-source simulators, each modeling a real system (a data center, a power-distribution grid, a road network, a water-distribution network), and run each on a fixed base configuration. From each run they pick one system variable as the target and a set of adjustable properties as covariates, then impose a controlled relationship between them — a relationship grounded in how that system actually works, but defined at every covariate value, so it can be inverse, sublinear, or piecewise rather than simply proportional. Because the relationship is known, a model's error reflects how well it handled the covariates rather than noise in the data.

Then they script "events." An event is a bounded window in which one or more covariates are moved away from their base value by a multiplicative factor (the magnitude) and held there, following either a step shape or a Gaussian-bell shape, while the target responds. A task stacks a sequence of such events into a history, then asks the model to forecast the target under one more event in the future. Two versions of the same task are built: one where that future event repeats a past event's covariate values (Replay), and one where it pushes covariates beyond the historical range (Novel).

By rebuilding tasks while changing only one property at a time — how many covariates an event moves, how long each event lasts, how many events fit in the context, how long the context and horizon are — the authors can attribute performance differences to a specific cause. Performance is scored with weighted quantile loss and weighted absolute percentage error, summarized as a skill score: the percentage reduction in error relative to a naive baseline that just repeats the last observed value. Because that naive forecast is identical on paired Replay and Novel tasks, it is a consistent reference for both.

Why This Matters

Impact on research. The paper separates two things prior benchmarks conflated: whether a model can condition on covariates at all, and whether it can extrapolate a covariate-target relationship to new values. Because the ground-truth relationships are known and the difficulty axes are controlled, failures become diagnosable rather than just aggregate numbers — for example, the finding that covariate count is harmless on Replay but damaging on Novel points to a specific weakness in joint extrapolation. It also supplies an evaluation setting that is deliberately hard to game by fitting the test conditions.

Real-world applications:

• Capacity planning in data centers: asking how CPU utilization would respond if demand were pushed above anything historically observed.
• Power grid operations: testing how a distribution network reacts to load patterns outside the recorded range.
• Traffic management: forecasting road-network behavior under demand levels not present in the historical record.
• Water network planning: predicting pressure or flow responses to settings never previously used.

More broadly, any planning workflow that wants to evaluate a policy before running it depends on exactly the extrapolation ability this benchmark shows models lack.

Industry relevance. The comparison is practically informative, not just academic: a one-line, untuned XGBoost reaches 42.9% skill, and the best foundation model adds roughly 22 points on top, while frontier LLMs add roughly 30 but at about 300 seconds per series versus under half a second. That trade-off between accuracy under novel scenarios and inference cost is the kind of calculation a deployment decision actually turns on.

Future Directions

• Close the Replay–Novel gap. The authors frame generalization to Novel futures as the central open problem; a model that has genuinely captured the covariate-target mechanism should transfer to new covariate values, and none of the evaluated models does so without large loss.
• Make history use more efficient. More past events help nearly every model on Novel futures, but models differ widely in how effectively they exploit that evidence, so better in-context learning from limited event data is an explicit target.
• Extend the environment. The authors list richer system dynamics and larger covariate spaces — where many drivers are irrelevant, confounded, or misleading — as natural extensions beyond the four simulators used here.
• Train in simulation, and accept natural-language scenarios. They suggest that training forecasters inside such simulated environments could yield models that learn a system in-context and reason over its possible futures, toward world models for planning; and that LLMs' covariate-aware ability, despite their latency, could let users pose what-if queries in natural language rather than as numeric covariates.

Target Audience

Researchers and practitioners working on time series foundation models and covariate-aware forecasting will get the most from this paper, particularly those who need to evaluate models for planning and scenario analysis rather than pure accuracy on held-out data. It is also useful to applied scientists in operations research, energy, transportation, and infrastructure planning who want to know whether current zero-shot models can be trusted for counterfactual queries, and to LLM researchers interested in zero-shot numerical reasoning over structured inputs. Readers need only a working familiarity with forecasting error metrics; the causal-inference framing and formal problem statement are motivated in plain terms before being formalized.

Authors’ abstract

Recent time series foundation models (TSFMs) can consume covariates in-context, extending zero-shot forecasting to problems driven by external factors. This enables scenario analysis: forecasting how a target responds to events that set covariates to chosen future values. Whether they can do this reliably is an open question. We introduce WhatIf-Bench, a benchmark measuring how well forecasters use covariates to forecast a target's response to future events. WhatIf-Bench is semi-synthetic: time series come from mechanistic simulators of data centers, power grids, road traffic, and water networks, with controlled covariate-target relationships grounded in each system. Each task presents a history of events intervening on covariates, then asks to forecast a future event that either replays seen values (Replay) or sets them to unseen ones (Novel). Varying the number of covariates, historical events, and event length yields 37,665 tasks. Evaluating TSFMs, frontier LLMs, and simple regression baselines, we find the LLMs reach the highest skill (improvement over a naive baseline), ahead of the best TSFMs. Every model forecasts well on Replay futures (48 to 85% skill) but drops to -7 to 50% on Novel futures, with the regression baselines falling the most. Skill on Novel futures also falls as events set more covariates jointly. More historical events raise skill on Novel futures across the board, but large differences remain. WhatIf-Bench reveals generalization to Novel futures as the core challenge, and offers a testbed toward arbitrary scenario analysis.