Forecasting Evaluation Across Time, Horizons, and Hierarchies

A demand model evaluated only during a calm month may look excellent. Then come the holidays, an outage, or a policy change, and it fails. That is why time-series evaluation has to sample several forecast origins, and preserve what information existed at each one.

Shuffling time destroys the deployment problem outright: it lets future conditions influence the past. The narrower failure is quieter and much more common. A single origin, honestly chosen, can still be decided by one outlier or one level shift that happened to sit just after the cut. The rolling-origin vignette for the R package greybox gives exactly that reason for using several origins rather than one. Multiple origins are what robustify the evaluation.

Each origin recreates a decision at a historical point. The technique has a name, a definition and a date. The greybox vignette states it plainly: “Rolling origin is an evaluation technique according to which the forecasting origin is updated successively and the forecasts are produced from each origin (Tashman 2000).” The reference is Tashman's 2000 review of out-of-sample tests of forecasting accuracy.

It is defined against something. The two main out-of-sample setups are fixed origin and rolling origin evaluation. Hewamalage and colleagues cite Tashman for both, in a 2022 survey of forecast evaluation. Rolling origin travels under two other names as well: time series cross-validation, and prequential evaluation.

The same survey adds the failure mode the five steps below do not show. Each origin's test period becomes training data for the next origin, which makes the procedure “naturally susceptible to data leakage dangers”. Advancing the origin is not by itself a guarantee of honesty. It is a guarantee only if the refitting at each origin is done with the same discipline as the first.

Aggregating all lead times can hide where a model loses value. The National Hurricane Center does not aggregate them. Its published scorecard is what the argument looks like with numbers in it.

For the Atlantic basin, the center's 2024 verification report, by Cangialosi and Martinez, puts it in one line: “Mean track errors ranged from 19 n mi at 12 h to 115 n mi at 120 h.” The report's table carries the decimals — 19.2 n mi at 12 h against 115.3 n mi at 120 h. Roughly a sixfold growth in error, across one season, from one system.

The evidence thins as the error grows. The number of verifiable cases falls from 311 at 12 h to 79 at 120 h. So the long-horizon figures are worse and less firmly established at the same time. That is the trade the comparison below records as “many origins available” at short range and “fewer independent outcomes” at long range.

Skill does not decay in a straight line either. Measured against CLIPER5, the climatology-and-persistence baseline, official track skill peaks at 77.2% at 48 and 60 h. It falls to 70.3% at 120 h. The forecast beats its baseline by the widest margin in the middle of its range, not at the start. One pooled error number would have shown none of this.

Useful baselines depend on the cadence of the series. They are also not a formality that a serious model clears on the way past. The US COVID-19 Forecast Hub is the case where a persistence baseline was operated in public, week after week, rather than assumed.

The baseline had a name and a rule. The COVIDhub-baseline's median prediction is simply the most recent week's reported deaths. That is the first row below, run as a live submission. Cramer and colleagues, reporting the evaluation in PNAS in 2022, scored 27–28 models against it, forecasting weekly incident deaths from April 2020 to October 2021. Two-thirds of the models beat it. The ensemble reached a relative weighted interval score of 0.61 — 39% less probabilistic error than the baseline. Observed prediction-interval coverage was typically below nominal. The intervals were narrower than they claimed to be.

The margin depended on the horizon. The same paper says so directly: “Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon.”

A separate ECDC-supported consortium found the same shape on other data. Its eLife study covered 32 countries and 48 models over 52 weeks. The ensemble's median scaled relative WIS for cases worsened from 0.62 at 1 week ahead to 0.9 at 4 weeks ahead. For deaths it moved only from 0.69 to 0.76.

Forecasts may exist at several aggregation levels. The M5 competition made that structure the object of the contest rather than a caveat about it.

Entrants were asked for 28-day-ahead forecasts of 42,840 Walmart series. Those series sat in 12 cross-sectional aggregation levels, built from 3,049 products sold in 10 stores across California, Texas and Wisconsin. The competition report defines the requirement: “forecasts at the lower levels have to sum up to the ones at the higher levels so that decisions made using the forecasts across different levels are aligned”.

The interesting part is how the competition obtained that property. It did not ask for coherence and check it afterwards. Prize eligibility required submitting only the 30,490 product-store series at level 12, and every higher level was obtained by summation. Coherence was therefore true of every eligible submission by construction. The modelling question became which level to forecast, rather than how to reconcile levels that disagree.

The figures survive independent handling. An unaffiliated 2024 paper on local versus global models for hierarchical forecasting reproduces the same dataset description: 3,049 products over five years, 42,840 hierarchically organised series over 12 levels.

Report point and probabilistic metrics by horizon, origin, series, season, event regime, and volume tier. Include calibration and width for intervals, plus the business cost of shortages, overstaffing, or missed capacity.

The M4 Competition ran this at scale in 2020, across 100,000 time series and 61 forecasting methods. One of its three stated aims was “including prediction intervals in the evaluation process as well as point forecasts”. Point accuracy had stopped counting as the whole result. A scorecard reporting one mean error reports less than the competition it inherits from.

If forecasts influence prices, inventory, or user behavior, later outcomes are partly policy-generated. The cost of being wrong may never appear as a metric at all. Zillow Group is the documented case. On 2 November 2021 the company announced it would wind down Zillow Offers. Its third-quarter results carried an inventory write-down of approximately $304 million on homes bought above its own estimates of their future selling prices. The wind-down cut roughly 25% of a 6,400-person workforce. Co-founder and CEO Rich Barton named the forecast itself as the reason, in the results press release filed with the SEC that day: “the unpredictability in forecasting home prices far exceeds what we anticipated and continuing to scale Zillow Offers would result in too much earnings and balance-sheet volatility”.

Read that sentence for what is missing from it. There is no error metric in it. The forecast set the offer price. The offer price bought the house. The purchases moved the company's own inventory position, and the bill arrived as a write-down and a headcount reduction. Monitoring has to distinguish model error from intervention effects, because the second is where the money is.