TS-Arena -- A Live Forecast Pre-Registration Platform

AI summary

Overview

Research area: Machine learning evaluation methodology, specifically benchmarking for Time Series Foundation Models (TSFMs), with empirical grounding in energy-domain forecasting.

Technical level: Intermediate. The core protocol concept is intuitive, but full appreciation requires familiarity with forecasting benchmarks, leakage modes, and rating systems such as ELO.

Scope: The paper introduces a live, continuously running benchmarking platform that eliminates information leakage in TSFM evaluation by requiring forecasts to be submitted before the corresponding ground truth data exists, and reports one year of operating results across 186 energy time series.

What This Paper Is About

Evaluating time series forecasting models on historical data is unreliable because benchmarks and model training sets can overlap, and because correlated time series from the same period leak future information across train and test splits. TS-Arena solves this by shifting evaluation to the actual unknown future: models must pre-register predictions before the real-world values they are predicting are physically generated, making contamination structurally impossible. The paper describes the protocol, the microservice platform that implements it, and the empirical results from running it continuously over a full year of energy data.

Key Contributions

1. Forecast Pre-Registration Protocol (FPRP): A formal, temporally strict protocol in which forecasting models commit to predictions for a fixed horizon before the first future ground-truth value exists, eliminating both direct test-set contamination and indirect temporal leakage by construction.

2. A live benchmarking platform (TS-Arena): A modular microservice architecture (TimescaleDB, Data Portal, API Portal, Dashboard API, front-end, Reference Model Service) that ingests real-time data, orchestrates challenge rounds, validates submissions against active registration windows, and triggers scoring automatically.

3. A leakage-free evaluation and ranking methodology: MASE as the scale-independent error metric, ELO ratings adapted from chess for pairwise model comparison, intersection-based pairwise updates to handle asynchronous model entry, and a Randomized Replay Bootstrap (B=500) to produce confidence intervals that reflect how much evidence backs each rating.

4. A public archive and initial benchmark results: Over 5,000 challenge rounds and 3,928,191 forecast points from the 2025 backtesting year, plus a quarterly-refreshed HuggingFace dataset mirroring the live leaderboard for offline experimentation.

Main Findings

• TSFMs dominate the leaderboard, but the top is statistically crowded. Chronos-2 leads with a global ELO of 1289, followed by TiRex (1270) and Moirai-2-small (1263). Their confidence intervals overlap, meaning more evaluation rounds are needed before any of these can be called definitively superior.

• ELO ranking and MASE can disagree meaningfully. TiRex achieves the lowest global MASE (0.682) but ranks second in ELO, because it produces consistently moderate errors, whereas larger models may win more head-to-head comparisons through occasional strong fits while suffering larger aggregate deviations.

• Statistical baselines are clearly separated from TSFMs. Seasonal-naive is the strongest baseline at ELO 922, while simple-moving-average sits at the bottom (ELO 609), establishing a floor that TSFMs comfortably exceed.

• Model scaling within a family improves performance monotonically. Within the Chronos-Bolt family, ELO rises steadily from tiny (1194) through mini (1200) and small (1206) to base (1214).

• Newer family iterations outperform their predecessors. Moirai-2-small (1263) beats both Moirai-base (1113) and Moirai-large (1099); TimesFM-2.5-200m (1240) beats TimesFM-2.0-500m (1178). Larger parameter counts within an older generation do not compensate.

• Domain matters more than architecture family. TSFMs achieve their best performance on energy consumption series and their worst on generation series, which is consistent with generation data being driven by exogenous factors such as weather.

• The vision-based approach underperforms. VisionTS++ (both base and large) ranks near the bottom among TSFMs with ELO scores of 862 and 797, below seasonal-naive.

• The live setting alters the competitive dynamic. New entrants can immediately occupy top positions, but their wide confidence intervals correctly flag that short-term success is not yet evidence of genuine generalization.

• Confidence intervals shrink with participation. TiRex, having completed 3,319 rounds, has narrower bounds than Chronos-2 at 1,488 rounds, illustrating that the platform rewards sustained participation with more reliable ratings.

Methodology in Plain English

The researchers inverted the standard benchmarking workflow. Instead of picking a historical dataset and splitting it into train and test, they set the split point at the current moment, which advances in real time. Models get all historical observations up to that point, then must submit their forecasts for the next fixed horizon during a short registration window that closes before the next real observation arrives. Any submission after the window is rejected outright.

Time series are bundled into "challenges" grouped by domain, frequency, and horizon. Each challenge runs as a repeating sequence of rounds, effectively producing continuous, automated time-series cross-validation against data that did not exist when the forecasts were made. The platform ingests data from external sources such as SMARD, Fingrid, and Gridstatus, normalizes it into a unified schema, and archives it using a slowly changing dimension strategy so that researchers can reconstruct exactly what information was available at any past inference moment, separate from later corrections.

Scoring uses MASE, chosen because it is scale-independent and anchored to a naive forecast baseline. Models are ranked with an ELO system computed from pairwise wins and losses per round. To handle models entering at different times, comparisons are only made in rounds where both models participated, so established models keep their full history while newcomers are compared only where they overlap. Because ELO is order-sensitive, the authors randomly permute the chronological sequence of rounds 500 times, replaying the full history each time to derive 95% confidence intervals. Long-tenured models converge to narrow intervals; recent entrants get wide ones that honestly signal uncertainty.

A Reference Model Service runs containerized implementations of published TSFMs using their official code and recommended zero-shot settings, submitting forecasts through the same public API as any external participant, so reference models face identical constraints.

Why This Matters

Impact on research. The paper reframes a structural problem in forecasting evaluation. Prior benchmarks such as TSFM-Bench, GIFT-EVAL, TempusBench, and Fidel-TS attempt to mitigate leakage through dataset curation, fixed test sets, or API-gated data, but none removes the possibility that today's test set becomes tomorrow's training data. By making ground truth nonexistent at inference time, TS-Arena converts a curation problem into a protocol guarantee. This changes what a leaderboard can legitimately claim about model generalization.

Real-world applications:

• Electricity grid operations: Load, generation, and price forecasting at market bidding zone, transmission system operator, and trading hub levels, directly supporting scheduling and balancing decisions.
• Energy trading: Day-ahead price forecasting at 15-minute and hourly resolutions where stale or contaminated evaluation gives a false sense of model reliability.
• Renewable integration: Assessing whether zero-shot foundation models can handle variable wind and solar generation without retraining, which matters for operators with limited labeled data.
• General practitioner model selection: The public leaderboard and quarterly archive let organizations identify which models actually generalize in their domain rather than relying on benchmarks that may be inflated.

Industry relevance. Organizations that deploy forecasting models face the same leakage risks in their internal evaluation pipelines. The pre-registration concept is transferable to any setting where ground truth arrives over time, and the platform's microservice design is a reusable template for continuous, tamper-resistant model monitoring in production.

Future Directions

• Expanding beyond univariate, point forecasting. The platform currently evaluates univariate series with point estimates only. Adding covariates, multivariate forecasting, and probabilistic scoring rules such as CRPS is explicitly planned.

• Broadening domain coverage. The current implementation covers energy exclusively. Extending to finance, operations, and other domains with real-time data availability would test whether the observed architecture and scaling patterns hold generally.

• Determining the true magnitude of leakage in existing benchmarks. The authors note that without a leakage-free reference, the actual inflation effect in other leaderboards remains unknown. Running comparable models under both regimes would quantify this.

• Distinguishing genuine generalization from regime-specific over-performance. The paper observes that newcomers can top the leaderboard early with wide confidence intervals. How long an evaluation window is needed before a ranking becomes trustworthy remains an open empirical question.

• Reintroducing excluded series and handling irregular generation patterns. Coal and natural gas generation were omitted due to long constant-zero stretches and exogenous spikes; a covariate-aware benchmark would allow their return.

Target Audience

Forecasting researchers and machine learning evaluation methodologists will find the protocol contribution most valuable, particularly those working on benchmark design or concerned about contamination in foundation model evaluation. Time series practitioners in energy and related domains benefit from the concrete leaderboard results and the downloadable quarterly archive for offline model comparison. Platform engineers and MLOps teams will find the microservice architecture useful as a reference design for continuous model evaluation systems. The paper is most useful to readers with some background in time series forecasting and benchmarking; those without that background can still follow the core argument, which rests on a straightforward temporal ordering constraint.

Authors’ abstract

Time Series Foundation Models (TSFMs) are transforming the field of forecasting. However, evaluating them on historical data is increasingly difficult due to the risks of train-test sample overlaps and temporal overlaps between correlated train and test time series. To address this, we introduce TS-Arena, a live forecasting platform that shifts evaluation from the known past to the unknown future. Building on the concept of continuous benchmarking, TS-Arena evaluates models on future data. Crucially, we introduce a strict forecasting pre-registration protocol: models must submit predictions before the ground-truth data physically exists. This makes test-set contamination impossible by design. The platform relies on a modular microservice architecture that harmonizes and structures data from different sources and orchestrates containerized model submissions. By enforcing a strict pre-registration protocol on live data streams, TS-Arena prevents information leakage offers a faster alternative to traditional static, infrequently repeated competitions (e.g. the M-Competitions). First empirical results derived from operating TS-Arena over one year of energy time series demonstrate that established TSFMs accumulate robust longitudinal scores over time, while the continuous nature of the benchmark simultaneously allows newcomers to demonstrate immediate competitiveness. TS-Arena provides the necessary infrastructure to assess the true generalization capabilities of modern forecasting models. The platform and corresponding code are available at https://ts-arena.live/.