Timestral

AI summary

Overview

Research area: Time series foundation models (TSFMs) for probabilistic forecasting, within artificial intelligence and machine learning.

Technical level: Advanced. The report assumes familiarity with Transformer architectures, attention masking, patch-based tokenisation, quantile (pinball) loss, and the prior-data fitted network (PFN) view of forecasting. The plain-language sections below make the core ideas accessible, but the paper itself is a technical model report.

Scope: A technical report introducing the Timestral family of four forecasting foundation models (5M to 209M parameters), describing their architecture (Decoupled Horizon Forecasting, covariate handling, two-stage synthetic-then-mixed training) and reporting benchmark results on GIFT-Eval, TIME, fev-bench, and IND-Eval, plus inference-speed measurements.

What This Paper Is About

Most forecasting practice fits a separate model per dataset. Time series foundation models instead train once on large collections of series and forecast new data without retraining. Timestral is a new family of four such models, and the paper's central claim is that a specific design change in how models are trained on forecast horizons, plus a careful treatment of covariates, yields better accuracy than much larger competitors: Timestral-209M ranks first on every reported metric of GIFT-Eval, TIME and fev-bench among the compared models, with Timestral-123M second.

Key Contributions

1. Decoupled Horizon Forecasting (DHF). A training formulation that gives each patch two tokens during training — a context token holding observed values and a prediction token at which the forecast is made. Context tokens never attend to prediction tokens, so the context is encoded once and shared by all forecast blocks, and longer horizons are trained without masking away context. This is positioned explicitly against span masking ("cpm", used in TiRex, Toto-2.0 and TimesFM-3), where patches masked to create long-horizon forecasts are also lost as context for later forecasts.

2. Role-aware covariate conditioning inside a causal architecture. Each variate is assigned one of three roles — target variate, past-only covariate, or known-future covariate — which enter only through attention masks and normalisation choice, not through learned role embeddings. Known-future covariates attend over the entire window including the forecast horizon, but only to each other, so their future values can inform forecasts without leaking future target values.

3. A released family spanning accuracy and inference cost. Four models with the same architecture and training recipe (5.2M, 27.8M, 122.6M and 209.3M parameters), all lying on the accuracy-versus-parameter-count Pareto front across the six reported benchmark panels, with weights on Hugging Face and inference code on GitHub.

4. A two-stage training pipeline. Pretraining exclusively on a designed synthetic corpus (univariate signals plus multivariate dependencies built from structured synthetic priors) at 8,192-step context, followed by post-training on a mixture of real and synthetic data at 32,768-step context for substantially fewer steps.

Main Findings

• Top rank on all six metrics: Timestral-209M ranks first on all six metrics among the models compared — lowest MASE and CRPS on GIFT-Eval and TIME, and highest MASE and SQL skill scores on fev-bench — using a single checkpoint. Timestral-123M ranks second on all six, ahead of every model outside the family.

• Smaller than the models it beats: Timestral-123M and Timestral-209M have fewer parameters than t0-gamma (256M) and TimesFM-3 (331M). Timestral-28M outperforms Chronos-2 and Toto-2.0-2.5B — roughly 4 and 90 times larger — on all six metrics.

• The smallest model is competitive: On fev-bench, Timestral-5M outperforms every Toto-2.0 model, including Toto-2.0-2.5B, which has about 480 times as many parameters.

• Industrial benchmark: On IND-Eval, Timestral-209M, Timestral-123M and Timestral-28M take the top three places, and the conclusion states the model leads by a wider margin there.

• Pareto front: All four Timestral models lie on the Pareto front of forecast error against parameter count across the six panels of Figure 1.

• CPU-capable inference: No GPU is required. For a 96-step univariate forecast from 2,048 steps of context with symmetric averaging, latency on 8 CPU threads is 17 ms (5M), 31 ms (28M), 85 ms (123M) and 113 ms (209M); GPU throughput on one NVIDIA H100 ranges from 3,650 to 18,770 series per second.

• Throughput translates to modest hardware: One CPU core forecasts 3.5 to 106 series per second, so 100,000 series every 15 minutes requires 2 to 32 cores; on one H100 the same batch takes 5 to 27 seconds.

• Accuracy/cost trade-off inside the family: Timestral-28M is 3.6 times faster than Timestral-209M, and Timestral-123M has about 0.2% higher GIFT-Eval MASE than Timestral-209M at 75% of its latency. Timestral-209M becomes 3.1 times faster going from 1 to 8 threads, whereas Timestral-5M is already as fast on a single core.

• Long-context cost profile: Latency changes little up to 2,048 steps (roughly doubling from 128 to 2,048), then grows by about 9 to 12 times from 2,048 to 32,768 steps; at 32,768 steps a forecast takes 119 ms with Timestral-5M and 1.3 s with Timestral-209M. A 32-variate series takes 7 to 15 times as long as a univariate one, and one target with seven known-future covariates takes 1.7 to 2.0 times as long as eight target variates.

• Not reported in the supplied content: The per-model aggregate scores for all compared systems are said to be in Table 3 of Appendix A and the pairwise win rates in Section A.3; those tables are not included in the text available here.

Methodology in Plain English

The framing. The paper adopts the PFN view: a forecasting model is trained on many synthetic series drawn from a prior over data-generating processes, so that at test time a single forward pass approximates Bayesian inference over which process produced the observed history. In this view, the composition of the training data is the prior, and it directly controls how confident or uncertain forecasts are. The paper notes that synthetic data already plays a large role in this literature — TimesFM mixes 20% synthetic series, Chronos samples 10% from KernelSynth, TiRex about a third from an extended KernelSynth, Toto-2.0 uses 57.5% in its pretraining mix, and Chronos-2 learns multivariate and covariate abilities from synthetic data only.

The horizon problem and DHF. Existing causal models forecast several patches ahead by masking spans of the input (cpm). The catch is that a masked patch is unavailable both as a forecast target's preceding context and as context for every later forecast in the same training sequence, which couples how often long horizons are trained to how much context is destroyed. DHF instead gives each observed patch two tokens: a context token carrying its real values, and a prediction token positioned where a forecast is requested. Prediction tokens see the context before their block and earlier prediction tokens in the same block; context tokens never see prediction tokens. A training sequence is tiled into consecutive forecast blocks whose lengths are sampled from a distribution, so the frequency of each forecast distance can be tuned without touching the context. A block of length k yields one forecast at each distance 1 to k, and the context is encoded once and shared. The cost is roughly double the tokens during training, since every patch except the first becomes a forecast target.

Covariates. Each variate is a target, a past-only covariate, or a known-future covariate. Along time, targets and past-only covariates are causal; known-future covariates see the whole window including the horizon. Across variates, information flows among targets and from covariates to targets, but known-future covariates attend only to each other and past-only covariates never read targets — treating covariates as exogenous and preventing a covariate from carrying a future target value backwards through time. During training, some variates in a random subset of series are reassigned as covariates (not scored), which turns ordinary multivariate data into covariate-informed tasks.

Architecture. Patches of P = 32 time steps are embedded by a two-layer residual MLP that takes both normalised values and binary missingness indicators. Values are standardised causally using statistics up to the end of the current patch, then passed through an inverse hyperbolic sine; known-future covariates are normalised over the whole window instead. The backbone is a pre-norm Transformer with RMSNorm and SwiGLU layers, applying temporal attention in every block and variate attention every fourth block, both with sigmoid output gating, and rotary position embeddings on half of each head's dimensions. No timestamps, frequency information, role embeddings, dropout or stochastic depth are used. A final MLP maps each output to 21 quantile levels from 0.01 to 0.99; the median is predicted directly and the other quantiles by cumulative softplus increments, which prevents quantile crossing by construction.

Training and inference. Training uses only the pinball loss, with no auxiliary objectives, averaged equally across quantile levels and scored positions; only observed target values inside forecast horizons are scored. A single inference block covers up to 32 patches (1,024 time steps) in one forward pass, and longer horizons are produced block by block conditioned on previous predicted medians.

Why This Matters

Research impact. The paper isolates a specific, transferable design question — whether horizon sampling during training must be coupled to context masking — and shows that decoupling it produces state-of-the-art results at small parameter counts. Its PFN framing also makes explicit that synthetic data design, not just scale, is a primary lever for forecast quality. The result that a 28M model beats models roughly 4 and 90 times larger, and that a 5M model beats a model about 480 times larger on one benchmark, reframes the efficiency conversation for TSFMs.

Real-world applications (forecasting tasks of the kind these models target):

• Demand and inventory planning, where many related item series must be forecast together and known-future signals such as promotions or holidays are available in advance.
• Energy and utility operations, including short-horizon load forecasting where 96-step-ahead forecasts from several weeks of 15-minute history — the paper's own latency workload — are the practical unit of work.
• Industrial monitoring and sensor telemetry in pipelines, where forecasts feed anomaly detection and alerting, an integration the authors mention directly.
• Financial and business metric forecasting where uncertainty matters, since the models emit full predictive quantiles rather than point estimates.

Industry relevance. Inference without a GPU, latencies of 17 to 113 ms on 8 CPU threads, and a 100,000-series batch needing only 2 to 32 cores for a 15-minute refresh cycle make these models deployable on commodity infrastructure. Released weights and inference code lower the barrier further, and the four-size family lets practitioners pick a point on the accuracy/latency curve rather than adopting a single fixed model.

Future Directions

• Scaling further. The authors state that accuracy still improves from 123M to 209M parameters, which they give as motivation for training larger models. Where the scaling curve saturates, and whether DHF's roughly doubled training token count becomes a bottleneck at larger sizes, is open.

• Embedding-based downstream tasks. The conclusion proposes reusing context representations as embeddings for classification and anomaly detection, extending the models beyond standalone point or quantile forecasting.

• Integration with language models and existing pipelines. The authors suggest embedding Timestral in monitoring pipelines or combining it with language models in agentic workflows; how forecast outputs and quantiles should be exposed to such systems is not addressed.

• Long-horizon behaviour beyond 1,024 steps. Horizons beyond one block are produced block by block conditioned on predicted medians, which the paper acknowledges as an error-compounding path inherited from prior work; how much accuracy is lost in this regime is not quantified in the reported results.

Target Audience

Machine learning researchers and engineers working on time series foundation models, especially those interested in attention-mask design, synthetic data priors, and the PFN interpretation of forecasting. It is also useful for practitioners choosing a pretrained forecaster for production — the latency, throughput and memory tables (17 to 113 ms CPU latency, 21 MB to 837 MB checkpoints) are directly actionable — and for applied scientists in demand planning, energy, and industrial monitoring who need covariate-aware probabilistic forecasts without GPU infrastructure. Readers without background in Transformers or quantile regression will find the architectural sections dense, though the problem framing and results are readable on their own.

Authors’ abstract

We introduce Timestral, a family of four time series foundation models for probabilistic forecasting, with 5M, 28M, 123M, and 209M parameters. Timestral sets a new state of the art on GIFT-Eval, TIME, and fev-bench, where Timestral-209M and Timestral-123M take the top two places on every metric. We achieve this with three ingredients: Decoupled Horizon Forecasting (DHF), which separates prediction tokens from the causally encoded context, so that longer horizons are trained without masking context and each observed value is both a forecast target and context for later forecasts; a covariate framework that natively supports known-future covariates within a causal architecture; and a two-stage pipeline that pretrains exclusively on a diverse, large-scale synthetic corpus before post-training on a mix of real and synthetic data with extended context. This report describes the model design and its benchmark results. Weights for all four models are available on Hugging Face and inference code on GitHub.