QUAL: Quantile Alignment Improves Zero-Shot Time-Series Foundation Models

AI summary

Overview

Research area: Time-series forecasting with foundation models; specifically, label-free (zero-shot) forecast combination and probabilistic (quantile) calibration.

Technical level: Intermediate. The paper is written for readers comfortable with quantiles, cumulative distribution functions, and ensemble aggregation, but every component is described in plain prose and the method itself involves no training.

Scope (1 sentence): The paper proposes QUAL, a training-free framework that uses a pool of time-series foundation models as a shared probability reference for re-expressing each member's quantile forecasts, then combines them robustly — evaluated on the GIFT-Eval benchmark across 97 configurations.

What This Paper Is About

Time-series foundation models can forecast series from datasets they were never trained on, but no single model is reliably best, and accuracy shifts from dataset to dataset. In a zero-shot deployment there are no target values, so a practitioner cannot tell which model in their pool is strongest, cannot learn combination weights, and cannot verify that the pool is any good. The paper's goal is to combine a set of heterogeneous foundation models into one probabilistic forecast using only the quantile forecasts they return for the current request — no labels, no retraining, no model modification.

Key Contributions

1. Identifies the pooled CDF as a label-free probability reference. Rather than collapsing a model pool into a single forecast (as the linear opinion pool does), the authors show that the pool's mixture CDF can serve as a shared reference for relabelling each member's quantiles. This improves every evaluated forecaster in aggregate, including under leave-one-out references where the evaluated model is removed from the pool.
2. Proposes QUAL (Quantile Alignment), an end-to-end framework. It chains three steps: member-wise quantile alignment, Huber robust aggregation across the aligned members, and quantile rearrangement to guarantee non-crossing quantiles — all without training or model updates.
3. Shows that keeping aligned members separate is what buys robustness. Because member forecasts are never collapsed before aggregation, disagreement remains visible, and the Huber centre can downweight an atypical member locally at each probability level instead of averaging it in.
4. Quantifies pool-selection risk empirically. Across 247 multi-model subsets of the strong pool, 218 beat the most accurate individual model on both MASE and CRPS, with win rates rising with pool size.

Main Findings

• Member-level improvement: On GIFT-Eval, quantile alignment reduces average member MASE by 7.4% and average CRPS by 9.8%. Every member improves on both metrics, from TiRex-2 (MASE 0.6969 → 0.6810, a 2.3% gain) down to the seasonal-naive reference (24.7% MASE and 34.3% CRPS gains).
• Not a self-inclusion artifact: Leave-one-out references, built after removing the evaluated model from the pool, give nearly identical results — average LOO MASE 0.6910 (7.4% improvement) and LOO CRPS 0.4872 (11.3% improvement).
• Full framework scores: QUAL reaches 0.6793 MASE and 0.4646 CRPS, and the paper states it beats every zero-shot system in the pinned GIFT-Eval snapshot.
• Robustness to weak forecasters: Adding three weak members (last value, a Gaussian fit, history mean) to the strong pool degrades CRPS by only 5.6% under QUAL versus 11.9% under the linear opinion pool; QUAL degrades less on 93 of the 97 configurations.
• Consistent against seasonal naive: Across subsets of the strong pool with seasonal naive added, QUAL has lower CRPS on all 154 subsets of three to five members and on 39 of 56 six-member subsets. When it wins, it wins by up to 0.037 CRPS; when it loses, by no more than 0.001.
• Pool-size dependence shrinks: Of 247 multi-model subsets, 218 beat the best individual model on both metrics. Appendix A.6 reports win rates of 61% at two members, 75% at three, 94% at four, and 100% from five members onward; the introduction and Figure 1 summarize this as 91.8% for subsets of at least three members and always for at least five.
• Ablation: Alignment carries most of the clean-pool gain — alignment plus mean aggregation already reaches 0.4646 CRPS, matching the full QUAL and beating Huber on raw quantiles (0.4657) and quantile mean (0.4677). Huber contributes little on a clean pool but is retained for robustness.
• Labeled weights help, but are unavailable in deployment: Fitted inverse-backtest-loss weights reduce CRPS from 0.4672 to 0.4642 for the quantile average and from 0.4634 to 0.4604 for the linear opinion pool on a six-model pool — but the fit uses observed targets pooled across a dataset.
• Skill transfers only weakly over time: Median Spearman correlation between backtest and forecast-period rankings is +0.43 (quartiles +0.03 to +0.71) and negative in 24 of 97 configurations. The backtest winner is the forecast-period winner in only 31 of 97 configurations, against 17% expected by chance.
• Weak members are genuinely weak: Scored alone against seasonal naive, last value reaches 1.0726 CRPS, the Gaussian 1.2286, and the history mean 1.5855; the history mean and Gaussian share a MASE of 1.7959 because both place their median at the history mean.

Methodology in Plain English

Each foundation model in the pool returns numbers at nine fixed probability levels (0.1 through 0.9). The naive way to combine them is to average the numbers at each level, which throws away information about how one model's value compares to the others.

QUAL instead treats the pool as a ruler. First, each model's reported quantiles are turned into a smooth curve describing how much probability that model assigns to values below any given point. Averaging those curves across all pool members gives a pooled curve — the average percentile that the whole pool assigns to any value. Second, each model's own quantile values are looked up on this pooled curve, producing a new set of probability levels that reflect where those values sit relative to the rest of the pool; interpolating back gives an aligned forecast for that model, expressed on a shared scale. Third, at each probability level the aligned values from all models are combined with a Huber robust centre: it behaves like an average when models agree but caps the pull of any model that deviates sharply, using a median-absolute-deviation scale to normalise deviations. A model keeps full weight where it agrees with the consensus and is downweighted only where it disagrees. Finally, the combined values are sorted so the output is a valid, non-crossing quantile function.

Nothing is trained. The method runs on CPU in seconds per configuration and requires only the quantile forecasts already returned for the current request.

Why This Matters

Research impact: The paper reframes ensemble combination for foundation models: the pool is useful as a reference distribution, not only as a final average. It also documents a negative result that the forecasting community often assumes away — that historical model rankings transfer only weakly to future periods (median rank correlation +0.43, negative in 24 of 97 configurations) — which undercuts a standard justification for learned ensemble weights.

Real-world applications:

• Supply chain and demand planning, where new product series often have no labelled history to validate a model choice against.
• Energy load forecasting, where operators may run several vendor or open-source models and need one defensible probabilistic forecast.
• Financial planning and risk, where prediction intervals matter as much as point forecasts and mis-weighting a bad model is costly.
• Transportation and operations scheduling, where a heterogeneous model pool is assembled ad hoc and cannot be validated on the target series.

Industry relevance: The framework imposes no retraining, no historical record-keeping, and no extra data collection, so it can be dropped on top of an existing multi-model deployment. The robustness tests address the realistic case where an unvetted model silently enters the pool, and the compute note makes clear that the combination step itself is negligible next to running the foundation models.

Future Directions

• Extending beyond the reported quantile range. Alignment is defined only inside each member's reported range, and QUAL does not extrapolate past the outermost quantiles; tail behaviour beyond 0.1–0.9 is unaddressed.
• Handling heterogeneous quantile grids. The method requires at least two members reporting quantiles at the same nominal levels; standardising or reconciling mismatched reporting conventions is an open engineering question.
• Diagnosing transferable skill without labels. Since backtest-to-future rank correlation is only +0.43 and negative in a quarter of configurations, a label-free signal for when weighting will help — the regimes where fitted weights improve 40 of 42 cases versus only 3 of 24 — remains unsolved.
• Testing outside GIFT-Eval. Evaluation covers 97 GIFT-Eval configurations; behaviour on domains with substantially different quantile calibration or reporting conventions is explicitly not measured.

Target Audience

Machine-learning engineers and forecasting practitioners deploying multiple time-series foundation models in production without labelled validation data; researchers working on ensemble methods and probabilistic calibration; and applied scientists in operations, energy, or finance who need a defensible way to combine off-the-shelf forecasters. Readers should be comfortable with quantiles and distribution functions, though the method itself is implementable from the paper's equations without training infrastructure.

Authors’ abstract

Time-series foundation models (TSFMs) provide zero-shot forecasts for unseen series, but their accuracy varies across datasets. Combining several models can reduce this variability, yet zero-shot deployment provides no target labels for selecting or weighting them. We find that a model pool is useful not only as a final ensemble, but also as a probability reference for transforming each member forecast. Aligning quantiles through this pool-relative reference improves every member forecast in aggregate. Motivated by this finding, we propose QUAL (Quantile alignment), a simple label-free framework for improving and combining diverse TSFMs. QUAL first produces aligned member forecasts, then applies Huber aggregation and rearrangement to obtain a robust combined forecast without training or model modification. On GIFT-Eval, quantile alignment reduces average member MASE and CRPS by 7.4% and 9.8%. The full framework remains robust when weak forecasters enter a candidate pool whose quality cannot be verified in advance. Code is available at https://github.com/junhyeokkang/QUAL.