Rethinking Agentic Model Orchestration for Time Series Forecasting

AI summary

Overview

Research area: Time series forecasting, specifically the evaluation and benchmarking of agentic model orchestration (AMO) systems that select or combine time series foundation models (TSFMs).

Technical level: Intermediate. The paper is readable without deep mathematical background, but it assumes familiarity with forecasting error metrics (MASE, CRPS), ensemble/weighting concepts, and the idea of foundation models. The core argument is conceptual and experimental rather than mathematically dense.

Scope: The paper introduces FairAMO, a controlled evaluation protocol with a frozen, cached bank of candidate forecasts, and uses it on GIFT-Eval to separate the contribution of an orchestrator from the contribution of the candidate models it chooses among.

What This Paper Is About

Different time series foundation models are good at different forecasting tasks, so recent work uses "agentic model orchestration" — systems that pick or blend several TSFMs — and reports large improvements over any single model. The problem is that these systems are each tested with different candidate pools, so it is unclear whether the gains come from clever orchestration or simply from having stronger or more diverse models in the pool. The paper builds a controlled protocol that freezes the candidate forecasts and the evidence available to every orchestrator, then measures how much of the reported improvement the orchestrator itself is actually responsible for.

Key Contributions

1. A taxonomy of AMO mechanisms. The authors sort existing systems into three groups by how the orchestration decision is formed: LLM-mediated (MoiraiAgent, TSOrchestra's agent, TimeCopilot's agent framework), feature-conditioned (TimeRouter, Toto-2.0 FnF, FFORMA), and historical-error-driven (RAES-Conductance-V, TSOrchestra SLSQP).

2. A reproducibility audit. Based on releases inspected in September 2026, the authors report that MoiraiAgent, TimeRouter, and Toto FnF ship inference artifacts but no complete training code for their orchestration components, while RAES-Conductance-V and TSOrchestra SLSQP expose code for deriving and applying decisions. Methods therefore have to be reproduced by reusing or adapting available repository components.

3. FairAMO, a controlled evaluation protocol. A three-stage framework — a Frozen Forecast Bank, orchestrator fitting on a fixed evidence budget, and test evaluation with the official GIFT-Eval scorer — in which every orchestrator sees the same candidate forecasts and the same permitted history, and must commit to a selection or a set of weights before test labels are revealed.

4. An empirical decomposition of orchestration gains. Results that separate pool properties (strength and complementarity) from orchestrator ability, using headroom, gain, and realization as the derived quantities.

Main Findings

• Orchestration gains are real but small and bounded. On the main 5-candidate pool under the val+B10 evidence budget, the best deployable orchestrator is TSOrchestra SLSQP at 0.658 MASE, a gain of 0.039 (5.6%) over the best single TSFM, TiRex2, at 0.697. The abstract frames this as orchestrators reducing MASE by less than 6% relative to the best single TSFM across all tasks.

• Most of the available ceiling is left uncovered. The window-level oracle — a non-deployable upper bound that reads test labels — reaches 0.580. Even the strongest orchestrator leaves roughly two thirds of the distance to that ceiling uncovered. Argmin Router reaches 0.662 (29.8% realization) and TimeRouter 0.665 (27.7%).

• Several orchestrators lose to the best single model. Random (0.731), Equal Weight (0.710), Toto-2.0 FnF (0.708), and Dev-Selected Best (0.702) all end up worse than the 0.697 best single TSFM. The two re-implementations, MoiraiAgent-Retrained and Small Patch Router, both reach 0.693, a gain of only 0.004 (3.6–3.8% realization).

• Every method that gains more than 0.01 relies on backtest evidence. Methods whose decisions do not read training backtests — Random, Equal Weight, TimeCopilot Median, RAES-Conductance-Ensemble-V — produce constant scores across every evidence budget tested.

• Gains come from complementarity, not pool size. As the pool grows from two to five members, the best single TSFM barely moves (0.699 to 0.697) while the oracle falls from 0.647 to 0.580, so headroom grows from 0.052 to 0.117. Argmin Router improves from 0.689 to 0.662 and its realization rises from 18.4% to 29.8%, but increments shrink: the third member adds 5.1 points of realization, while the fourth and fifth add about 3 points each.

• Historical evidence saturates quickly, and too much can hurt. Argmin Router improves from 0.672 with validation only to 0.661 once three training backtests are added, then drifts back to 0.664 at 100 backtests and 0.663 at 300. TSOrchestra SLSQP peaks at 10 backtests with 0.658 and degrades beyond that.

• Composition matters as much as size. The same Argmin Router spans 0.670–0.690 MASE across the ten size-3 subsets, a spread of 0.020 that matches in magnitude the 0.027 gained by growing the pool from two to five members. Realization spreads from 9.8% to 32.3%, a factor of three. Best-single-TSFM values across subsets fall between 0.697 and 0.703, a difference of roughly 0.006 — so the sensitivity is about complementarity, not strength.

• A weak model can still help if its errors differ. Toto2-313m is the weakest single model at 0.703, yet adding it to the six size-2 subsets that exclude it improves five of them, by up to 0.013 MASE. Conversely, Chronos2+Toto2-313m has a headroom of 0.048, close to the median, but realizes only 3.1% because its members fail on the same configs and the evidence cannot separate them.

• Simple rules dominate. The strongest methods are simple selection and weighting rules (Argmin Router, SLSQP), leaving the contribution of LLM-mediated orchestration unclear.

Methodology in Plain English

The authors treat every orchestrator as a function over a fixed set of candidate models, and remove all the moving parts that normally differ between systems.

First, they run each candidate TSFM once on GIFT-Eval (a benchmark covering more than 144,000 time series and 7 domains) and cache its quantile forecasts. Nothing is retrained or fine-tuned afterwards. The cache is indexed by dataset, horizon term (short, medium, or long), series, rolling forecast window, candidate, and quantile, and split into three regions: training backtests and optional validation backtests, both with ground truth, and test forecasts, whose ground truth is withheld.

Second, each orchestrator is fitted using only the permitted evidence — the N most recent training backtests plus validation backtests if available — and must commit to either one model selection or a set of combination weights. After commitment, no rerunning, reweighting, or peeking at test scores is allowed.

Third, the frozen decision is applied to the cached test forecasts and scored by the official GIFT-Eval scorer, which returns MASE and CRPS. Because every method draws from the same bank, differences in score can be attributed to the orchestrator rather than to the candidates.

To make pools comparable, the authors report three derived quantities: headroom (best single TSFM minus the window-level oracle, a property of the pool alone), gain (best single TSFM minus the method's MASE), and realization (gain divided by headroom, the fraction of available headroom a method converts into accuracy). They then sweep three things independently: pool composition, pool size, and evidence budget. Eleven deployable orchestrators are compared — five reference baselines (Random, Dev-Selected Best, Equal Weight, Argmin Router, and the non-deployable Oracle), five published methods reproduced as style adaptations, and two of the authors' own re-implementations (MoiraiAgent-Retrained and Small Patch Router).

Why This Matters

Impact on research. The paper argues that leaderboard position alone cannot establish that an orchestration mechanism is responsible for a system's score, because candidate strength and candidate complementarity jointly determine the result before the orchestrator does anything. It offers a protocol and a set of reporting conventions — notably that orchestration gains should be reported alongside pool composition and realization against headroom, not as raw MASE deltas. It also documents that key training code for several prominent AMO systems was not publicly available as of September 2026, which constrains reproducibility.

Real-world applications:

• Energy load and demand forecasting, where operators must choose among several pretrained forecasters and need to know whether a router actually adds value or just happens to be pointed at a good model.
• Retail and supply-chain planning, where forecasting pipelines mix short, medium, and long horizons across thousands of series with heterogeneous seasonality.
• Financial and economic monitoring, where practitioners deploying foundation models need an honest estimate of the marginal benefit of added orchestration infrastructure.
• Industrial and infrastructure telemetry, where rolling forecast origins and covariates change the best-performing model from series to series.

Industry relevance. The main practical message is deflationary for complex orchestration stacks: a simple per-config argmin over recent backtest error, or a small weighting rule, was competitive with or better than the published and re-implemented agentic systems. The finding that a few recent windows capture essentially all the value of historical evidence, and that larger evidence budgets can hurt, also means the data plumbing for an orchestrator can be kept small. For teams building forecasting platforms, that translates into lower engineering cost and a clear argument for selecting pool members by error-profile diversity rather than by single-model rank.

Future Directions

• Finer-grained commitments. A single decision per config limits how much of the window-level headroom can be realized; the authors suggest exploring decisions made at a finer granularity than one per config.
• Agentic methods that use context beyond backtests. The paper notes that LLM-mediated orchestration's contribution remains unclear under FairAMO, and proposes agents that draw on signals other than historical forecast errors.
• Pool construction by complementary error profiles. Since subsets containing TiRex2 realized 28.5% of headroom on average versus 13.4% for subsets without it, and since a weak model (Toto2-313m) improved five of six pools it joined, the authors argue pools should be grown by difference in error profile rather than by model rank.
• Closing the gap to the oracle. The strongest orchestrator still leaves roughly two thirds of the 0.117 headroom on the main pool unrealized, so better selection and weighting mechanisms — or better ways of exploiting evidence — remain open.

Target Audience

Researchers and practitioners working on time series foundation models, forecasting benchmarks, and model-selection or ensembling systems will get the most from this paper. It is particularly relevant to anyone who reads the GIFT-Eval leaderboard, builds forecasting platforms that route between multiple pretrained models, or evaluates agentic and LLM-based orchestration — and to reviewers and benchmark maintainers who need a controlled protocol for attributing system-level gains to a specific mechanism. Readers without a forecasting background can follow the argument, but will need to look up MASE and CRPS to interpret the numbers.

Authors’ abstract

Time series foundation models (TSFMs) perform unevenly across forecasting tasks. Recent work uses agentic model orchestration (AMO) to select or combine TSFMs, often outperforming individual forecasters. Yet these gains may reflect stronger or more complementary candidates rather than better orchestration. To disentangle these effects, we introduce FairAMO, a controlled evaluation protocol with frozen candidate forecasts. Experiments on GIFT-Eval show that AMO improves accuracy, but its gains are bounded by the candidate TSFMs. The best orchestrators reduce MASE by less than 6% relative to the best single TSFM across all tasks. These gains depend on complementarity (different candidates excel on different tasks) rather than the number of candidates. Adding members yields diminishing returns, and a redundant member can erase these gains. The value of historical evidence also saturates quickly. A few recent windows capture all gains, and further history adds none. These findings separate system-level forecasting gains from the contribution of orchestration itself.