TimeRSI: Towards Self-Improved Agentic Deployment for Time-Series Foundation Models

AI summary

Overview

Research area: Time-series forecasting, with a focus on agentic (LLM-driven) deployment around frozen time-series foundation models (TSFMs). Technical level: Advanced — the paper assumes familiarity with foundation-model forecasting, quantile/probabilistic metrics (MASE, CRPS, SQL, WQL, WAPE), and LLM coding agents. Scope: The paper proposes TimeRSI, a two-stage system that lets a coding agent iteratively rewrite the deployment pipeline around a frozen forecasting model using forecast-error feedback, then selects among the resulting pipelines per forecast.

What This Paper Is About

Time-series foundation models forecast well out of the box, but making them work for a specific real-world task — encoding holidays, tides, business rules, or sensor quirks into how the model is fed and how its outputs are corrected — still requires substantial human expertise. The authors build TimeRSI, which automates that adaptation: an LLM coding agent proposes and revises pipeline code, tests it on completed historical forecast windows, and uses the resulting errors to guide the next revision. A second stage then picks, for each new forecast, which of the discovered pipelines to trust based on how those pipelines performed under similar past conditions, while the foundation model's weights stay frozen throughout.

Key Contributions

1. Iterative pipeline improvement with forecasting feedback. A coding agent edits four stages of a deployment pipeline (preprocessing, feature construction, context selection, post-processing) and receives forecasting scores, window-level errors, and execution logs after each revision, with all prior code and evaluation records retained for later revisions.
2. Adaptive pipeline selection. A selector chooses among a retained set of candidate pipelines per target and per forecast origin, using similarity-weighted historical errors of those pipelines under comparable conditions rather than a single fixed pipeline.
3. A benchmark-scale evaluation. TimeRSI is evaluated on all 98 tasks of TIME and all 100 tasks of FEV-bench (198 tasks; 111,177 and 235,039 forecast instances respectively), against 32 published TIME submissions and 31 FEV-bench submissions, reporting the best score on six point and probabilistic metrics.
4. Controlled ablations and generalization checks. A non-iterative search baseline with the same candidate budget isolates the value of feedback-guided revision, and the framework is re-run with a different coding agent (Antigravity / Gemini 3.8 Flash) and different frozen backbones (Chronos-2, LPTM).

Main Findings

• Best across six metrics: TimeRSI ranks first on both benchmarks across MASE and CRPS (TIME) and MASE, SQL, WQL, and WAPE Skill Scores (FEV-bench). Labels in the ranking figure name the frozen backbone of each TimeRSI entry.
• Gains over direct inference with the same backbone: Relative to direct TimesFM 3.0 inference in the authors' environment, TIME MASE falls from 0.639759 to 0.630011 (1.52% relative error reduction) and CRPS from 0.536328 to 0.527713 (1.61%). On FEV-bench, MASE Skill rises from 37.786 to 40.579 (2.79 percentage points) and SQL Skill from 48.984 to 51.243 (2.26 points).
• Non-iterative search fails: With the same number of candidate pipelines, non-iterative search without feedback-guided revision underperforms even direct inference (MASE 0.650 vs 0.640, CRPS 0.549 vs 0.536, SQL Skill 42.832 vs 48.984, MASE Skill 30.509 vs 37.786), while TimeRSI improves on both benchmarks.
• Works with other coding agents: With TimesFM 3.0 fixed, Antigravity (Gemini 3.8 Flash) reaches normalized MASE/CRPS of 0.637/0.532 on TIME, still ahead of compared external submissions.
• Works with other frozen models: With Claude Code fixed, Chronos-2 improves from 0.662/0.556 to 0.647/0.545 (MASE/CRPS) and LPTM from 0.669/0.572 to 0.641/0.549. On all 100 FEV-bench tasks, LPTM improves from SQL/MASE/WQL/WAPE Skill of 43.20/30.70/47.60/34.18 to 46.33/34.63/50.96/38.58.
• Feedback changes strategy, not just parameters: In the parking task, a calendar-predictor blend raised adaptation error (0.9421 to 0.9578), so the agent removed that contribution and instead built 22 calendar input features (daily and weekly sine/cosine terms, weekend and Saturday indicators, weekend–time-of-day interactions), lowering adaptation error to 0.9208. The deployed revision was not the adaptation winner.
• Task-level case studies: Discovered pipelines reduce task-level MASE by 2.86% to 6.70% on the four illustrated tasks — supply chain 0.557750 to 0.520378 (6.70%), parking 0.678730 to 0.641658 (5.46%), ocean currents 0.871531 to 0.835129 (4.18%), solar counters 0.574087 to 0.557693 (2.86%). Appendix cases include auto-production (0.756537 to 0.233980, 69.07%) and COVID reports (3.138536 to 2.890121, 7.91%; CRPS 15.62%).
• Holiday rule discovery: In the supply-chain task (36 targets, 12 windows of 30 days), the agent required at least four past holiday observations with at least 80% zeros before zeroing all nine quantiles, and all 144 target values on the four relevant holiday dates are zero.
• Submission status is stated precisely: The TIME entry is accepted; the FEV-bench TimeRSI row is described as the submitted, not yet merged result.

Methodology in Plain English

The foundation model is treated as a black box whose weights never change. Everything else about deployment is written as Python code in four functions: preprocess (clean and transform the observed history), engineer (build extra variables such as calendar terms or tidal components), select_context (decide which histories and variable groups the model sees), and postprocess (map outputs back to the target scale, combine forecasts, apply corrections).

A coding agent starts from the trivial pipeline — just call the model — and reads the task description. It proposes one code revision at a time, the system runs it on earlier historical forecast windows (the "adaptation" set), and returns scores, per-window errors, and execution logs. The agent can refine its approach, combine ideas that worked, or abandon a direction, and it retains access to all earlier attempts. The default budget is six iterations with one candidate per iteration.

A separate, later block of historical windows (the "selection" set) is used to rank the resulting candidates. From these, the system keeps the top three plus the reference pipeline as the deployment set. At each new forecast origin, for each target, the selector describes current conditions with six numerical features of up to 96 recent observations and the reference median forecast (level relative to recent median, short-vs-earlier mean, first forecast relative to latest observation, mean forecast relative to recent median, forecast spread, and last-vs-first forecast). It standardizes these features, finds up to 12 nearest historical records, weights each by 1/(1+d), and picks the pipeline with the lowest weighted historical error relative to the reference. Selection keeps adapting as outcomes arrive, while the pipeline code stays fixed.

Evaluation uses the official benchmark definitions. On TIME, errors are normalized and aggregated as a geometric mean of each task's error relative to Seasonal Naive over N = 98 tasks. On FEV-bench, Skill Scores aggregate N = 100 tasks with relative errors clipped to [0.01, 100], where 0 matches Seasonal Naive.

Why This Matters

Impact on research. The paper reframes TSFM deployment as a search-and-selection problem over executable code rather than a weight-updating problem, and it provides a clean controlled comparison showing that feedback-guided revision beats non-iterative candidate search at the same budget. It also separates code search (on adaptation windows) from deployment-time selection (on later windows), which is a reusable design pattern for any agent that operates on temporally ordered data.

Real-world applications demonstrated in the paper:

• Supply-chain demand: zeroing forecasts on holidays where a target's own history shows repeated zero demand (for example July 4, Thanksgiving, Christmas, New Year's Day).
• Parking availability: supplying weekend-aware calendar features to the frozen model for 15-minute forecasts across a week (evaluated on 354 car parks).
• Ocean currents: fitting tidal constituents such as M2 (12.42 hours) and K1 (23.93 hours), forecasting the residual, and adding the tide back — for 72-hour, ten-minute-resolution forecasts.
• Solar counters and atmospheric variables: restoring a daily rise–plateau–reset shape for accumulation counters, and removing/restoring a daily harmonic cycle for wind, geopotential height, and temperature.

Industry relevance. The approach requires no retraining or access to model weights, so a company using a vendor's TSFM API can adapt it to its own domain by editing surrounding code. It also converts scarce forecasting expertise into an automated loop, which matters for organizations that have task knowledge (holiday rules, sensor structure) but not forecasting specialists.

Future Directions

• Reduce dependence on good task descriptions. The authors list reliance on clear task and variable descriptions as a limitation, so extracting that context automatically from data and metadata is a natural next step.
• Handle distribution shift. The paper explicitly flags that gains on historical windows do not guarantee future gains, calling distribution shift an open problem — better selection under regime change, or detecting when a discovered pipeline has gone stale, follows directly.
• Scale the search. The default budget is six iterations, one candidate per iteration, with a 256-turn and 90-minute cap; whether more search, parallel agents, or cross-task transfer of discovered programs helps is untested here.
• Richer action spaces. The auto-production case shows the agent can fit auxiliary predictors rather than only adding input features, suggesting room to expand what deployment code is allowed to do (for example learned ensembles or richer probabilistic calibration).

Target Audience

Researchers and practitioners working on time-series foundation models, LLM-based coding agents, and automated machine learning engineering. It is most useful to readers who already understand forecasting metrics and want to see how agentic code search can be coupled to temporally ordered feedback; the case-study traces in the appendix are also valuable for engineers who want concrete patterns for encoding domain knowledge into a frozen model's inputs and outputs.

Authors’ abstract

Time-series foundation models provide strong zero-shot forecast, but deployment for real-world applications still requires task-specific designs for model inputs and outputs. Direct inference can leave domain knowledge and recurring forecast errors unaddressed, while tailored pipelines require substantial human expertise and effort. We introduce TimeRSI, a self-improving agentic deployment system around frozen forecasting models. A coding agent translates task descriptions into executable pipeline changes. It tests each revision on historical forecasting windows and uses observed errors to guide subsequent revisions. During deployment, an adaptive selector chooses among the resulting pipelines using completed forecasts under similar conditions. Across 198 tasks on TIME and fev-bench, TimeRSI outperforms all submissions on six point and probabilistic forecasting metrics. Controlled comparisons support the value of feedback-guided revision over non-iterative pipeline search. Further experiments verify the generalization with different coding agents and forecasting models. Code is available at https://github.com/haoxin1998/TimeRSI.