TSFLab: A Self-Improving Agent-Agnostic Workbench for Time Series Forecasting Research

AI summary

Overview

Research area: Machine learning — automated/AI-agent-driven research infrastructure for time series forecasting (TSF), sitting at the intersection of forecasting model libraries, benchmarking frameworks, and LLM coding agents.

Technical level: Intermediate. The paper is readable without deep mathematical background, but it assumes familiarity with forecasting benchmarks, model codebases, and LLM coding agents.

Scope (1 sentence): The paper introduces TSFLab, a shared, agent-agnostic "workbench" of domain guidance, structured knowledge, and a unified experiment runner that lets any compatible coding agent reproduce, benchmark, and develop TSF methods more completely and more cheaply.

What This Paper Is About

Time series forecasting produces far more papers than usable code, and the libraries that do exist are hand-maintained and fall behind the literature — the authors report that only about one in five of the 1,707 TSF arXiv papers since 2020 provides a code link, and that TSLib grew from 18 to 41 methods despite over 1,200 new papers since 2024 before announcing in April 2026 that it is no longer maintained. Directly pointing coding agents like Claude Code or Codex at this landscape is expensive and unreliable: their knowledge is frozen at a training cutoff, and each agent independently re-derives experimental settings, producing incomparable results and repeated work. TSFLab's goal is to move domain support outside the agent into a reusable, self-maintaining workbench so that any compatible agent can conduct TSF research at lower cost and under consistent protocols.

Key Contributions

1. Problem diagnosis. The authors identify two causes of agent inefficiency in TSF research: slowly updated TSF knowledge (training cutoffs, costly web search, propagating errors — illustrated by the official Informer code mistakenly dropping the last test batch in evaluation, which influenced many later studies) and inconsistent research settings across independently configured agents.

2. An agent-agnostic workbench. TSFLab is proposed as plug-and-play infrastructure that augments any compatible agent without changing its model or harness, formalized as π* = Augment(π, W). It has four modules: TSF-Domain Guide, Knowledge Structurer, Experiment Workflower, and Recursive Improver.

3. A large integrated resource catalog. Using TSFLab, the authors integrated 323 TSF models and compiled 93 datasets, plus 71 model units reusable for autoresearch; 263 methods reuse at least one shared unit.

4. Three-task evaluation. They evaluate on paper reproduction, benchmarking on new data, and an agent-driven method-development pilot, plus module and agent ablations and an analysis of coverage, code compactness, and maintenance.

Main Findings

• Higher completion, lower cost in reproduction. Using Codex CLI with GPT-6 Luna, TSFLab completed 12/12 reproduction attempts in 1.96 h with 0.434 M tokens and 159.67 tool calls per completed outcome, versus TSLib (10/12, 2.61 h, 0.533 M), TFB (7/12, 2.46 h, 0.487 M), and no workbench (8/12, 3.50 h, 0.706 M).

• Higher completion in benchmarking. On benchmarking two new datasets (ERCOT and Openmeteo) with eight methods, TSFLab completed 42/48 results, against TSLib 30/48, TFB 16/48, and no workbench 26/48.

• Reported headline gains. Across baselines, TSFLab improves task completion by 20–163% and cuts fresh tokens per completed task by 11–66%.

• Closer agreement with published numbers. TSFLab reproduces 73% of the 96 scheduled reproduction results within 10% of the paper's reported MSE, versus 21–46% for the other settings.

• Simpler added code. In benchmarking, TSFLab requires zero added code because all eight methods are already integrated and run from configuration. In reproduction, agents add a similar number of lines but 15–29% fewer functions and 18–28% less added cyclomatic complexity.

• An existing codebase alone is not enough. TFB completed fewer outcomes than the no-workbench setting on both tasks, mainly because agents replaced requested methods with substitutes.

• Every module matters, and each fails differently. Removing the TSF-Domain Guide dropped completion to 3/12 (agents wrote their own data loaders and every rejected attempt used wrong ETTh1 split boundaries); removing the Experiment Workflower gave 5/12 with the most added code and evaluation errors such as tuning on test scores; removing the Knowledge Structurer gave 8/12 with more added code and complexity.

• Agent-agnostic behavior. Swapping in Claude Code with Sonnet 5.5 gave 11/12 and Codex CLI with GPT-6 Sol gave 12/12, matching the full workbench's completion. Stronger configurations reduced tool calls by 48.5% (Sol) and 51.1% (Sonnet 5.5) and fresh tokens by 9.2% and 55.1%, but list prices imply estimated costs of 18.2× and 9.0× Luna's — so a lightweight agent can match completion at lower estimated cost.

• Broad coverage with compact code. TSFLab contains 44 of the 50 most cited methods, compared with 29 across the three other codebases combined. Median implementation length is 144 lines (close to TSLib's 145), and the largest is 505 lines versus 2,071 in TFB.

• Maintenance through reuse. One unit extraction reduced the average function count from 11.9 to 8.6; a refactor reduced maximum function complexity from 20 to 9 while preserving bit-identical parameters, outputs, and gradients on reference cases for all 323 methods. All 17 initial usability failures were repaired before integration.

• Integration cost is mostly not coding. Model implementation accounts for only 9% of method-intake tokens; source reading, catalog search, and verification account for 28%, 27%, and 28%. The mean intake uses 6.1 M tokens including cache reads.

• Method-development pilot. On ETTh1 and Weather, TSFLab reaches the lowest test MSE with 32.8–73.6% fewer fresh tokens and 9.2–68.8% less development time than the other settings.

Methodology in Plain English

The researchers built a shared "workbench" that sits outside any single AI agent and gives it four things.

First, a domain guide: one harness-neutral entry file (AGENTS.md) plus 17 task-specific skills that spell out workflows, protocol checks, and known pitfalls — for example, reminding an agent to check whether forecasts are scored on one target or all variables, and to keep metrics consistent across methods.

Second, a knowledge structurer: every method, dataset, and reusable unit gets a "knowledge card" read at three levels — a short summary for screening candidates, a card with capabilities, parameters, interfaces, and limitations, and the original source (paper, pinned code revision) for resolving doubts. This lets an agent avoid reading full code and data details unless a decision requires it.

Third, an experiment workflower: agents declare experiments in configuration files rather than writing training scripts; a unified runner expands configurations into a run matrix, validates them before training, executes runs with shared data splitting, train-only scaling, validation-based checkpointing, and consistent metric computation, and supports status, cancel, and resume. Results are compared only within the same protocol.

Fourth, a recursive improver: usability checks (continuous integration, method tests, dataset schema checks), complexity checks (cyclomatic complexity and method size), and a consistency review by an independent read-only review agent that compares cards and implementations against papers and pinned official code. Only resources passing these checks enter the workbench, so each research session can leave behind reusable methods, units, or fixes.

The evaluation compares TSFLab against TSLib, TFB, and a no-workbench setting (empty directory with PyTorch), holding prompt, data, network access, and budgets fixed, on NVIDIA RTX 6000 Ada GPUs. Tasks are: reproducing four recent papers with no official code and absent from all compared codebases (RhyMix, StateFlow, AdaRDiff, CoRe), benchmarking eight methods on two new datasets, and a method-development pilot on ETTh1 and Weather. Reproduction and benchmarking are scored by reviewed completion, cost per completed outcome, and net code changes.

Why This Matters

Impact on research. The paper reframes agent infrastructure: instead of improving one agent's harness, it improves a shared artifact that all agents and teams draw from. It also targets a concrete measurement crisis in TSF — that results and rankings shift with evaluation settings and tuning budgets, and that implementation errors like the Informer test-batch drop can propagate through the literature. By tying every reported number to a declared protocol and recording paper-code discrepancies openly, TSFLab makes comparisons inspectable and reusable rather than re-derived.

Real-world applications:

• Energy forecasting, where the ETTh1 dataset (hourly, 7 channels, target OT, 6:2:2 split) and the ERCOT benchmarking dataset are drawn from — utilities and grid operators comparing methods on their own data.
• Weather and meteorology forecasting, via the Openmeteo and Weather datasets used in benchmarking and method development.
• Transportation and finance demand/series forecasting teams that need consistent baseline comparisons before deploying a model.
• Public health and general operational forecasting, where teams with limited ML engineering capacity need verified baselines without reimplementing them.

Industry relevance. The cost findings matter for organizations running agents at scale: tokens and time per completed outcome, not raw capability, determine feasibility. The finding that a lightweight agent can match stronger, roughly 18.2× and 9.0× more expensive configurations on these tasks suggests the workbench, not the model tier, is where the leverage is. The catalog and pinned, license-recorded, protocol-checked resources also map directly onto reproducibility and audit requirements in industrial ML teams.

Future Directions

• Generalize beyond forecasting. The authors state that TSFLab's guidance, resource descriptions, and experiment interfaces are designed for TSF, and their effectiveness in other domains remains untested; extending it would require adapting resources and checks to each domain's tasks and conventions.

• Produce a genuinely stronger method. Within the limited search time, method development did not yield a new method that substantially outperforms all leading forecasting models, so better search strategies and experimentation over longer research cycles remain open.

• Isolate which checks matter. The authors note that the maintenance cases illustrate how the Recursive Improver turns recurring problems into shared improvements, but do not isolate the effect of individual checks.

• Scale the shared-update cycle. The claim that research outputs feed both improvement and maintenance as models, tasks, and agent harnesses evolve is demonstrated on a fixed snapshot; how the workbench behaves as coverage and contributions grow is left to future work.

Target Audience

Researchers and engineers working on time series forecasting who need to reproduce, benchmark, or extend many methods under consistent protocols; practitioners evaluating whether LLM coding agents are cost-effective for ML experimentation; and builders of agent infrastructure or scientific workbenches interested in a concrete, domain-specific design where support lives outside the agent. Readers focused on new forecasting architectures will find the paper is about research tooling, not a new model — the authors state the pilot demonstrates support for development tasks rather than a broadly superior forecasting method.

Authors’ abstract

As publications and products on time series forecasting (TSF) proliferate, researchers face a growing burden of analyzing existing work. First, not all publications release code. For instance, only about 20% of arXiv TSF papers have a code link in the surveyed period. Second, released code also requires manual maintenance. Thus, TSF researchers must constantly adapt or implement existing works. However, independent implementations can differ in experimental settings and are difficult to maintain. We therefore propose to use AI agents for benchmarking TSF research and even supporting autoresearch, e.g., developing new TSF models. Yet directly applying existing coding agents, such as Claude Code and Codex, is suboptimal: prohibitive costs, limited scalability to large research projects (e.g., benchmarking), and lack of self-maintenance. To address these challenges, we propose TSFLab, a self-improving agent-agnostic "workbench" that equips compatible coding agents with shared resources and tools for low-cost TSF research and continuous maintenance of research resources. Our experiments cover reproduction, benchmarking, and new method development. In reproduction and benchmarking, TSFLab improves task completion by 20-163% and cuts fresh tokens per completed task by 11-66% across baselines. A pilot study of agent-driven method development also shows improved forecasting accuracy via TSFLab, with 32.8-73.6% fewer fresh tokens and 9.2-68.8% less development time than the other settings.