IND-Eval: Aligning Evaluation and Simulation for Industrial Time Series Foundation Models

AI summary

Overview

Research area: Time series foundation models (TSFMs) and benchmarking for industrial forecasting; simulation-based pretraining data design.

Technical level: Intermediate. The paper assumes familiarity with forecasting metrics (nMASE, nCRPS), benchmark construction, and pretraining/simulation pipelines, though its arguments are stated in plain operational terms.

Scope (one sentence): The paper introduces IND-Eval, a public benchmark of 187 source-defined industrial forecasting tasks from nine public datasets, and pairs it with an exploratory case study showing how target-domain statistics can guide the composition of simulated pretraining data.

What This Paper Is About

Industrial forecasting models are usually compared on broad, heterogeneous leaderboards, but a real deployment concerns one specific process, asset, or batch with its own target variables, sampling rate, forecast horizon, and episode boundaries. The authors build IND-Eval to make those structural choices explicit and to test whether aggregate leaderboard scores actually identify the right model for a given industrial setting. They then ask a second question: when real industrial data are too scarce to close a domain gap by fine-tuning, can simple statistics about a domain guide how simulated training data should be designed?

Key Contributions

1. IND-Eval, a public benchmark of 187 source-defined forecasting tasks drawn from nine public datasets — a six-source industrial core (NEATS, 3W, PRONTO, Mesa del Sol, tablet compression, MetroPT-3) plus three industrially relevant sources shared with GIFT-Eval (ETT, Electricity, Solar). It exposes target variables, sampling rates, forecast horizons, and episode boundaries, and is released on Hugging Face at cognitedata/IND-eval.

2. A zero-shot comparison of three released TSFMs (Chronos-2, TiRex-2, TimesFM-3) under a common protocol capped at 4,096 context steps, showing that source-specific ranking reversals make benchmark-wide rankings an unreliable basis for deployment model selection.

3. A pretraining-provenance audit of every model–source pair, labeling evidence as documented, possible, or unresolved, plus a core-only aggregate sensitivity check.

4. An exploratory 3W case study linking target-statistic coverage, simulator composition, and downstream forecasting, comparing a generic SARIMA-based simulator (SarSim0), an oscillator-augmented variant (SarSim0-oscillator), and a 50/50 mixture of the two.

Main Findings

• TimesFM-3 leads on aggregate: TimesFM-3 achieves the lowest source-balanced aggregate nMASE (0.713) and nCRPS (0.473), and leads seven of nine sources under each metric. Chronos-2 scores 0.755 nMASE / 0.539 nCRPS; TiRex-2 scores 0.779 nMASE / 0.537 nCRPS.

• But rankings reverse by source: TiRex-2 leads NEATS nMASE; Chronos-2 leads Electricity nMASE and both Electricity and Solar nCRPS. The authors note that Solar nCRPS differs by only 0.004 and they do not treat that ordering as evidence of a meaningful difference.

• Source variation exceeds model variation: Mean pairwise absolute differences across sources are 0.18 for nMASE and 0.19 for nCRPS, while model differences within sources are 0.05 and 0.08. Descriptively, which source you are on matters more than which of these three models you pick.

• Structural coverage differs from GIFT-Eval: IND-Eval uses finer sampling rates — core horizons span five minutes to two days, whereas most GIFT-Eval forecast spans exceed one day. The source-level share of multivariate tasks averages 75% in IND-Eval versus 21% in GIFT-Eval. The benchmarks overlap substantially in generic signal statistics: median ACF-1 is 0.960 for IND-Eval versus 0.895 for GIFT-Eval, median ACF-5 is 0.823 versus 0.555, and median spectral entropy is 0.428 versus 0.559.

• No documented pretraining overlap for the industrial core: No model names any of the six industrial-core sources in its disclosed corpus, and five of the six do not appear in the forecasting benchmarks reviewed. MetroPT-3 is the exception, appearing in TIME after downsampling to five-minute resolution, whereas IND-Eval retains gap-bounded episodes at 10-second cadence. All six core sources are labeled "unresolved" — absence of documented evidence, not evidence that overlap is absent. Electricity and Solar are "documented" for Chronos-2 and TiRex-2; ETT is "possible."

• Core-only aggregate preserves the ordering: Restricting to the six industrial-core sources gives nMASE/nCRPS of 0.674/0.436 for TimesFM-3, 0.732/0.528 for Chronos-2, and 0.757/0.509 for TiRex-2 — the same model ordering on both metrics.

• The generic simulator mismatches target regimes: On 3W severe-slugging (ev3) and flow-instability (ev4) windows, SarSim0 poorly matches the target's smoothness, spectral entropy, and temporal persistence.

• The oscillator-augmented simulator closes part of the gap: Along the leading principal component of ten standardized per-window statistics (34% explained variance), SarSim0-oscillator overlaps substantially more with 3W than SarSim0 does. Under log-band power spectra, a simulated window is the nearest neighbor of 17.0% of 3W windows for SarSim0-oscillator versus 2.5% for SarSim0. The match remains partial: autocorrelation and spectral entropy move closer, while roughness and trend-strength gaps remain.

• Better coverage accompanies lower target error: On ev3, nMASE drops from 0.703 (SarSim0) to 0.668 (SarSim0-oscillator), with nCRPS tied at 0.576. On ev4, nMASE drops from 0.714 to 0.698 and nCRPS from 0.471 to 0.459.

• Specialization costs generality: On the 54 of 97 GIFT-Eval tasks with natively univariate targets, SarSim0-oscillator degrades to nMASE 1.137 with 21/54 tasks beating seasonal naive, versus SarSim0 at 0.794 with 43/54 tasks beating seasonal naive.

• A 50/50 mixture wins on the target domain and recovers most of the loss: The mixture achieves the lowest error of the three recipes on all four 3W event–metric pairs (ev3: nMASE 0.639, nCRPS 0.539; ev4: nMASE 0.677, nCRPS 0.437) while recovering most of the lost GIFT-Eval performance (nMASE 0.826, 42/54 tasks below 1).

• Evaluation is heavily imbalanced in scored pairs: Mesa del Sol contributes 115,515 of 200,757 scored series–window pairs (58%) and Electricity 37,117 (18%). Tablet compression, 3W, NEATS, and Solar contribute 13,785, 11,429, 9,934, and 8,905 pairs (4–7% each); MetroPT-3, ETT, and PRONTO together contribute about 2.0%. Source-balanced aggregation is designed to be insensitive to this imbalance.

Methodology in Plain English

The authors first consolidate nine public industrial datasets into one context–target schema. A task pairs one source-specific series group with one forecast horizon, with no window crossing a source instance boundary. Cadence gaps, batch breaks, and acquisition-order discontinuities define where instances end, and known-future covariates (such as PRONTO's three control channels) and within-instance operating transitions are retained where available. Model inputs are capped at 4,096 context steps, and inputs are not scaled before inference.

They then evaluate three released zero-shot checkpoints under a shared protocol, scoring with nMASE and nCRPS normalized against seasonal naive so that values below one beat the baseline. Scores are aggregated by geometric mean within each source, then across sources with equal weight.

For the case study, they pick two 3W event regimes and diagnose where a generic SARIMA-based simulator (SarSim0) fails to match the target using cheap per-window statistics — roughness, spectral entropy, and temporal persistence. Because the target windows are quasi-periodic and persistent, they add a mechanism designed to reproduce that: SarSim0-generated sequences drive random planar spring networks (15–50 nodes connected by a Delaunay triangulation, two pinned nodes, one or two drivers, two to five readouts), and projected node displacements become synthetic sensor traces. Each of the three training recipes (SarSim0, SarSim0-oscillator, 50/50 mixture) trains the same channel-independent PatchTST-based architecture under the same training budget, one run per recipe, then is evaluated on the IND-Eval 3W tasks and on the natively univariate GIFT-Eval subset.

Why This Matters

Impact on research. The paper argues that aggregate leaderboard rankings can mislead model selection for a specific process or asset, and that evaluation should be interpreted within the task taxonomy represented by the evaluation data. It also makes a methodological point about simulated pretraining: cheap, interpretable target statistics can diagnose simulator coverage gaps and motivate targeted data design when representative real data are scarce. It connects this to intended-purpose testing relevant to high-risk AI systems under the EU AI Act, while explicitly not claiming that IND-Eval establishes regulatory compliance.

Real-world applications.

• Selecting a forecasting model for a specific offshore well, manufacturing line, or energy asset rather than picking from a general leaderboard.
• Deciding whether to fine-tune on scarce local data or instead adapt simulated pretraining data to match the operating regime.
• Deciding how to split a training mixture between generic and domain-specialized simulated data so that a model does not lose generality.
• Determining, per asset, what sampling rate, forecast horizon, and episode boundaries an evaluation should expose — and how those choices differ for batch processes versus continuous telemetry.

Industry relevance. The industrial core covers offshore drilling streams, oil-well flow-assurance events, multiphase-flow transients, microgrid operation, pharmaceutical tablet compression, and metro-train air production. The authors motivate the work by noting that accurate forecasts can reduce operational risk, operating costs, and unplanned downtime, and that aggregate scores can mislead model selection with potential safety consequences.

Future Directions

• Replicate the case study with more training runs and a design holdout. The authors state that selection of ev3/ev4 and the oscillator recipe was informed by target-context statistics and preliminary event-level forecasts, with no independent design holdout, and that they used one training run per recipe.

• Test cross-channel structure rather than channel-wise alignment. The case study models each channel independently, so it tests channel-wise temporal alignment and not the cross-channel structure that IND-Eval also emphasizes.

• Extend statistic-guided simulator design beyond 3W. The workflow is demonstrated on two offshore well-monitoring regimes; whether the same diagnostic-and-augment procedure transfers to NEATS, PRONTO, tablet compression, or MetroPT-3 is an open question.

• Resolve pretraining-overlap uncertainty. No model documents its pretraining corpus in full, so the audit cannot rule out overlap. The authors note that IND-Eval releases raw source series alongside the evaluation schema, enabling independent value-level overlap checks whenever a pretraining corpus is available, and that IND-Eval does not restrict evaluation windows to segments postdating model data cut-offs.

• Characterize the specialization–generality trade-off more precisely. The 50/50 mixture is an untuned probe of the trade-off rather than a calibrated result; how the optimal mixture ratio depends on the target domain and evaluation suite is not established.

Target Audience

Practitioners and researchers working on time series foundation models who need to choose or evaluate models for industrial deployments; benchmark designers interested in how task taxonomy, aggregation rules, and provenance auditing shape conclusions; and applied machine learning engineers in energy, oil and gas, manufacturing, and process industries who are considering simulation-based pretraining when representative historical data are scarce. The paper is also relevant to readers tracking how evaluation practice connects to intended-purpose testing requirements for high-risk AI systems.

Authors’ abstract

We introduce IND-Eval, a public benchmark that organizes industrial time series across domains into source-specific forecasting tasks. Relative to established benchmarks such as GIFT-Eval, it emphasizes finer sampling resolutions and a greater prevalence of multivariate forecasting. Evaluating Chronos-2, TiRex-2, and TimesFM-3, we find that TimesFM-3 achieves the lowest source-balanced aggregate nMASE and nCRPS and leads seven of nine sources under each metric; source-specific exceptions reinforce the need for domain-representative evaluation for model selection. These findings also motivate improving domain alignment during pretraining, where real industrial data are often too scarce to close domain-specific gaps directly; we therefore investigate whether a domain's statistical structure can guide simulator design. In a case study on two offshore well-monitoring regimes, we improve a general simulator's coverage using an oscillator-augmented simulation pipeline. This lowers forecasting error in the target domain but reduces performance on the subset of GIFT-Eval with natively univariate targets. A 50/50 mixture of the general and augmented simulation recipes achieves the lowest error among the three training recipes for all four event-metric pairs in the case study while recovering most of the lost GIFT-Eval performance. Together, these results illustrate how domain-representative evaluation can guide model selection and the composition of simulated data while revealing a trade-off between specialization and generality.