Synthetic Series-Symbol Data Generation for Time Series Foundation Models

AI summary

Overview

Research area: Time series analysis (TSA) foundation models, synthetic data generation, cross-modal (series–symbol) representation learning.

Technical level: Advanced. The paper assumes familiarity with Transformer pre-training objectives (masked modeling, contrastive learning, momentum distillation), symbolic dynamics, and standard TSA benchmark protocols.

Scope: The paper proposes a mechanism for generating unlimited synthetic time series paired with symbolic expressions (the "series-symbol" or S² dataset) and a pre-trained foundation model, SymTime, that learns from these pairs to serve five downstream TSA tasks.

What This Paper Is About

Foundation models for time series analysis are held back by training data scarcity and by imbalanced dataset composition, since time series corpora remain much smaller than those in computer vision or natural language processing, and some domains (the paper names finance and healthcare) are especially sparse. Rather than collecting more real data, the authors use Takens' theorem and symbolic dynamics to argue that any time series is an external representation of a complex dynamical system, and that complex systems can be abstracted into symbolic expressions. Their goal is therefore to generate arbitrary quantities of time series together with the symbolic expressions that produced them, and to pre-train a model that exploits the correlation between the two modalities.

Key Contributions

1. A series-symbol (S²) data generation mechanism. The authors construct random multivariate symbolic expressions built from binary operators (+, −, ×), unary operators (inv, abs, pow2, pow3, sqrt, sin, cos, tan, arctan, log, exp), random constants, and affine transformations, then forward-propagate sampled input series through these expressions to obtain output series, producing a dataset of 40M series–symbol pairs (50B total length).

2. SymTime, a dual-modality pre-trained foundation model. SymTime combines a 6-layer Transformer time series encoder trained with masked time series modeling (MTM), a 6-layer DistilBERT symbol encoder trained with masked language modeling (MLM), and momentum encoders, joined by series–symbol contrastive learning and momentum distillation.

3. Demonstration of scaling benefit. The authors divide the S² dataset into {0B, 1B, 10B, 25B, 50B} timestamp-based subsets and show that downstream performance improves monotonically as pre-training data grows, with 0B denoting direct fine-tuning without pre-training.

4. Generality across five TSA tasks. Unlike many existing foundation models limited to zero-shot forecasting, SymTime is evaluated on long-term forecasting, short-term forecasting, classification, imputation and anomaly detection using the TimesNet benchmark, reporting performance "rivaling foundation models pre-trained on real-world datasets."

Main Findings

• Representation coverage (RQ1): Radviz visualization of 256-length segments compared S² against the Monash dataset (sampling 200K segments per domain across weather, traffic, electricity, tourism, medicine and energy). S² closely matched Monash across stationarity (ADF test), forecastability, frequency domain (FFT mean), complexity, seasonality and trend (Mann-Kendall test). Expanding from 100K to 200K single-channel samples broadened coverage, surpassing Monash in some regions, and mixing single- and dual-channel samples (100K) increased diversity further.

• Competitive downstream performance (RQ2): Across five tasks, SymTime showed competitive results against Transformer-based, LLM-based, CNN-based, MLP-based and pre-trained baselines. On long-term forecasting with a 96 look-back window and results averaged over forecasting lengths {96, 192, 336, 720}, SymTime reached an average MSE of 0.336 and MAE of 0.349, compared with 0.355/0.358 for Peri-midFormer, 0.346/0.361 for Moirai, 0.372/0.378 for Timer, 0.341/0.357 for Time-LLM, 0.344/0.364 for TSLANet, 0.348/0.366 for S²IP-LLM, 0.345/0.359 for GPT4TS, and 0.355/0.357 for TimeMixer.

• Efficiency: On the ETTh1 long-term forecasting task with a 720 forecasting length and 96 look-back window, SymTime achieved better performance with a smaller parameter count and smaller GPU memory footprint (forward and backward propagation at batch size 1) than existing foundation models.

• Scaling with pre-training data (RQ3): As pre-training data grew from 0B to 50B, performance improved progressively across tasks. On long-term forecasting averages, MSE and MAE moved from 0.358/0.366 at 0B to 0.336/0.349 at 50B. Pre-trained SymTime consistently outperformed its non-pre-trained counterpart. Short-term forecasting (measured by OWA on the M4 benchmark) and imputation showed corresponding improvements, and classification and anomaly detection were compared using critical difference diagrams.

• Ablation on pre-training objectives (RQ4): Control groups were Freeze, Real-Data, w/o Pre-train, w/o MTM, w/o MLM, w/o T2S, w/o S2T, w/o Symbol, and w/o Distill, evaluated on ETTh1 and ETTh2 long-term forecasting with MSE, using multiple fine-tuning trials with varied random seeds. Full pre-training with the combined objective significantly improved forecasting. Notably, both pre-training on real data of the same scale through MTM alone (Real-Data) and removing the symbol encoder component on synthetic S² data (w/o Symbol) degraded performance, indicating that symbolic semantics and series–symbol contrastive learning improve the temporal encoder. Results were replicated on short-term forecasting.

• Backbone robustness: Ablation replacing SymTime's backbone showed no significant impact on downstream tasks; the authors attribute performance gains mainly to the dual-modal time series and symbolic expression pre-training paradigm rather than to the specific backbone.

• Learned representation structure (RQ5): Using 20K single-channel series–symbol pairs with t-SNE (50 perplexity), the untrained time series encoder produced entangled clusters (aside from outliers such as inv and exp), whereas the pre-trained encoder formed clear operator-specific clusters (trigonometric: sin, cos; polynomial: pow2, pow3, sqrt). The symbol encoder likewise evolved from poorly separated to distinct clusters for expressions of the same type, and paired time series and expressions showed similar clustering. The time series encoder also demonstrated zero-shot imputation capability on both the S² dataset and real-world data.

Methodology in Plain English

The authors start from the idea that a time series is a window onto some underlying dynamical system, and that such systems can be written as mathematical expressions. They therefore build an automatic expression generator: a random binary tree is constructed over input variables using random operators, random constants and variables are inserted at the leaves, unary operators are inserted at random positions, and each variable and unary operator is affine-transformed (replaced by a·x + b or a·u + b) to diversify the expressions. Input dimensions are exhaustively traversed over M ∈ [1, M_max] with M_max = 6 and N ∈ [1, N_max] with N_max = 12 rather than randomly sampled, so that multivariate cases are fully covered.

Input series X ∈ R^(M×L) are then sampled either from a mixture of distributions (with k ~ U(1, k_max) components, weights w_j ~ U(0,1) normalized to sum to 1, and Gaussian or uniform components) with probability P ≤ 0.5, or from an ARMA(p,q) process otherwise, with stationarity enforced. Series are normalized per channel, passed through the expression to yield Y = f(X) ∈ R^(N×L), and discarded if X falls outside f's domain or if |Y| > 10^4. The paper reports that the time complexity of generating S² data scales linearly with series length L, approximating O(L).

SymTime is then pre-trained on these pairs. The time series branch patches the series and uses a 6-layer Transformer with random masking and reconstruction (MTM). The symbol branch treats expressions as natural language and uses a 6-layer DistilBERT with masked language modeling (MLM). A contrastive objective pulls matching series–symbol pairs together across modalities using momentum encoders and a learnable temperature, and a momentum-distillation term aligns the online encoder's predictions with soft pseudo-targets from the momentum encoders. For downstream use, the pre-trained time series encoder is fine-tuned: classification patches, encodes and applies a linear head, while reconstruction tasks (forecasting, imputation, anomaly detection) decompose each series into trend and periodic components, regress the trend directly, and encode the periodic part before recombining.

Why This Matters

Impact on research: The work offers a route around the data bottleneck that the authors argue limits time series foundation models. Because the S² dataset can in principle be scaled without bound, and because the paper shows performance correlating with pre-training volume, it suggests a synthetic, theoretically motivated alternative to aggregating scarce and privacy-constrained real-world time series. It also repositions symbolic expressions as a pre-training signal rather than only a modeling target, and demonstrates the value of moving beyond forecasting-only zero-shot capability to a model fine-tuned across five task families.

Real-world applications:

• Forecasting in finance, energy demand and inventory management, domains the paper explicitly names as beneficiaries of forecasting.
• Healthcare monitoring, since the authors identify healthcare as one of the scarce and imbalanced data types and cite COMET's medical time series foundation model as related work.
• Imputation for sensor or meter data with missing values, supported by the paper's zero-shot imputation demonstration.
• Anomaly detection in operational time series such as traffic and electricity, both included in the evaluation.

Industry relevance: The complexity analysis positions SymTime as a smaller and more memory-efficient alternative to existing foundation models, which matters for deployment. The work was supported by Huawei's Al Hundred Schools Program and carried out on the Huawei Ascend AI technology stack, and the code is publicly released, lowering the barrier to reproduction and industrial adoption.

Future Directions

• Scaling up to larger models trained on synthetic datasets, which the conclusion explicitly names as the next aim.
• Establishing how far the S² dataset can be pushed: the paper claims theoretical unboundedness but evaluates only up to the 50B-scale point, leaving the shape of the scaling curve beyond that unexamined.
• Extending the evaluation to real-world domains the authors flag as scarce, such as finance and healthcare, rather than the benchmark suites used here.
• Determining which aspects of the generation pipeline drive downstream gains, since the backbone ablation showed the architecture is not the source of improvement and the operator-choice analysis is deferred to an appendix.

Target Audience

This paper is most valuable to researchers and engineers working on time series foundation models, self-supervised pre-training, or synthetic data generation, and to practitioners who need general-purpose time series backbones spanning forecasting, classification, imputation and anomaly detection. Readers interested in the intersection of dynamical systems theory and machine learning will also find the Takens' theorem and symbolic dynamics framing useful, though familiarity with Transformer pre-training objectives is assumed throughout.

Authors’ abstract

Foundation models for time series analysis (TSA) have attracted significant attention. However, challenges such as training data scarcity and imbalance continue to hinder their development. Inspired by complex dynamic system theories, we design a series-symbol data generation mechanism, enabling the unrestricted creation of high-quality time series data paired with corresponding symbolic expressions. To leverage series-symbol data pairs with strong correlations, we develop SymTime, a pre-trained foundation model for enhancing time series representation using symbolic information. SymTime demonstrates competitive performance across five major TSA tasks when fine-tunes with downstream tasks, rivaling foundation models pre-trained on real-world datasets. This approach underscores the potential of series-symbol data generation and pretraining mechanisms in overcoming data scarcity and enhancing task performance. The code is available at https://github.com/wwhenxuan/SymTime.