From Retrieval to Recognition:Knowledge-Enhanced Post-training of Time Series Foundation Models for Classification

AI summary

Overview

Research area: Time series classification (TSC) with time series foundation models (TSFMs); specifically, the post-training stage (how a pretrained model is adapted to a downstream classification task).

Technical level: Intermediate. The paper assumes familiarity with time series classification, self-supervised pretraining, supervised fine-tuning, and retrieval-augmented methods, but its core idea (retrieve similar examples instead of relying only on fine-tuned weights) is conceptually simple.

Scope (one sentence): The paper proposes R2R, a plug-and-play post-training framework that retrieves explicit numerical knowledge from a constructed pretraining knowledge base and fuses it with a TSFM's parameter-level embeddings, evaluated on 128 UCR datasets under supervised fine-tuning and 30 additional datasets under zero-shot feature extraction.

What This Paper Is About

Pretrained time series foundation models are usually adapted to a new classification task by fine-tuning their parameters only. In that process, the original pretraining data is discarded, and the task-relevant knowledge stays buried among lots of irrelevant information inside a large parameter space. The paper asks whether, beyond parameter transfer, the model can also draw on explicit data-level knowledge from the pretraining corpus — and answers by building a retrieval module that pulls numerically similar, class-discriminative examples for each input, then fusing those retrieved embeddings with the foundation model's own class token.

Key Contributions

1. A plug-and-play post-training framework (R2R). It enhances pretrained TSFMs with external numerical knowledge from a constructed pretraining knowledge base, enabling instance-level knowledge integration beyond parameter-only transfer, without changing the original pretraining process.
2. A decoupled query encoder with a gradient-free backbone. It extracts and retrieves discriminative numerical patterns using local, global (multi-scale downsampling plus turning-point gating) and positional features, bridging retrieval with downstream classification for multi-domain inputs.
3. A knowledge-enhancement fusion module. Retrieved feature-level knowledge and the TSFM class token are combined through gated residual fusion before a lightweight classifier, so parameter and feature transfer are optimized jointly.
4. Large-scale empirical validation. Experiments on 158 time series datasets (128 UCR supervised, 30 zero-shot) show consistent gains across three integrated TSFMs (NuTime, Mantis, UniShape), plus ablations, knowledge-base size studies, augmentation comparisons and continual fine-tuning analysis.

Main Findings

• Consistent improvement over strong TSFM baselines (128 UCR datasets). Under fully supervised fine-tuning, R2R_UniShape reaches average rank 3.01 with 48 wins, versus 4.47 / 31 for the parameter-only UniShape baseline. R2R_Mantis reaches 6.18 / 29 wins versus 8.07 / 21 for Mantis, and R2R_NuTime reaches 7.90 / 23 wins versus 10.26 / 15 for NuTime. R2R_UniShape significantly outperforms all baselines by the Wilcoxon signed-rank test (p-values all below 0.05).
• Non-deep-learning baselines remain competitive. HIVE-COTE 2.0 (average rank 5.16, 31 wins), MultiRocket-Hydra (6.27, 22 wins), RDST (7.38) and Rocket (7.62) are stronger than most deep models; TS2Vec (12.65), UniTS (16.74), GPT4TS (17.06), TimesNet (17.11), MOMENT (17.58) and PatchTST (18.19) rank much lower.
• Retrieval-augmented baseline is not enough. TS-RAG, the only retrieval-augmented baseline, ranks 9.31 with 11 wins in the supervised setting and 3.97 with 5 wins in the zero-shot setting, even though it retrieves from the same knowledge base as R2R. The authors attribute the gap to R2R's classification-specific retrieval target and fusion rather than to the corpus.
• Best in zero-shot feature extraction (30 datasets). With a parameter-frozen TSFM plus a Random Forest probe (R2R built on UniShape), R2R achieves average rank 1.73 with 21 wins, ahead of TS-RAG (3.97), UniShape (5.20), TSPulse (5.37), NuTime (5.63), RF on raw series (5.80), Mantis (5.80), Chronos (6.33), MOMENT (6.53), GPT4TS (9.00), Moirai (9.00) and UniTS (11.87).
• Both transfer channels matter. Removing feature transfer drops R2R_UniShape from rank 1.48 (86 wins) to 2.45 (48 wins). Every "w/o Parameter transfer" variant is worse still (best is MiniRocket with retrieval at 4.14), showing that pretrained TSFM parameter transfer is the main knowledge source while retrieval adds further gains.
• Retrieval design components contribute. Without the retrieval module, MiniRocket-backed performance falls to 5.09; removing multi-scale global and positional patterns ("w/o ReMGP") falls to 5.59; without retrieval, Rocket scores 5.91, MultiRocket 4.06, and a frozen UniShape encoder 4.52 — so R2R is not merely inheriting strength from the Rocket family.
• 100K class-balanced knowledge base is the best trade-off (18 UCR datasets). 100K balanced bags rank 2.33 at 92.11 s, versus 4.11 for 50K, 3.28 for 200K, 3.11 for 500K, 3.17 for 1M and 4.17 for the full 1.89M (414.80 s). Random sampling of 100K gives 3.17 and unbalanced 100K gives 6.50, so both class diversity and balance matter. Scaling from 50K to 100K adds about 4 seconds per dataset; scaling to 1.89M raises time by about 4.5× but stays within 7 minutes per dataset.
• Retrieval beats standard augmentation. Across 18 UCR datasets, R2R ranks 2.00 while "Raw" (fine-tuning without augmentation) ranks 2.39, and Jitter (5.11), Time warp (5.06), Time flip (5.06), Permutation (5.50), Magnitude warp (4.89), Scaling (4.83) and Window slice (4.33) all perform worse than Raw — i.e., these augmentations cause negative transfer.
• Better class separation. t-SNE on the SonyAIBORobotSurface1 dataset shows classes 1 and 2 partially mixed under parameter fine-tuning, with clearer separation for the retrieval embeddings and the fused embeddings.
• Less forgetting in continual fine-tuning. On four dataset pairs, R2R has higher accuracy before and after continual training and smaller forgetting than parameter-only transfer (e.g., AsphaltRegularityUni_eq → FloodModeling1_disc: forgetting 0.4767 versus 0.1825; GesturePebbleZ1_eq → Colposcopy: 0.4244 versus 0.2384).

Methodology in Plain English

1. Frame the setting. The task is univariate time series classification: an input of length L must be assigned one of C classes. The paper adds a knowledge base built from the pretraining corpora of NuTime, Mantis and UniShape — 1.89M samples in total, reduced to a class-balanced subset of 100K for practical retrieval.
2. Describe each series with three kinds of features (knowledge extraction). A frozen, non-trainable "gradient-free" backbone turns a raw window into a fixed-length vector (local patterns). An envelope feature summarizes amplitude/energy. Multi-scale downsampling creates sparse versions of the series to capture longer-range structure, gated by a turning-point density measure, and a positional feature records where the peak and turning points sit. All of these are concatenated and passed through a small learnable projection to give one compact embedding per series. Because the backbone is frozen, these features are cheap to compute.
3. Retrieve similar examples (knowledge retrieval). Embeddings are L2-normalized and indexed with IVFPQ (Faiss, GPU-accelerated), which partitions the space into Voronoi cells and compresses residuals. A query visits only a small number of cells, giving sublinear search cost and millisecond-level latency for databases up to 10^6 items. The Top-K neighbors by cosine similarity are combined into an augmented embedding using optimal-transport-derived weights (a temperature-scaled softmax over the transport cost), and the query's offset from the weighted neighbor average is concatenated in, so both retrieved knowledge and query-specific information survive.
4. Fuse and fine-tune (knowledge enhancement). The TSFM's class token is combined with the query embedding and the retrieved embedding through gated residual fusion: two sigmoid gates computed from the class token decide how much of each retrieval-derived vector to add, followed by layer normalization. A lightweight classifier produces predictions, trained with cross-entropy. Gradients flow through both the pretrained TSFM and the projection layers, so the model learns to align retrieved numerical knowledge with its own parametric representations.
5. Evaluate under two protocols. Fully supervised fine-tuning on 128 UCR datasets (compared against 18 baselines spanning non-deep-learning, supervised deep learning and TSFM methods, using official splits and released hyperparameters), and zero-shot feature extraction on 30 additional datasets where frozen embeddings train a Random Forest.

Why This Matters

Impact on research. The paper shifts attention from "pretrain a bigger foundation model" to "post-train it better," proposing retrieval of explicit data-level knowledge as a complement to parameter fine-tuning. This is a new post-training axis for time series classification that leaves pretraining untouched, and it contrasts with forecasting-oriented retrieval work (TS-RAG and others) by targeting discriminative patterns in fixed-length windows rather than future-trend extrapolation.

Real-world applications (as motivated in the paper):

• Traffic monitoring, where sensor windows must be classified into congestion or anomaly categories.
• Medical diagnosis, where physiological recordings must be assigned to conditions.
• Action recognition, where motion sequences must be mapped to activity classes.
• Any deployment of an off-the-shelf TSFM onto a new labeled dataset, especially in a zero-shot or probe-only setting where fine-tuning is impractical.

Industry relevance. Because the query encoder's backbone is frozen and retrieval runs at millisecond latency for databases up to 10^6 items (with the full 1.89M-sample base costing at most about 7 minutes per dataset), R2R is a practical add-on rather than a new training pipeline. The demonstrated reduction of catastrophic forgetting under continual fine-tuning is directly relevant to teams that must keep adapting one model across successive customer datasets.

Future Directions

• Beyond univariate series. The paper explicitly limits its setting to univariate TSC; only a preliminary multivariate evaluation on ten UEA datasets is reported in the appendix, so a full multivariate study is open.
• What to put in the knowledge base. Results show that a 100K class-balanced subset beats the full 1.89M samples, and that unbalanced sampling collapses to rank 6.50. Better selection or compression strategies for the knowledge base, and the accuracy/efficiency frontier beyond 1M entries, remain open.
• Better retrieval targets for classification. The paper argues that forecasting and classification need fundamentally different retrieval targets; how to design query encoders that capture even more class-discriminative structure (beyond local, global and positional features) is an active question.
• Combining retrieval with other post-training mechanisms. R2R currently pairs retrieval with supervised fine-tuning and zero-shot probing; whether it extends to other post-training regimes (the appendix reports hyperparameter analysis, suggesting sensitivity remains to be fully mapped) is left for future work.

Target Audience

Researchers and practitioners working on time series classification, time series foundation models, and retrieval-augmented learning who want a post-training technique that complements rather than replaces fine-tuning. It is also useful for applied machine learning engineers who deploy pretrained time series models under limited tuning budgets, and for readers interested in how retrieval ideas from NLP transfer to non-text, numerical domains.

Authors’ abstract

Time series foundation models (TSFMs) have achieved notable progress in classification tasks through large-scale pretraining. However, their post-training practice mainly focuses on implicit parameter fine-tuning, where task-relevant information is entangled with abundant irrelevant knowledge in vast parameter spaces. This may hinder TSFMs from precisely activating useful knowledge for downstream classification. To this end, we propose Retrieval-to-Recognition (R2R), a plug-and-play post-training paradigm that adapts pretrained TSFMs to time series classification. Rather than pretraining a new foundation model, R2R improves how existing TSFMs are post-trained: it introduces a decoupled query encoder to extract discriminative patterns and retrieve instance-level numerical knowledge from a constructed knowledge base. The retrieved explicit knowledge is then fused with the TSFM's implicit parametric embeddings, enabling joint knowledge transfer at both the parameter and feature levels. In this way, R2R allows TSFMs to recall task-relevant patterns learned during pretraining, improving the efficiency of knowledge utilization in downstream classification transfer. Experiments on 158 time series datasets show that R2R consistently improves the classification performance of state-of-the-art TSFMs. Ablation and knowledge-enhanced studies further validate the effectiveness of the proposed framework.