DISC: Domain-Integrated Similarity Cost for Time-Series Dataset

AI summary

Overview

Research area: Time-series machine learning, specifically dataset-level similarity measurement for transfer learning and time-series foundation models.

Technical level: Intermediate. The paper assumes familiarity with forecasting, fine-tuning/transfer-learning protocols, and standard distance metrics, but its core idea (learning weights over similarity metrics) is conceptually simple.

Scope: The paper proposes DISC (Domain-Integrated Similarity Cost), a learnable framework that combines multiple time-series dataset similarity metrics and calibrates their weights against observed downstream forecasting errors, evaluated on 46 GluonTS datasets with two foundation models under out-of-distribution and transfer-learning protocols.

What This Paper Is About

Before adapting a pretrained time-series model to a new target dataset, practitioners want to know whether a candidate source dataset will actually help, without running every possible source-target combination. Many similarity metrics exist (distribution-based, transport-based, shape-based), but they capture different relationships and no single metric consistently predicts downstream forecasting error. The paper's goal is to learn, from a small amount of observed transfer data, how to weight and combine several similarity metrics into one score whose magnitude tracks downstream error better than any single metric.

Key Contributions

1. A learnable aggregation framework (DISC) that integrates complementary dataset similarity metrics so the resulting score better separates source-target pairs whose downstream transfer behavior differs.
2. Task adaptability: the aggregation parameters are learned per downstream setting, so metric composition and weighting shift with the model and protocol rather than being fixed once globally.
3. Interpretability: the learned base and prior metric weights show which dataset relationships matter for a given model and task, and the paper reports these weights for all four tasks.
4. An empirical correlation-alignment study across four forecasting settings (two foundation models × OOD and transfer-learning protocols) showing single metrics are task-dependent and that learned integration exceeds both the best average single metric and a per-task oracle single metric.

Main Findings

• Single metrics are inconsistent across tasks. The strongest single metric differs by setting: MAD for OOD-Time-MoE (0.580), DBA-DTW for OOD-Lag-Llama (0.618), PCA-ED for TL-Time-MoE (0.537), and MMD for TL-Lag-Llama (0.435). The best average single metric is PCA-ED at 0.403.
• DISC beats the per-task oracle single metric. A per-task oracle (best single metric chosen separately for each task) reaches 0.543; DISC reaches 0.598 on the same test split.
• Gains over the best single metric in every task: OOD-Time-MoE 0.580 → 0.601 (+3.55%), OOD-Lag-Llama 0.618 → 0.697 (+12.84%), TL-Time-MoE 0.537 → 0.617 (+14.75%), TL-Lag-Llama 0.435 → 0.479 (+10.15%), for an average of 0.543 → 0.598 (+10.29%). The paper describes the OOD-Time-MoE margin as modest and interprets it as a small directional gain, not a large effect.
• Optimal Transport is a weak reference. OT scores average 0.103, with the lowest per-task value −0.007 on OOD-Time-MoE; DISC's average gain over OT is +0.496.
• Learned mixtures are task-specific, not metric-specific. For example, MMD is used in the TL-Lag-Llama base (weight .833) but not in the OOD-Lag-Llama base; PCA-LCSS appears as both base (.383) and prior (.935) for OOD-Lag-Llama, while Time-MoE tasks use PCA-ED as a prior (.717 for OOD, .719 for TL).
• Gate modulation varies by task. The mean positive gate factor on test pairs ranges from 1.17× for TL-Time-MoE (range [1.00, 3.81]) to 2.89× for TL-Lag-Llama (range [1.00, 3.95]); OOD-Time-MoE is 2.26× [1.00, 3.35] and OOD-Lag-Llama 1.21× [1.00, 3.79].
• Qualitative source-wise alignment. Figure 3 shows four fixed sources (m3_monthly, electricity_weekly, wiki2000_nips, fred_md), one per task, where larger DISC scores generally correspond to larger downstream forecasting errors.
• Learned aggregation outperforms manual search. The paper states the answer to RQ3 (can broad non-learned parameter search replace learned aggregation?) is no; the specific Table VI numerical values are cut off in the available content and are therefore not reported here.
• Metric disagreement is visible at the pair level. In Figure 1, WSD, MAD, and DBA+LCSS all rank the semantically mismatched group covid_deaths, m1_quarterly, and tourism_quarterly as close, while MMD ranks them as distant.

Methodology in Plain English

The researchers start by treating "how similar are two datasets?" as a directed question: a model trained on dataset X and tested on Y need not behave like the reverse. They assemble a universe of 46 GluonTS datasets, represent each by 100 sampled windows of length 16, and compute ten similarity configurations: four dataset-level metrics (Wasserstein distance, MMD, Optimal Transport, Match-and-Deform) and six combinations of two dataset summarizers (DBA and PCA) with three sequence distances (DTW, Euclidean distance, LCSS).

They then run the costly part once: fine-tuning Time-MoE and Lag-Llama on each of the 46 source datasets, producing 2,070 directed source-target pairs per model per protocol, plus a transfer-learning variant where each source model is additionally adapted on each target. The resulting error matrices become calibration signals.

DISC itself is a small parametric model with two branches. A base mixture takes each metric's value, standardized within the source row, and takes a learned softmax-weighted sum — this is the primary score. A prior mixture takes each metric's percentile rank within the same source row (a scale-free relative position) and takes a separate learned softmax-weighted sum. A sigmoid gate built from the prior value multiplies the base score: the multiplier starts at one and shifts the score's magnitude according to the rank evidence. The gate has three learnable scalars controlling modulation strength, threshold, and sharpness, with sharpness bounded to [1, 55].

Training maximizes a blend of source-wise Pearson correlation (average within-source-row correlation between DISC score and error, primary) and global Pearson correlation, weighted λ = 0.10, plus entropy regularization (η = 0.006) to keep weights sparse and an inner-product penalty (μ = 0.02) to discourage the base and prior from picking identical metrics. Optimization uses Adam with learning rate 0.03, 500 epochs, patience 100, 8 random restarts, and softmax temperature 0.55. Parameters are fit on 24 training datasets, checkpointed on 8 validation datasets, and reported on 14 held-out test datasets. After calibration, scoring a new pair requires only its metric values — no downstream error is needed.

Why This Matters

Impact on research. The paper reframes time-series dataset similarity as a correlation-alignment problem rather than a fixed metric choice, and provides evidence that metric suitability is task- and model-dependent. It also supplies an interpretable readout of which metric families matter for which settings, which is useful for anyone curating benchmarks or analyzing cross-domain generalization.

Real-world applications (domains named in the paper):

• Energy and electricity forecasting, where the paper's own analysis clusters electricity, electricity_hourly, and electricity_nips as a tight family.
• Transportation and mobility, where uber_tlc_daily, uber_tlc_hourly, and vehicle_trips_wm are identified as a related group.
• Healthcare, where covid_deaths is the example of a semantically mismatched dataset that some metrics wrongly rank as close.
• Finance (fred_md) and web traffic (wiki2000_nips), both used as illustrative fixed sources in the source-wise analysis.
• Also named: weather systems and general time-series repositories.

Industry relevance. Because running 2,070 fine-tuning jobs per model is expensive, a cheap pre-adaptation screening score has practical value for prioritizing which public or internal datasets to try before committing compute. The paper is explicit that this is a correlation-alignment analysis and not a direct source-selection benchmark, so it is an argument for the approach rather than a validated retrieval tool.

Future Directions

• Repeated-split uncertainty analysis. The authors state their evidence is "source-row transfer alignment under the shared split, rather than a repeated-split uncertainty analysis over all possible dataset partitions" — testing over multiple partitions is the obvious next step.
• Turning correlation alignment into actual source selection. The current evaluation is source-conditioned correlation, not target-wise top-k source retrieval; closing that gap is an open problem.
• Separating target difficulty from pair-specific transfer compatibility. The paper notes that downstream error E(X,Y) also reflects the intrinsic difficulty of target Y, and does not claim DISC decomposes this.
• Expanding the metric pool and task families. DISC is evaluated on ten similarity configurations and four forecasting settings (two models, two protocols); whether the same learned-integration pattern holds for classification, anomaly detection, or other foundation models is untested here.

Target Audience

Researchers and engineers working on time-series foundation models, transfer learning, and domain adaptation who need to decide which datasets to reuse; benchmark curators and dataset-repository maintainers interested in principled dataset comparison; and methodologists studying how well similarity measures align with downstream utility. Readers should be comfortable with Pearson correlation, softmax weighting, and standard fine-tuning protocols, but the aggregation mechanism itself requires no specialized mathematics.

Authors’ abstract

The growth of time-series datasets and foundation models has made dataset comparison a central part of time-series workflows. Assessing source-target transfer before model adaptation requires similarity estimates that can summarize transfer-related dataset relationships without exhaustive source-target evaluation over every candidate pair. Many metrics have been proposed for measuring similarity between time-series datasets, but they capture different dataset relationships and vary in how well they align with downstream utility across tasks. We evaluate this gap by measuring the correlation between source-target similarity scores and downstream forecasting errors in out-of-distribution and transfer-learning settings. We propose Domain-Integrated Similarity Cost (DISC), a learnable aggregation framework that uses calibration transfer observations to select, weight, and combine similarity metrics for different downstream tasks. By integrating complementary metric values, DISC better distinguishes source-target dataset differences that matter for downstream behavior. After calibration, DISC scores source-target pairs using only their similarity-metric values and yields higher alignment than single-metric references on the reported split. These results show that time-series dataset similarity is task-dependent, and that learned metric integration provides a stronger proxy for downstream transfer behavior than manually selecting one similarity metric.