TSAQA: Time Series Analysis Question And Answering Benchmark

AI summary

Overview

Research area: Benchmarking and evaluation of large language models (LLMs) and time series foundation models (TSFMs) on time series analysis question answering (QA).

Technical level: Intermediate. The paper is a benchmark and evaluation study; the model architectures used are off-the-shelf, but the task taxonomy and evaluation protocol assume familiarity with time series concepts (anomaly detection, seasonality, Fourier and wavelet transforms, temporal patches).

Scope: TSAQA is a unified benchmark of 210k samples across 13 domains that recasts 6 time series analysis tasks into three closed-ended question formats (true-or-false, multiple-choice, and a new puzzling format) to measure the temporal analytical capabilities of language-capable models.

What This Paper Is About

Existing time series QA benchmarks cover a limited and fragmented set of analytical capabilities, and questions that require open-ended answers are hard to score objectively, which complicates fair comparison across models. The authors build TSAQA to address this by combining conventional tasks (anomaly detection, classification) with advanced analytical tasks (characterization, comparison, data transformation, temporal relationship) under one standardized evaluation framework. The goal is to measure how well current LLMs and language-capable TSFMs can reason about temporal patterns rather than only produce numeric forecasts.

Key Contributions

1. A new large-scale benchmark. TSAQA contains 210k samples curated from 13 domains, covering 6 tasks and 3 question types, including the newly introduced puzzling (PZ) format.
2. Full construction documentation. The paper details the data collection process, benchmark construction, dataset statistics, and evaluation protocol, with the data released at https://huggingface.co/datasets/TSAQA/TSAQA-Benchmark.
3. Extensive model evaluation. The authors evaluate zero-shot commercial LLMs, medium and small open-source LLMs, instruction-tuned versions of the open-source models, and language-capable TSFMs (ChatTime and ChatTS).
4. In-depth diagnostic analysis. The paper includes accuracy correlate analysis (input length, topics/subtopics, domain), task-specific analysis, case studies, and a human evaluation of the automatically generated labels.

Main Findings

• Zero-shot commercial models lead, but scores are moderate. The best-performing commercial model, Gemini-2.5-Flash, reaches 65.08 average accuracy. GPT-4.1 reaches 62.82, Claude-3.5-Sonnet 61.19, and GPT-4o 60.73.
• Commercial models outperform open-source models zero-shot, and 8B models outperform 1B models. Zero-shot open-source overall scores range from 35.70 (LLaMA3.2-1B) to 51.04 (Qwen3-8B).
• Instruction tuning substantially improves open-source models. LLaMA3.1-8B is the best-performing model overall after tuning at 85.26, followed by Qwen3-8B at 84.29.
• A tuned small model beats a top commercial model. Gemma3-1B reaches 69.70 after instruction tuning, surpassing Gemini-2.5-Flash's zero-shot 65.08.
• Language-capable TSFMs trail general LLMs. ChatTime scores 55.19 overall and ChatTS 30.10, which the authors present as evidence that TSAQA extends beyond general-purpose LLM evaluation.
• Puzzling questions are the hardest format. The best PZ score is only 67.68, and PZ performance is substantially lower than TF and MC in both zero-shot and tuned settings. Across question types, open-source models perform best on TF, worse on MC, and poorest on PZ.
• Input length affects accuracy in opposite directions by task. For all five tasks except Temporal Relationship, accuracy declines as input length increases. Temporal Relationship improves with longer inputs, and the PZ question exhibits the opposite trend — which the authors interpret as evidence that models use global context for chronological ordering.
• A Smoothness Bias appears in PZ predictions. Models tend to repair legitimate discontinuities by predicting transitions smoother than the ground truth, especially in the Web and Sales domains.
• The Comparison task is domain invariant. No significant performance difference was found between same-domain and different-domain comparisons, in either MC or TF questions.
• Topic complexity does not directly drive accuracy. Varying the number of topics and subtopics per question did not directly impact accuracy, which the authors read as the benchmark being largely unbiased. Questions involving seasonality, autocorrelation, dispersion, and noise were harder.
• Domain difficulty varies. Synthetic, IT, Robotics, and Web domains were most challenging zero-shot; Sales and Web remained the most difficult after instruction tuning.
• Instruction-tuning largely closes the gap on conventional tasks. LLaMA-3.1-8B reaches 91.02 and 91.27 on anomaly detection and classification after tuning, while zero-shot performance on those tasks was poor.
• Human evaluation supports label quality. Six Ph.D.-level experts annotated 600 questions (300 each for characterization and comparison). Uncertainty rates were low (5% characterization, 7% comparison), and benchmark answers aligned with human judgments in 91.2% of characterization and 87.4% of comparison cases.
• Response-format biases exist. For incorrectly answered PZ questions, Qwen3-8B tends to output C more frequently while LLaMA3.1-8B tends to output A.

Methodology in Plain English

The authors collected real-world time series from public sources used by time series foundation model benchmarks (Lotsa, Time-300B, UTSD), keeping only sequences at least 1k long, dropping sequences with a missing rate above 1% or an outlier rate above 5% (points beyond three times the interquartile range). Additional data came from anomaly detection benchmarks (ECG, SMD, MGAB, Genesis, GHL, Occupancy) and from the univariate UCR Archive, where they selected datasets with at most four classes and sequence lengths under 400.

Each task gets an equal allocation of 30k samples, except temporal relationship, which gets 60k because puzzling questions are very challenging. Samples outside classification and anomaly detection are drawn using Hierarchical Random Sampling to balance domains, datasets, and sequences. Each sample has a random length within [32, 256], fixed decimal levels, and is z-score normalized. Data is split 70% training, 10% validation, 20% testing.

To unify the tasks, every instance is framed as a model function A = f(X, C, Q), where X is the time series, C is natural-language context, Q is the question, and A is the answer. True-or-false questions ask whether a claim holds; multiple-choice questions ask for the correct claim; puzzling questions give the first patch of a series plus shuffled remaining patches and ask the model to restore chronological order.

Construction differs by task. Data transformation uses Fourier transform, wavelet transform, or first-order differencing, computed with professional libraries. Temporal relationship tests structural continuity, chronological reasoning, and contextual discrimination, with distractors sampled from different sequences. Characterization QA pairs are generated by GPT-4o, self-checked with a confidence score, then validated by GPT-4.1, Gemini-2.5-Flash, and Claude-3.5-Sonnet to produce consensus answers. Comparison builds 10 comparison samples per anchor: one from the same sequence, two from other sequences in the same dataset, three from other datasets in the same domain, and four from other domains. Anomaly detection downsamples normal samples to a 1:1 ratio with anomalous ones, and classification converts numeric labels into textual choices.

Evaluation is accuracy for TF and MC. PZ is scored by the proportion of correctly matched positions: given ground truth A, B, C, D and prediction B, A, C, D, only the last two match, yielding 50%. Instruction tuning used LoRA with rank 16, a fixed learning rate of 10⁻⁵ with a cosine schedule, and 2 epochs on a single A100 GPU.

Why This Matters

The paper argues that time series research has traditionally been narrow — forecasting, anomaly detection, imputation, and classification — while real temporal analysis involves contextual understanding, comparison, and structural reasoning. TSAQA provides a standardized, closed-ended framework that makes such analysis objectively scorable, and it shows that even strong commercial LLMs leave substantial headroom, especially on the puzzling and advanced tasks. It also positions the benchmark as a potential training resource, not just an evaluation tool.

Real-world applications grounded in the paper's domains:

• Finance: the benchmark draws on exchange rate and macro-economic series (ExchangeRate, FRED-MD), relevant to detecting anomalies and reasoning about market movement.
• Healthcare: data from ICU and physiological recordings (BIDMC32HR, PigArtPressure, US Births) support questions about vital-sign patterns.
• Energy and environmental science: electricity demand, load forecasting (Australian Electricity Demand, GEF12, BDG-2), sunspot activity, river flow, and subseasonal precipitation series.
• Transportation and retail: pedestrian counts, traffic sensor data (PEMS03), Uber pick-ups, hierarchical sales, and the M5 Walmart dataset.

Industry relevance: practitioners evaluating whether a language model can be used for time series monitoring and diagnosis get a standardized scorecard; the finding that instruction-tuned open-source models can match or beat commercial models is directly relevant to cost and deployment decisions; and the TSFM results (ChatTime, ChatTS) show that domain-specific time series models are not automatically better at analytical QA than general LLMs.

Future Directions

• Expand beyond closed-ended questions. The authors suggest extensions toward multivariate settings, multimodal inputs, open-ended analysis, and agentic systems.
• Cover more realistic temporal structure. Irregular sampling, mixed-frequency series, and exogenous-aware scenarios are named as gaps, since the current benchmark only partially reflects them.
• Address the PZ limitations. The puzzling format may penalize models with locally smooth priors and carries higher computational cost; complementary formats are proposed to disentangle local continuity, long-range consistency, and domain-specific volatility.
• Move from a static benchmark to a dynamic one. Real deployments face evolving distributions and emerging domains, so evaluating adaptation and robustness under distribution shift is identified as an important next step. The paper's limitations section also notes the taxonomy focuses on capabilities expressible through structured questions, leaving richer forms of temporal reasoning for future task design.

Target Audience

This paper is most useful for researchers studying the temporal reasoning capabilities of language models, practitioners looking for a reference benchmark for time series QA, and the broader time series question answering community developing new models, training datasets, or evaluation methods. It is also relevant to teams considering instruction-tuned open-source models for time series analytical tasks, since it compares them directly against commercial LLMs and language-capable TSFMs.

Authors’ abstract

Time series data are integral to critical applications across domains such as finance, healthcare, transportation, and environmental science. While recent work has begun to explore multi-task time series question answering (QA), current benchmarks remain limited to forecasting and anomaly detection tasks. We introduce TSAQA, a novel unified benchmark designed to broaden task coverage and evaluate diverse temporal analysis capabilities. TSAQA integrates six diverse tasks under a single framework ranging from conventional analysis, including anomaly detection and classification, to advanced analysis, such as characterization, comparison, data transformation, and temporal relationship analysis. Spanning 210k samples across 13 domains, the dataset employs diverse formats, including true-or-false (TF), multiple-choice (MC), and a novel puzzling (PZ), to comprehensively assess time series analysis. Zero-shot evaluation demonstrates that these tasks are challenging for current Large Language Models (LLMs): the best-performing commercial LLM, Gemini-2.5-Flash, achieves an average score of only 65.08. Although instruction tuning boosts open-source performance: the best-performing open-source model, LLaMA-3.1-8B, shows significant room for improvement, highlighting the complexity of temporal analysis for LLMs.