TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models

AI summary

Overview

Research area: Benchmarking and evaluation of generalist AI models (LLMs, VLMs, and time-series LLMs) on temporal reasoning tasks.

Technical level: Intermediate — familiarity with large language models, multimodal models, and basic time-series concepts is helpful, but the paper's framing and findings are accessible.

Scope: TSRBench is a multi-domain, multi-modal benchmark of 4,125 problems spanning 14 domains and 15 tasks, designed to stress-test the full spectrum of time-series reasoning in generalist models.

What This Paper Is About

Existing benchmarks for generalist models rarely test whether models can actually reason about time series data. When they do, they typically reduce time series to isolated numeric sequences or focus on narrow tasks like forecasting or surface-level pattern recognition, stripping away the causal and semantic context that real problems require. This paper builds a comprehensive benchmark that evaluates LLMs, VLMs, and specialized time-series LLMs across four core dimensions — Perception, Reasoning, Prediction, and Decision-Making — to expose where current models break down.

Key Contributions

1. A new multi-dimensional benchmark (TSRBench) with 4,125 problems drawn from 14 domains and organized into 4 capability dimensions covering 15 distinct tasks — including novel categories like etiological reasoning, abductive reasoning, and quantitative decision-making that earlier benchmarks mostly ignored.

2. Multi-modal evaluation support across four input representations: textual numeric sequences, rendered line-chart visualizations, combined text+visual inputs, and native time-series embeddings for specialized TSLLMs.

3. A large-scale empirical study of over 30 models — 6 proprietary (GPT-5, Gemini-2.5-Flash, Claude-4.5-Haiku, o4-mini, GPT-5-mini, DeepSeek-V3.2), 13 open-source LLMs, 13 open-source VLMs, and 4 open-source TSLLMs — using a unified evaluation setup.

4. Diagnostic analyses of scaling behavior, task correlations, modality complementarity, tool augmentation, visualization resolution, and inference-time compute, yielding actionable insights for future model and benchmark design.

Main Findings

• Perception is strong, reasoning is weak. Models handle pattern recognition, noise characterization, and anomaly detection reasonably well, but collapse on complex reasoning (e.g., temporal relation reasoning), forecasting, and decision-making. GPT-5 with combined text+visual input achieves the best overall accuracy at only 55.6%.

• Scaling laws hold — except for prediction. Spearman correlations between model size and performance are strongly positive for Perception (ρ=0.89 for LLMs), Reasoning (ρ=0.98), and Decision-Making (ρ=0.74), but turn negative for Prediction (ρ=-0.24), meaning bigger models do not forecast better.

• Reasoning and prediction are decoupled. Tasks across Perception, Reasoning, and Decision-Making correlate strongly with each other, but show weak correlation with Prediction — semantic understanding does not translate into accurate numerical or event forecasting.

• Text and vision are complementary but poorly fused. Textual and visual time series solve largely different subsets of problems, yet when models are given both (T+V), accuracy barely improves over single-modality input. The T+V answers mostly overlap with those already solved by one modality alone, revealing a fusion failure.

• Reasoning effort helps, but selectively. Enabling high-reasoning mode boosts o4-mini by +4.3% and GPT-5-mini by +7.2% overall. Perception tasks remain robust when reasoning is disabled, while Reasoning, Prediction, and Decision-Making degrade sharply.

• Tool augmentation gives modest returns. Appending explicit time-series statistics (trend, seasonality, change points, outliers, etc.) to prompts yields only small gains, suggesting that feature injection is not a substitute for genuine temporal understanding.

• Mid-range visualization resolution works best. 100 PPI outperforms both lower (10, 50 PPI) and higher (200, 400 PPI) resolutions; text input can partially compensate for low-resolution visualizations.

• Error analysis points to perception and reasoning. Sampling 150 failure cases across three models shows Reasoning and Perception errors dominate, while Question Understanding and Domain Knowledge errors are marginal — current failures are not knowledge gaps.

• Two regimes of task difficulty. High-variance tasks (e.g., Abductive Reasoning, Event Prediction) show that weaker models could be improved via distillation from stronger ones; low-accuracy, low-variance tasks (e.g., Quantitative Decision-Making, Time Series Forecasting) reflect a shared blind spot requiring better training data.

• Specialized TSLLMs underperform. OpenTSLM variants, ChatTS-14B, and TS-Reasoner-7B score between 30.7% and 36.4% overall — competitive with same-size generalist models but far behind leading proprietary systems.

Methodology in Plain English

The authors curated problems from 14 real-world domains (finance, healthcare, industry, energy, and others) and organized them into a four-level capability taxonomy. Each problem was manually designed or synthesized to test a specific skill, and answers were produced either through high-fidelity simulations (where the ground truth is mathematically unambiguous) or direct extraction from real data and context. Strict data-collection principles ensured that the textual context genuinely complements the time series, that domains are diverse enough to prevent overfitting, and that ground truth is verifiable.

For evaluation, the same underlying time series was fed to models in different forms: as a plain numeric text sequence for LLMs, as a rendered line chart for VLMs, as both combined for proprietary multimodal models, and as raw embeddings for specialized time-series LLMs. All inputs were multiple-choice questions, scored by accuracy. The team then ran correlation analyses across model scales and task dimensions, ablated visualization resolution, toggled reasoning modes, and manually categorized 150 failure cases.

Why This Matters

Research impact: TSRBench fills a clear gap in evaluation infrastructure. It shifts the field away from treating time series as isolated numerical sequences and toward a framework that rewards genuine temporal reasoning. The finding that scaling laws break for forecasting is a significant signal that current architectures may be fundamentally ill-suited to prediction, pushing research toward new pre-training paradigms rather than just larger models.

Real-world applications:

• Healthcare: Clinical monitoring and patient trajectory prediction depend on reasoning over vital-sign time series combined with textual notes.
• Finance: Algorithmic trading and risk management require both causal reasoning about market events and precise numerical forecasting.
• Energy and infrastructure: Grid management and demand forecasting combine sensor data with contextual news and weather.
• Disaster response: Early warning systems rely on interpreting complex sensor streams with domain knowledge and making time-critical decisions.

Industry relevance: Any organization deploying generalist models on time-series data — from cloud providers to industrial automation vendors — needs a standard way to compare model capabilities. TSRBench provides that yardstick and identifies concrete gaps (multimodal fusion, numerical prediction, quantitative decision-making) that no current model solves well.

Future Directions

1. Multi-view fusion. Develop alignment techniques that genuinely combine high-resolution visual patterns with semantic textual context, rather than concatenating the two modalities and hoping the model figures it out.

2. Time-series foundation models. Given the shared failure on quantitative prediction, pre-train models directly on massive, diverse time-series corpora to bridge the gap between semantic reasoning and precise numerical extrapolation.

3. Multi-agent systems. Decompose complex temporal problems into specialized sub-agents (pattern recognition, logical deduction, domain retrieval) that collaborate and verify each other's outputs.

4. Adaptive test-time scaling. Explore reasoning strategies that allocate more inference compute to hard sub-task categories (reasoning, prediction, decision-making) while keeping perception tasks on fast heuristic paths.

Target Audience

AI researchers working on evaluation benchmarks, multimodal and time-series models, and reasoning capabilities of large models; practitioners in finance, healthcare, energy, and industrial systems who deploy generalist models on temporal data; and model developers looking for concrete failure modes and design guidance to improve time-series understanding, forecasting, and decision-making in future architectures.

Authors’ abstract

Time series are ubiquitous in real-world scenarios and crucial for applications ranging from energy management to traffic control. Consequently, the ability to reason over time series is a fundamental skill for generalist models to solve complex problems. However, current benchmarks for generalist models largely overlook this dimension. To bridge this gap, we introduce TSRBench, a comprehensive multi-modal benchmark designed to stress-test the full spectrum of time series reasoning capabilities. TSRBench features: i) a diverse set of 4125 problems from 14 domains, and is categorized into 4 major dimensions: Perception, Reasoning, Prediction, and Decision-Making. ii) 15 tasks from the 4 dimensions evaluating essential reasoning capabilities (e.g., numerical reasoning). Through extensive experiments, we evaluate over 30 leading proprietary and open-source LLMs, VLMs, and TSLLMs within TSRBench. Our findings reveal that: i) scaling laws hold for perception and reasoning but break down for prediction; ii) strong reasoning does not guarantee accurate context-aware forecasting, indicating a decoupling between semantic understanding and numerical prediction; and iii) despite the complementary nature of textual and visual forms of time series as inputs, current multimodal models fail to effectively fuse them for reciprocal performance gains. TSRBench provides a standardized evaluation platform that not only highlights existing challenges but also offers valuable insights to advance generalist models. Our code and dataset are available at https://tsrbench.github.io/.