TimeBraid: Unifying Time Series and Language for Understanding and Forecasting

AI summary

Overview

• Research area: Multimodal machine learning at the intersection of time-series analysis and natural language processing — specifically, unified models that handle continuous numerical signals and discrete text within one set of weights.
• Technical level: Advanced. The paper assumes familiarity with transformer architectures, mixture-of-experts/Mixture-of-Transformers designs, attention mechanisms, quantile (pinball) loss, and forecasting evaluation metrics such as MSE, MAE, MASE, CRPS, and RCRPS.
• Scope (one sentence): The paper introduces TimeBraid, a family of unified time-series and language models at 1.2B, 2.5B, and 6.7B parameters that align a pretrained language model with a pretrained time-series foundation model through interleaved global residual attention, and evaluates them across twelve understanding and forecasting benchmarks.

What This Paper Is About

Time series — physiological recordings, power-grid loads, market prices, climate measurements — are numerical records of dynamical systems and carry little of the semantic context needed to interpret them, while language supplies exactly that context. Existing systems connect the two modalities by bolting specialized components (projection layers, signal tokenizers) onto pretrained language or time-series models, which introduces representational gaps, causes semantic misalignment, degrades the pretrained capabilities, and typically covers only one direction of the series–text interface.

The goal of this work is a single model that natively understands and generates both modalities: it should inherit instruction following and reasoning from the language side and continuous-signal perception and zero-shot forecasting from the time-series side, fusing both in a shared representation space rather than in a grafted-on adapter.

Key Contributions

1. A tri-expert Mixture-of-Transformers architecture with global residual attention. Separate pretrained experts handle time-series perception, text reasoning, and target-series forecasting. At every paired layer, each tower projects its hidden states into a shared attention space of width d, and the projected tokens of both streams are reordered onto one global chronological axis and attended jointly under a causal mask with RoPE. Output projections are zero-initialized so each global layer starts as an identity map, and cross-modal information is exchanged token-to-token in both directions.

2. A unified prompting scheme plus a two-stage data curriculum. Every task is expressed as a segment sequence S = (s₁, …, s_M) of text and time-series segments, each marked as context or target, with unconstrained segment count and ordering (a multivariate series enters as one segment per variable). Stage 1 (alignment) uses 2.23M paired examples split 3:7 between understanding and forecasting; Stage 2 (supervised fine-tuning) uses a heterogeneous mixture with a one-pass inventory of 4,881,583 samples.

3. Stabilized joint training. Because the time-series regression loss can spike by orders of magnitude on strongly non-stationary segments and disrupt language learning through shared backbone updates, the recipe combines robust normalization (targets transformed as asinh((x − μ)/σ) using rolling history statistics) with loss capping (each response's time-series loss rescaled by min(1, c / stop_gradient(ℓ))), plus separate losses ℒ = ℒ_CE + α·ℒ_TS with α = 0.5.

4. Systematic ablations of the design space. Six research questions are tested empirically — interaction space, expert separation, optimization stability, modality balancing, necessity of the alignment stage, and inference-time text weighting — isolating the shared attention space, stabilized joint optimization, and the alignment stage as the choices that matter most.

Main Findings

• Time-series understanding (TSAQA): TimeBraid-6.7B reaches 80.65% overall accuracy, versus 85.26% for TSAQA-specific LLaMA3.1-8B SFT (and 84.29% for the analogous Qwen3-8B SFT reported in the text) and 63.10% for GPT-5.4. TimeBraid-2.5B reaches 78.31% and TimeBraid-1.2B reaches 76.61%, so even the smallest variant substantially exceeds GPT-5.4.
• TemporalBench multiple choice (TB-MCQ): TimeBraid-6.7B scores 36.58%, on par with GPT-5.4 at 38.50%; TimeBraid-2.5B scores 35.74% and TimeBraid-1.2B 34.17%.
• TimeSeriesExam: TimeBraid-6.7B reaches 63.14%, compared with 49.06% for TimeOmni-VL.
• Contextual description (CaTS-Bench human-rewritten split): In the primary comparison, TimeBraid-6.7B ranks second on DeBERTa-F1 (0.712), BLEU (0.123), and ROUGE-L (0.313) and achieves the best SimCSE (0.886); TimeBraid-2.5B remains second or third on the same alignment metrics, and all three variants lead every other series–text model by a wide margin on every alignment metric. Numeric fidelity is the weakest axis (0.684 for TimeBraid-6.7B), below general-purpose models, which the authors attribute to the time-series backbone compressing each patch of 32 raw values into a single embedding — a reduction that preserves shape and dynamics but loses fine-grained local values.
• Contextual forecasting: A TimeBraid variant ranks among the top three on eight of the nine TimeMMD domains in the summary table, and TimeBraid attains the best Macro MSE and Macro MAE on CGTSF. On TimeMMD, TimeBraid-6.7B reports MSE values such as 0.018 (Economy), 0.231 (Energy), 0.925 (Health), and 0.153 (Traffic).
• Controlled forecasting (Ctrl-F): TimeBraid-6.7B achieves 43.33% Top-1 accuracy against a 33.33% chance level and an MSE of 6.232 — the only non-LLM model whose forecasts follow the stated condition, whereas the listed time-series foundation models and multimodal forecasting models all sit at chance (33.33%). The strongest LLMs remain ahead on Top-1 (GPT-4o 60.00%, DeepSeek-Chat 59.33%, GPT-5.4 51.33%).
• CAF and CiK: TimeBraid-6.7B achieves the lowest pooled CRPS on CAF (0.225, measured over all 904 correct-context test cases), while TimeBraid-2.5B (0.289 RCRPS) performs comparably to the strongest listed time-series foundation model on CiK. LLMs stay ahead on CiK (Gemini-2.5-Flash 0.110, GPT-5.4 0.145), which the authors attribute to their ability to convert the control signal to output and their deep reasoning.
• Unimodal forecasting (GIFT-Eval): The aggregate table reports MASE and CRPS for TimeBraid alongside statistical methods, task-specific supervised models, and time-series foundation models; the paper states TimeBraid largely maintains comparable zero-shot forecasting to state-of-the-art time-series foundation models, outperforming the listed statistical and task-specific supervised baselines while the strongest specialized foundation models remain ahead.
• Long-horizon ETT and Weather: TimeBraid-2.5B achieves the second-best average MSE rank (2.48) and third-best average MAE rank (3.45) across the 20 dataset–horizon settings, largely preserving the unimodal forecasting capability of TimesFM 2.5, with the authors expecting the gap to shrink under a more balanced data recipe.
• Ablation — interaction space (RQ1): Replacing global residual attention with a VLM-style MLP connector makes the language loss plateau early at roughly twice the level residual attention reaches, and the gap never closes; the authors observe that time-series patches get projected to global statistics such as mean and standard deviation, and conclude the language space is a poor projection target for temporal representations.
• Ablation — expert separation (RQ2): A tri-tower variant with an independent perception expert and forecasting expert produces trajectories "nearly identical" to the dual-tower model on both losses, unlike vision models where understanding and generation favor decoupled representations.
• Ablation — optimization stability (RQ3): With plain causal RevIN, the time-series loss spikes recurrently over several orders of magnitude and every spike disrupts the language loss through shared updates; robust normalization plus loss capping removes both symptoms.
• Ablation — balancing (RQ4): The language loss is stable across the tested shared-learning-rate grid while the time-series loss degrades at the largest rate, motivating α = 0.5. Every forecasting-heavy data mixture (3:7 to 2:8) reaches the same lower time-series loss, whereas a balanced 5:5 mixture is clearly worse, so 3:7 is adopted as the tested range's Pareto point. The time-series objective is far more sensitive than the language objective.
• Ablation — alignment stage (RQ5): Aligned initialization stays ahead at every checkpoint on TSAQA understanding, with the largest margins early; on TimeMMD the two runs end at nearly the same MSE; on CAF the aligned run keeps a lower qCRPS at every checkpoint and SFT alone never closes the gap — so alignment pays where the forecast must draw on language.
• Ablation — inference-time text weight (RQ6): No single text weight works across all domains. At least one nonzero weight helps at every history length (8–128) in Public Health, Economy, Agriculture, Energy, Environment, and Traffic; Security prefers λ = 0; Climate and Social Good change sign with history. A single λ = 0.3 is selected on validation data and fixed across all domains, reported as a sensitivity analysis rather than model selection.

Methodology in Plain English

The researchers started from two already-trained models — a language model and a time-series foundation model — and asked how to let them work together without destroying what each already knows.

Rather than squeezing time series into the language model's representation space (the usual approach), they give each modality its own tower and add a small number of shared "global residual attention" layers that let tokens from both towers see each other. The towers are interleaved by depth: because the time-series tower has fewer layers, each of its layers is paired with one language layer, and the unpaired language layers process only their own tokens. Every example is written as a chat transcript in which time-series segments are wrapped in <ts></ts> tokens, each context segment is preceded by a statistics block giving its length, mean, and standard deviation, and forecasting targets are normalized with rolling RevIN based on their own history prefix.

Training happens in two stages. First, an alignment stage teaches basic series understanding and controllable forecasting on 2.23M constructed pairs, including captions built "attribute-first" so each description matches its series by construction, causal-model-generated multivariate examples, and forecasting examples whose futures are synthesized from one shared history with captions stating the condition — annotations see the future, but the model never does. Second, supervised fine-tuning mixes question answering, captioning, long-horizon forecasting, and domain tasks. Because the forecasting loss is scale-sensitive and can blow up on non-stationary data, the targets are compressed with an inverse-hyperbolic-sine transform and the per-response loss is capped.

At inference, the model

Authors’ abstract

We present TimeBraid, a series of unified time-series and language models that align pretrained language models and pretrained time-series foundation models through interleaved global residual attention layers. Each model inherits knowledge, instruction following, and reasoning from one side, continuous-signal perception and zero-shot forecasting from the other, and fuses the two in a shared representation space where both modalities are understood and generated. We study the design choices that make such unified modeling work: where to align the two representation spaces, how to ground language in temporal structure, how to balance understanding with generation, and how to keep joint optimization stable. The resulting recipe combines a unified prompting scheme for diverse time-series and text tasks, stabilized joint training, and supervision from 2.2M curated series--text pairs and 4.9M instruction-tuning samples. Across benchmarks spanning time-series perception, understanding, reasoning, and both context-aided and unimodal forecasting, TimeBraid remains competitive with far larger general-purpose models and task-specific counterparts.