MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision

AI summary

Overview

Research area: Multimodal machine learning — specifically the evaluation and modeling of temporal (time-series) data jointly with vision and language.

Technical level: Advanced. The paper combines a large-scale benchmark construction effort with an architectural contribution (a time-series encoder integrated into an omni-modal LLM), and assumes familiarity with MLLMs, LoRA fine-tuning, RoPE positional encodings, and forecasting metrics such as MAPE.

Scope: The paper introduces MMTA, a tri-modal benchmark of time series, text, and real-world vision across nine domains, plus TimeOmni-v, a multimodal LLM extension with a dedicated time-series encoder that the authors built to address the failures MMTA exposes.

What This Paper Is About

Most existing multimodal time-series benchmarks pair a temporal signal with either text or with plots rendered from that same signal, so the "visual" input is not independent evidence. This leaves unanswered whether multimodal large language models (MLLMs) can integrate time series with genuine real-world visual observations such as video of a robot manipulation or a chest X-ray. The paper builds a benchmark, MMTA, to test exactly that, and then proposes a model, TimeOmni-v, to fix the bottleneck the benchmark reveals.

Key Contributions

1. The MMTA benchmark. A collection of tri-modal tasks with diverse cross-modal complementarity patterns among vision, text, and time series, spanning nine domains under two layouts (video+TS and image+TS), with a 3,524-sample test split covering both classification and forecasting.

2. A standardized evaluation suite. Heterogeneous real-world sources are unified into a shared schema with domain-specific prompts, provenance and license documentation, human and mechanical quality audits, and success-weighted metrics that penalize both incorrect answers and unparsable outputs.

3. A systematic characterization of failure. Eight representative open-source and proprietary MLLMs are evaluated, exposing a common failure mode in which serializing time series as text produces excessive token length and weak visual–temporal alignment.

4. TimeOmni-v. A multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head, which improves long-video classification and inference efficiency and can be fine-tuned on long-TS domains where the same backbone with textualized time series runs out of memory.

Main Findings

• Baselines struggle badly on long-sequence domains. On the AgiBot robotics domain, all eight reference MLLMs stay below 0.13 swF1, and Qwen3-VL-30B-A3B-Instruct runs out of memory on every sample. Overall Avg-swF1 for the baselines ranges from 0.268 (Qwen2.5-Omni-7B) to 0.442 (Gemini-3-flash).

• TimeOmni-v leads on classification, not on forecasting. TimeOmni-v attains the highest Avg-swF1 (0.601 versus 0.442 for Gemini-3-flash) with its largest gains on AgiBot (0.844), CoVLA (0.759), and HoloAssist (0.694). On Avg-swMAPE it is mid-pack at 90.8, while Qwen3-Omni-30B-A3B-Instruct achieves the best 56.7.

• The classification gap concentrates on long video plus long time series. On AgiBot, CoVLA, and HoloAssist — which pair multi-second to multi-minute video with hundreds-to-thousands of TS steps — every baseline stays below 0.6 swF1. On short-or-fixed subsets the picture flips: Gemini-3-flash leads on CUHK-X by 12.0 percentage points, and TimeOmni-v leads on Future Factories by 4.7 percentage points, a margin the authors note is within the uncertainty of its 46 test samples.

• Scale alone does not close the gap. Neither the 30B-class Qwen3-Omni-30B-A3B-Instruct nor the proprietary models exceed 0.13 swF1 on AgiBot.

• Text serialization of time series is 232.8x more expensive in tokens. On the AgiBot test set (131 samples, the longest median TS in MMTA), Qwen2.5-Omni-7B using TS-as-text consumes 167,041 TS tokens, 27.70 s per sample, and 108.6 GB peak VRAM, completing 113 of 131 samples. TimeOmni-v uses 722 TS tokens, 1.20 s per sample, and 35.0 GB, completing 131 of 131 — ratios of 232.8x, 23.0x, and 3.1x respectively.

• Compression is dataset-independent, speedup is not. TimeOmni-v represents a T×C input as P·C+2 tokens with patch size p=16. The compression ratio approaches p·k, where k is the sub-word tokens spent per numeric value (typically above 10). On AgiBot the effective k is approximately 14.5. Per-sample speedup shrinks toward unity on short-TS image+TS domains.

• Every modality matters, but the dominant one varies. In the training-time modality ablation, no subset improves on the full model by more than 0.7%, and each reduced subset is worse on at least two datasets where it is defined. Dropping text costs nothing on CoVLA (vision-led); dropping time series costs nothing on MIMIC (text-led). On the three forecasting datasets every subset raises MAE by 19.7–81.4%.

• Fine-tuning explains much of the short-series gap; the encoder makes long series trainable. The adapted TS-as-text baseline matches TimeOmni-v on MIMIC (−0.4%) and beats it on CoVLA (+11.0%), but runs out of memory on AgiBot and HoloAssist. On forecasting, TS-as-text has 7.8–55.0% higher MAE.

• Non-LLM baselines beat the LLM on some classification tasks. On AgiBot, the TS probe, vision probe, and late fusion exceed TimeOmni-v by 7.9–8.9%; on CoVLA and MIMIC they trail by 1.7–19.6%. The same features with an MLP regression head collapse on forecasting, raising MAE by 45.1% on PixelRec and by more than an order of magnitude on SP500 and Terra.

• The integration is not tied to one backbone. Ported to InternVL3.5-8B, it parses every sample and stays within 12% of the Qwen2.5-Omni-7B version on all four tested datasets, reaching 0.746 swF1 on AgiBot.

• Data quality was audited. CoVLA rule-derived brake labels reach 96% accuracy (Cohen's κ=0.910) and Terra LLM-generated descriptions 93% (κ=0.687), each on 100 independently checked samples. All 25,057 video+TS samples passed a mechanical audit; 4 HoloAssist samples (0.016%) do not fully cover the target window, and 1.42% of released frames lie outside it.

Methodology in Plain English

The authors first curated nine openly available real-world datasets in which video or images, time series, and text naturally co-occur. Each was converted into a single unified instance format: an atomic unit of analysis (a robot task frame, a braking segment, an action trial, a production cycle, an action segment, a clinical anchor point, an item, an asset, a grid cell) with an aligned visual stream, a numeric temporal stream, and a text prompt. Tasks were kept in two formats — multiple-choice classification and numerical forecasting — and scored with success-weighted metrics: Macro-F1 multiplied by the fraction of parsable answers for classification, and MAPE divided by that fraction for forecasting. This way a model that crashes or emits malformed output is penalized rather than ignored.

They then ran eight off-the-shelf MLLMs through the benchmark. The recurring problem was that turning a long numeric series into digit strings bloats the prompt beyond the context window and gives the model no way to know which numeric reading corresponds to which video frame.

To address this, TimeOmni-v bolts a pretrained Chronos-2 time-series encoder onto Qwen2.5-Omni in parallel with its vision encoder. A small adapter maps patch embeddings into the LLM's hidden dimension. Time-series tokens get their own positional code so that a TS patch and the video frame covering the same moment share the same temporal coordinate. The tokens are then either interleaved frame-by-frame with the video or grouped as a block next to a still image. For forecasting, a lightweight head projects the LLM's hidden states directly to a multi-step prediction instead of decoding digits one at a time. Classification reuses the standard language-model head. The multimodal LLM half is fine-tuned with LoRA while the TS branch, adapter, and forecast head are fully fine-tuned.

Three controlled comparisons isolate the causes: a TS-as-text baseline with the same backbone, LoRA settings, and budget; non-LLM linear probes and late fusion over the frozen TimeOmni-v feature extractors; and a port of the TS integration onto InternVL3.5-8B.

Why This Matters

Impact on research. The paper argues that the field's existing multimodal time-series benchmarks conflate a signal with a rendering of itself. MMTA provides a substrate where the visual modality is independent evidence, and the results show a failure mode — text serialization of long series — that neither model scale nor proprietary tuning fixes. The success-weighted metrics also make generation failures a first-class part of the score rather than an excluded artifact.

Real-world applications.

• Industrial and robotic monitoring: verifying whether a manipulation episode succeeded or diagnosing anomalies such as a paused task, a meaningless operation, or a fall, using egocentric video plus joint and end-effector traces.
• Clinical decision support: predicting in-hospital mortality or discharge readiness by combining vital-sign trajectories, a chest radiograph, and the aligned radiology report.
• Autonomous driving: classifying why a vehicle braked from dashcam clips synchronized with ego and lead-vehicle state channels.
• Finance and climate forecasting: projecting closing prices over the next 12 trading days from candlestick charts, price series, and news snippets, or forecasting 20 daily climate variables per grid cell from remote-sensing image stacks.

Industry relevance. The efficiency results are directly practical: a 232.8x reduction in time-series tokens and a 23.0x per-sample latency reduction with 3.1x lower peak VRAM on the worst-case domain is the difference between a workload that fits on available hardware and one that does not. The authors report that 14% of AgiBot samples cause the text-serialization baseline to exceed a 140 GB H200 activation budget, whereas TimeOmni-v finishes all samples.

Future Directions

• Multimodal forecasting remains unsolved. TimeOmni-v does not lead on Avg-swMAPE (90.8), and the paper explicitly frames the remaining difficulty of multimodal forecasting as an open problem surfaced by MMTA.
• Closing the short-series gap. Fine-tuning explains most of the short-series performance, and the TS-as-text baseline is ahead on CoVLA by 11.0%, suggesting that exact numeric readout, which patch compression loses, remains an open question for encoder-based designs.
• Generalizing the integration recipe. The port to InternVL3.5-8B was evaluated on only four datasets (CoVLA, AgiBotWorld, MIMIC, FinMultiTime), leaving open how widely the design transfers to other omni-modal backbones.
• Robustness under missing or weakly aligned evidence. The paper positions MMTA as a substrate for studying modality reliance and for reasoning when evidence is missing or poorly aligned; the training-time ablation shows every reduced input loses somewhere, but the test-time reliance study is deferred to Appendix D.3.

Target Audience

Researchers and engineers working on multimodal large language models, time-series foundation models, or multimodal benchmark design. The paper is most useful to readers who need to evaluate whether an LLM can genuinely fuse temporal signals with visual evidence, and to those designing architectures that must consume long time series alongside video or images without exhausting the context window. Practitioners in robotics, healthcare, autonomous driving, finance, and climate modeling will find the domain-specific task definitions directly relevant.

Authors’ abstract

Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal. This leaves open whether multimodal large language models (MLLMs) can integrate time series with real-world visual evidence. We introduce MMTA (multimodal temporal analysis), a benchmark of tri-modal tasks with diverse cross-modal complementarity patterns among time series, text, and vision. MMTA spans nine domains -- robotics, autonomous driving, human action, manufacturing, AR assistance, healthcare, recommendation, finance, and climate -- under video+TS and image+TS layouts, with a 3,524-sample test split covering classification and forecasting. The benchmark standardizes heterogeneous real-world sources into a shared schema with domain-specific prompts, provenance documentation, and success-weighted metrics that penalize both wrong answers and failed generations. Evaluating eight open-source and proprietary MLLMs reveals a shared bottleneck: representing time series as text creates very long token sequences and weakens fine-grained alignment with visual evidence, especially in long-video settings. We then introduce TimeOmni-v, a multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head. TimeOmni-v improves long-video classification and time-series inference efficiency, showing that MMTA not only exposes representation and grounding failures but also motivates targeted model design.