Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting

AI summary

Overview

Research area: Multimodal time-series forecasting (numerical series plus paired text), specifically frequency-domain / spectral fusion methods.

Technical level: Intermediate to Advanced. Readers should be comfortable with the Discrete Fourier Transform, complex-valued neural network layers, and cross-attention mechanisms.

Scope: A single paper proposing SpecTF, a frequency-aware framework that fuses text embeddings with time-series spectral components, evaluated on two multimodal benchmarks and compared against unimodal, multimodal, and LLM-based baselines.

What This Paper Is About

Most multimodal forecasting methods combine text with time series one timestep at a time, aligning each text document to a single temporal position. The authors argue this misses the fact that a single piece of text — a policy announcement, a supply-chain report — can simultaneously drive short-term volatility and long-term trends, which live in different frequency bands. SpecTF instead decomposes the time series into frequency components, projects the text into the same complex frequency space, and lets cross-attention decide which text is relevant to which frequency band before predicting and converting back to the time domain.

Key Contributions

1. A frequency-domain formulation for multimodal forecasting. The authors state they are the first to model the multiscale temporal influence of text on time series by fusing both modalities in the spectral domain rather than the time domain, where high-frequency components represent short-term volatility and low-frequency components capture prolonged trends.

2. The Frequency Cross-Modality Fusion block (FreqCMF). A fusion mechanism combining Frequency Cross Attention (FCA), which queries textual features from time-series frequency components using softmax(|Q K^T| / sqrt(d_k)) · V, with Multiplication Fusion (MF), a complex-valued multiplication that alters both amplitude and phase of spectral components.

3. Demonstrated accuracy gains on the Time-MMD benchmark. SpecTF outperforms both unimodal and multimodal baselines on MSE and MAE in 8 of the 9 Time-MMD domains, with an average improvement across the 9 datasets of 3.82% (MSE) and 2.25% (MAE) over the best multimodal baseline.

4. Strong parameter and compute efficiency. On the Energy dataset at a forecast horizon of 48, SpecTF uses 397.1K trainable parameters versus 1.72M for TimeXer and 6.41M for both MM-TSF and TaTS, with 1.8M MACs and 9.1ms inference time.

Main Findings

• Time-MMD benchmark results. Averaged across three seeds and all prediction lengths, SpecTF achieves the best MSE/MAE in Agriculture (0.103 / 0.218, an 8.35% / 8.02% reduction versus the best baseline), Climate (0.938 / 0.762, 6.38% / 3.54%), Energy (0.246 / 0.359, 6.82% / 4.01%), Environment (0.259 / 0.367, 2.63% / 0.81%), Health (1.276 / 0.733, 4.78% / 3.93%), Security (108.411 / 4.745, 0.97% / 2.56%), Social Good (0.962 / 0.443, 4.94% / 1.34%), and Traffic (0.171 / 0.210, 2.85% / 0%).

• One domain where it loses. On Economy, SpecTF records MSE 0.0085 and MAE 0.0780, a −2.41% / −1.30% change relative to the best baseline. TaTS edges it out. The authors attribute this to noisier text containing shorter-term economic indicators better suited to time-domain alignment.

• TTC benchmark results. On the TimeText Corpus, SpecTF reaches MSE 0.693 / MAE 0.605 on Climate (8.81% / 4.12% improvement) and MSE 1.298 / MAE 0.828 on Medical (1.00% / 5.90% improvement). Comparisons here were limited to the best-performing baselines from earlier experiments, citing computational constraints.

• Multiplication fusion matters most in some domains. Removing it degrades performance — roughly 2.6% in MAE on Health (1.276 → 1.289) — while Agriculture moves only from 0.103 to 0.104 MSE.

• Cross-attention is not optional. Replacing cross-attention with summation causes an 11.7% MSE drop on Agriculture (0.103 → 0.115) and raises Health MSE to 1.291.

• Joint complex modeling beats split real/imaginary MLPs. Replacing FreqMLPs with separate MLPs on the real and imaginary parts increases Environment MAE by 2.3% (0.367 → 0.423).

• Text helps, but numbers help more. Excluding text degrades all three ablation domains (Agriculture MSE 0.103 → 0.132, Health 1.276 → 1.342, Environment 0.259 → 0.355) with pronounced effects on Environment. Removing the time series entirely causes the largest drop, a 30.7% MSE reduction on Health (1.276 → 1.668).

• Text targets middle-frequency bands. Visualizing the Climate dataset, documents 13, 16, and 20 — referencing "warm-than-average temperatures", "extreme drought conditions", "abnormally dry conditions", and "moderate-to-exceptional drought" — draw strong attention in middle-frequency bands (2-7), and amplitudes after fusion increase most in those bands. Documents 0 and 5, with no useful information, contribute less.

• Efficiency comparison (Energy, horizon 48). SpecTF: 397.1K parameters, 1.8M MACs, 9.1ms inference. TimeXer: 1.72M, 11.1M, 88.4ms. MM-TSF: 6.41M, 8.23M, 14.1ms. TaTS: 6.41M, 84.23M, 41.3ms.

Methodology in Plain English

SpecTF processes each variable of the input time series independently (a channel-independent strategy) through five stages.

Time Series Embedding. A historical numerical window is converted with the real Fast Fourier Transform, producing complex spectral components. A frequency-domain MLP (FreqMLP, an MLP with complex weights and biases) embeds this spectrum into a higher-dimensional space, and positional encoding tags each frequency band with a unique signature.

Text Embedding. Each paired text document goes through a pretrained language model (BERT or GPT-2; the experiments use GPT-2). A timestamp embedding is added so the text aligns temporally with the series. Two separate MLPs then produce the real and imaginary parts of a complex text vector, placing the text in the same complex space as the spectral series.

FreqCMF. The time-series embedding supplies the query; the text embedding supplies the keys and values, all computed with FreqMLPs. Attention weights are formed from the absolute value of the complex query-key product, so weights reflect alignment strength. The resulting attention output is multiplied element-wise with the time-series embedding using complex multiplication. Because complex multiplication scales amplitude and adds phase, it can simultaneously amplify or damp a frequency band and shift its timing. Via the convolution theorem, this multiplication corresponds to a convolution over all timesteps at once — a global rather than step-wise interaction.

Forecaster. A FreqMLP maps the fused historical frequency representation to the frequency representation of the future window.

Projection. A final FreqMLP restores the spectrum's original dimensionality, and the inverse real Fast Fourier Transform returns the prediction to the time domain.

Training uses MSE as the loss and MAE as the evaluation metric, on a single NVIDIA Tesla V100. Full results are averaged over three seeds and all prediction lengths.

Why This Matters

The paper reframes multimodal fusion as a spectral reweighting problem rather than a temporal alignment problem. Instead of guessing which timestep a news article belongs to, the model learns which frequency band a document modulates. The theoretical appendix supports this by noting that complex multiplication gives combined amplitude modulation and phase shift, and that multiplication in frequency equals global convolution in time — a property time-domain approaches cannot match with equivalent efficiency.

Real-world applications:

• Energy and utilities: Weather, policy, and supply reports affecting both immediate load spikes and seasonal demand curves. Energy is one of the domains with the largest reported gains (6.82% MSE).
• Agriculture: The paper's case study, where text-enhanced predictions track ground truth more closely in both time and frequency views.
• Public health: The Health domain shows a 4.78% MSE improvement on Time-MMD and a 5.90% MAE improvement on the Medical TTC dataset.
• Climate and environment: Drought and temperature reporting, the setting for the paper's attention visualization.

Industry relevance: With 397.1K parameters, 1.8M MACs, and 9.1ms inference time, SpecTF is explicitly positioned for resource-limited deployment alongside large-scale use — a contrast with LLM-based baselines such as TimeLLM and ChatTime, which the paper shows are consistently less accurate here.

Future Directions

• Addressing the Economy failure case. The authors attribute the loss to noisy, shorter-horizon text; a hybrid that routes noisy text through time-domain alignment and structured text through frequency-domain fusion is an open question.
• Theoretical work is incomplete in this excerpt. Appendix A.3 is headed "Energy Conservation Guarantees" but the content is truncated, so what is actually proven there is not reported in the supplied text.
• Broadening the benchmark comparison. TTC experiments were limited to the best-performing baselines "due to computational constraints," leaving the full comparison untested on that benchmark.
• Language model choice. BERT and GPT-2 are the stated options, with GPT-2 used in practice; the paper does not report how a larger or domain-specialized encoder would change results.
• Which frequencies to trust. The Climate visualization shows text influencing middle bands (2-7), but the paper does not report whether this band preference is consistent across domains or whether it could be exploited as a prior.

Target Audience

Researchers and practitioners in time-series forecasting who work with paired textual data; engineers building forecasting systems in energy, agriculture, health, climate, or finance who need accuracy without a large parameter budget; and readers interested in frequency-domain deep learning or cross-modal attention design. Readers without background in Fourier analysis or complex-valued networks will find the method section demanding, though the motivation and empirical results are accessible.

Authors’ abstract

Multimodal time series forecasting is crucial in real-world applications, where decisions depend on both numerical data and contextual signals. The core challenge is to effectively combine temporal numerical patterns with the context embedded in other modalities, such as text. While most existing methods align textual features with time-series patterns one step at a time, they neglect the multiscale temporal influences of contextual information such as time-series cycles and dynamic shifts. This mismatch between local alignment and global textual context can be addressed by spectral decomposition, which separates time series into frequency components capturing both short-term changes and long-term trends. In this paper, we propose SpecTF, a simple yet effective framework that integrates the effect of textual data on time series in the frequency domain. Our method extracts textual embeddings, projects them into the frequency domain, and fuses them with the time series' spectral components using a lightweight cross-attention mechanism. This adaptively reweights frequency bands based on textual relevance before mapping the results back to the temporal domain for predictions. Experimental results demonstrate that SpecTF significantly outperforms state-of-the-art models across diverse multi-modal time series datasets while utilizing considerably fewer parameters. Code is available at https://github.com/hiepnh137/SpecTF.