Decentralized Attention Fails Centralized Signals: Rethinking Transformers for Medical Time Series

AI summary

Overview

Research area: Machine learning for medical time series (MedTS) classification — specifically the architecture of Transformer attention when applied to multi-channel physiological signals such as EEG and ECG.

Technical level: Intermediate. The core argument is intuitive (centralized signals vs. decentralized attention), but the paper includes formal module definitions, tokenization equations, and a large multi-dataset benchmark.

Scope: The paper proposes CoTAR, a centralized MLP-based replacement for self-attention, and TeCh, a framework built on it, then evaluates both across five medical datasets plus two human activity recognition (HAR) datasets against 10 Transformer-based baselines.

What This Paper Is About

Medical time series are generated by centralized biological sources — EEG rhythms emerge from thalamo–cortical circuits, and ECG waveforms arise from impulses propagating from the sinoatrial node — so the channels of these recordings are coordinated by a shared, central driver. Transformer attention, by contrast, is decentralized: every token attends equally to every other token. The authors argue this structural mismatch makes attention poor at modeling channel dependencies in MedTS, because the uniform treatment of inter-channel interaction dilutes the principal, centrally driven patterns such as cardiac pacemaker rhythms. Their goal is to design a replacement for attention that keeps flexible cross-channel interaction while matching the centralized organization of the data.

Key Contributions

1. A diagnosis of the mismatch: The paper frames the weakness of Transformers on channel dependencies in MedTS as a structural mismatch between inherently centralized signals and attention's decentralized pairwise token interaction, rather than as a data or training limitation.
2. CoTAR (Core Token Aggregation-Redistribution): A plug-in, MLP-based module that replaces attention. It introduces a global core token that aggregates information from all tokens and then redistributes it back to each token, acting as a proxy for inter-token communication. This reduces token-interaction complexity from quadratic to linear.
3. TeCh, a unified framework: A CoTAR-based architecture using "Adaptive Dual Tokenization" that can model temporal dependencies, channel dependencies, or both by tuning the number of temporal (M) and channel (N) Transformer encoder blocks — setting M = 0 or N = 0 removes the corresponding branch entirely.
4. Broad empirical validation: Experiments on five MedTS benchmarks and two HAR datasets, with efficiency, robustness, and ablation analyses, plus released code and training scripts at https://github.com/Levi-Ackman/TeCh.

Main Findings

• Best performance across all five MedTS datasets: Under the subject-independent protocol, TeCh outperforms Medformer (described as the previous state-of-the-art Transformer-based MedTS model) across all six metrics on four of the five datasets, with an overall 4.11% relative performance gain over Medformer aggregated across all six metrics.
• Largest gain on APAVA: TeCh achieves up to a 11.6% relative improvement in the average of all metrics on APAVA compared with Medformer. Absolute numbers include 86.86 accuracy, 86.30 F1, 94.02 AUROC, 93.79 AUPRC, and 88.99 Avg, versus Medformer's 79.74 Avg.
• Comparable but not universally better on ADFTD: TeCh's Avg (54.16) is slightly below Medformer's (54.63), while TeCh ranks first in Accuracy (54.54) and Precision (53.02) and second in overall Avg on that dataset.
• Strong results on the remaining datasets: TeCh reaches Avg 95.07 on TDBrain (Medformer: 91.91), 87.70 on PTB (Medformer: 84.69), and 69.95 on PTB-XL (Medformer: 69.28).
• Generalization to HAR: On the two human activity recognition datasets, TeCh substantially outperforms Medformer across all metrics, with an average improvement of 4.28%. TeCh reaches Avg 84.22 on FLAAP (Medformer: 80.42) and 95.51 on UCI-HAR (Medformer: 91.94).
• Improved robustness: TeCh shows lower average standard deviation across all datasets than Medformer (0.84 vs. 0.96, a 12.37% reduction), which the authors interpret as greater stability.
• Efficiency gains: Because CoTAR is linear rather than quadratic, TeCh delivers 10.3% better accuracy on APAVA while using just 33% of the memory and 20% of the inference time compared to Medformer, at batch size B = 128.
• Noise robustness ablation: Adding Gaussian noise to the last channel of the PTB dataset while sweeping intensity β from 0.0 to 20.0 and measuring F1 shows attention is highly sensitive to noise, since each channel is directly influenced by the corrupted channel, whereas CoTAR's centralized structure prevents direct interference. Channel embedding also proved more robust and higher-performing than Temporal embedding in this test.
• Tokenization ablation: Removing the representation-learning phase degrades performance significantly across all datasets. Temporal tokenization excels on TDBrain, Channel tokenization excels on PTB, and combining both yields an 11% improvement in Accuracy and a 13% improvement in F1-Score on APAVA. Dual tokenization also excels on UCI-HAR.
• CoTAR module ablation: Replacing CoTAR yields consistent performance gains across all five datasets and both metrics, with competitive or lower standard deviations.

Methodology in Plain English

The authors start from an analogy to star-shaped network topologies, where a central server mediates all communication between clients instead of every client talking directly to every other client. They port this idea into the Transformer block: rather than computing pairwise query–key interactions among tokens, CoTAR projects each token through an MLP, produces per-token weights via a softmax over the token dimension, and uses those weights to pool all tokens into a single core vector. That core vector is then repeated back across all token positions, concatenated with the original token representations, and passed through a second MLP to produce the output. Every token therefore interacts with the rest of the sequence only through the core token, which the authors compare to the brain or heart acting as the central coordinator in EEG or ECG.

Around this module they build TeCh, which tokenizes the input two ways. Temporal tokenization flattens a window of L timestamps across all channels into one token, producing P = ⌈T/L⌉ tokens from an input X ∈ ℝ^(T×C). Channel tokenization, following iTransformer, treats the entire time series of a single channel as one token, producing C tokens. Each embedding stream passes through its own stack of Transformer encoders (M for temporal, N for channel) with attention swapped for CoTAR. The two resulting representations are averaged across channels, summed, and projected to the output logits Ŷ ∈ ℝ^K. Because M and N are tunable, the model can be configured to use temporal modeling, channel modeling, or both, depending on which patterns a given dataset exhibits.

Evaluation uses the subject-independent protocol, where all samples from a given subject appear in exactly one of the training, validation, or test splits, reflecting deployment on unseen patients. Six metrics are reported — accuracy, precision, recall, F1, AUROC, and AUPRC, with precision, recall, F1, AUROC, and AUPRC macro-averaged. Training uses five random seeds (42–46) to compute means and standard deviations, runs on a single NVIDIA RTX 4090 GPU, and selects the model with the best validation F1. Baseline results on the five MedTS datasets are taken directly from Medformer, and HAR results are reproduced using Medformer's official code.

Why This Matters

Impact on research: The paper reframes a modeling limitation as an architectural mismatch, suggesting that inductive biases borrowed from engineering (centralized star topologies) may suit biomedical signals better than the fully connected interaction pattern that dominates sequence modeling. It also draws attention to channel dependencies, which the authors note remain underexplored in MedTS relative to temporal dependencies. The linear complexity of CoTAR offers a path to scaling to long or high-dimensional medical sequences where attention is expensive.

Real-world applications:

• Neurological diagnosis, including Alzheimer's disease detection and Parkinson's disease detection from EEG, and three-way classification of Healthy, Frontotemporal Dementia, and Alzheimer's disease.
• Cardiac diagnosis, including binary myocardial infarction detection and five-class classification of heart conditions from ECG.
• Brain-computer interfaces and continuous health monitoring, where multi-channel physiological data is collected over long periods.
• Human activity recognition from smartphone accelerometer and gyroscope sensors, relevant to fitness tracking, rehabilitation, and elder care.

Industry relevance: The efficiency results are directly relevant to deployment on resource-constrained hardware. Using 33% of the memory and 20% of the inference time while improving accuracy matters for clinical edge devices, ambulatory monitors, and any setting where inference cost limits how much data can be processed. The robustness to a corrupted channel is also relevant to real deployments, where electrode failure or motion artifacts are common.

Future Directions

1. Theoretical grounding for the centralized proxy: The paper motivates CoTAR by analogy to star-shaped systems and biological central coordination but does not provide formal analysis of when the core-token bottleneck preserves information and when it discards it. Establishing such conditions is a natural next step.
2. Systematic study of the granularity hyperparameter: Temporal tokenization depends on a predefined window length L, and the encoder counts M and N are described as tunable to match data. The paper does not report a systematic search over these settings, leaving open how to select them automatically for a new dataset.
3. Extending beyond EEG and ECG: The evaluation covers three EEG datasets and two ECG datasets, with EMG and EOG mentioned as other MedTS modalities but not tested. Whether CoTAR transfers to those, and to other centralized multi-sensor signals, is untested here.
4. Closing the remaining channel-dependency gap: The authors note channel dependencies remain underexplored relative to temporal dependencies. The ADFTD result, where TeCh's overall average trails Medformer's while individual metrics lead, suggests the centralized design does not improve every dataset uniformly — understanding which signal regimes benefit most is an open question.

Target Audience

This paper is most useful to machine learning researchers working on time series architectures and attention alternatives, biomedical engineers and signal-processing researchers building diagnostic models from EEG and ECG, and applied ML practitioners in clinical or wearable-health settings who need accurate multi-channel classification under tight memory and latency budgets. Readers interested in inductive-bias design — matching model structure to the physical generative process of data — will find the paper's central argument particularly relevant.

Authors’ abstract

Accurate analysis of medical time series (MedTS) data, such as electroencephalography (EEG) and electrocardiography (ECG), plays a pivotal role in healthcare applications, including the diagnosis of brain and heart diseases. MedTS data typically exhibit two critical patterns: temporal dependencies within individual channels and channel dependencies across multiple channels. While recent advances in deep learning have leveraged Transformer-based models to effectively capture temporal dependencies, they often struggle with modeling channel dependencies. This limitation stems from a structural mismatch: MedTS signals are inherently centralized, whereas the Transformer's attention mechanism is decentralized, making it less effective at capturing global synchronization and unified waveform patterns. To address this mismatch, we propose CoTAR (Core Token Aggregation-Redistribution), a centralized MLP-based module designed to replace decentralized attention. Instead of allowing all tokens to interact directly, as in standard attention, CoTAR introduces a global core token that serves as a proxy to facilitate inter-token interactions, thereby enforcing a centralized aggregation and redistribution strategy. This design not only better aligns with the centralized nature of MedTS signals but also reduces computational complexity from quadratic to linear. Experiments on five benchmarks validate the superiority of our method in both effectiveness and efficiency, achieving up to a 11.6% improvement on the APAVA dataset, while using only 33% of the memory and 20% of the inference time compared to the previous state of the art. Code and all training scripts are available at https://github.com/Levi-Ackman/TeCh.