Adapting Neural Audio Codecs to EEG

AI summary

Overview

Research area: Machine learning for biomedical signal processing, specifically neural audio compression (audio codecs) adapted to electroencephalography (EEG).

Technical level: Intermediate. The paper assumes familiarity with autoencoders, residual vector quantization, and basic EEG terminology, but its central idea is intuitive and clearly explained.

One-sentence scope: The paper tests whether a neural audio codec pretrained on 44.1 kHz audio (DAC) can be repurposed to compress EEG, and measures how well the resulting reconstructions retain clinically relevant information.

What This Paper Is About

EEG data is expensive to collect and privacy-constrained, so public EEG datasets are far smaller than audio datasets, making it hard to train general-purpose EEG codecs from scratch. The authors ask whether a codec already trained on large-scale audio can instead be reused for EEG — turning raw brain signals into compact discrete tokens — despite the two modalities differing in sampling rate, channel structure, and scale. Their goal is compression that preserves information a clinician or downstream classifier would care about.

Key Contributions

1. Demonstration that an off-the-shelf audio codec reconstructs EEG without modification. Feeding EEG into the pretrained DAC model produces stable and interpretable reconstructions, and fine-tuning on EEG improves fidelity and generalization over training from scratch.

2. DAC-MC, a multi-channel extension. The authors add attention-based aggregation across electrodes plus channel-specific decoding via learned style vectors (inspired by StyleGAN), while keeping the audio-pretrained initialization. Channels are processed in groups of at most 5, using either random grouping or manual anatomical/montage groups.

3. A systematic compression-quality study. They vary residual codebook depth, codebook (vocabulary) size, and internal sampling rate to map the trade-off between bitrate and reconstruction fidelity, including separate "pre fine-tune" and "post fine-tune" pruning of codebooks.

4. Clinical-indistinguishability evaluation. Reconstruction is assessed both by spectrogram-based loss and by whether downstream classification of TUAB (abnormal) and TUEP (epilepsy) still works on reconstructed signals alone.

Main Findings

• Fine-tuning beats scratch and beats no fine-tuning. Spectrogram reconstruction loss was approximately 1.05 for the Audio-to-EEG fine-tuned model, versus 1.46 for Scratch and 2.50 for the Audio-Pretrained model without fine-tuning. The authors note this nearly matches the original DAC model's audio performance of 1.09.

• Moderate codebook pruning is cheap. Reducing residual codebooks from 9 to 6 caused less than a 10% increase in spectrogram loss, described as a favorable trade-off. Reducing from 9 to 3 codebooks doubled the spectrogram loss.

• Pruning after fine-tuning hurts more than training with fewer codebooks. Post fine-tune pruning consistently produced higher spectrogram losses than pre fine-tune pruning; the authors trained 7 distinct models for residual depths from 3 to 9.

• Upsampling gains are marginal; shrinking the vocabulary mostly hurts. Going from 256 Hz to 512 Hz gave negligible fidelity improvement relative to its higher computational cost, and reducing the alphabet from 1024 to 512 entries degraded reconstruction unless combined with iterative pruning and further fine-tuning.

• Epilepsy detection benefits from cross-channel modeling. Grouped DAC-MC (Random or Manual groups) reached 85% on epilepsy versus 80% for the single-channel DAC-SC, against a baseline of 84% on original signals.

• Abnormal detection does not. Grouped DAC-MC underperformed at 78% on the Abnormal task, whereas the same multi-channel model run in single-channel decoding mode reached 81%, closer to DAC-SC (83%) and the baseline (82%). The authors suggest per-channel decoding preserves abnormality-related features better than cross-channel grouping.

• Repurposing works despite the temporal mismatch. A fixed window of 512 samples corresponds to roughly 13 ms in 44.1 kHz audio but a full 1 second at 512 Hz EEG; the pretrained model still yields stable outputs.

• Bitrate figures reported are 90 bts at 512 Hz and 180 bts at 256 Hz, as stated in the paper's configuration discussion.

Methodology in Plain English

The authors take DAC (an improved residual vector-quantized GAN codec), pretrained on 44.1 kHz audio, and repurpose it for EEG. Raw EEG is clipped to ±200 μV, normalized to the [-1, 1] range (the scale audio codecs expect), resampled to 512 Hz, high-pass filtered at 0.1 Hz, and cut into non-overlapping 30-second windows. The signal is then fed directly into the codec's fixed framing, letting the audio-pretrained encoder and decoder be reused.

For multi-channel EEG, each channel is first encoded separately, the resulting latents are concatenated, self-attention mixes information across channels and time, and the result is projected back for quantization and decoding. Learned per-channel style vectors apply scale-and-bias affine transformations in the decoder so each channel can be reconstructed distinctly; fixed channel ordering and positional encodings retain spatial context.

Training uses three comparison setups: from scratch on EEG only, pretrained without fine-tuning, and pretrained then fine-tuned. The optimizer is Adam with a learning rate of 1e-5 and beta parameters (0.8, 0.999), across 4 RTX3090 GPUs. The loss is DAC's composite loss (waveform, multi-scale STFT, Mel-spectrogram, adversarial, commitment, and codebook terms), with the Mel-spectrogram loss swapped for a standard spectrogram loss suited to EEG. Adversarial losses proved unstable on EEG, so training used two phases: reduced-weight GAN losses first, then removal once divergence appeared.

Evaluation uses two prongs: spectrogram-based reconstruction loss between original and reconstructed signals, and downstream classification with Random Forests and Decision Trees trained on frequency-domain features extracted using the Brainfeatures library, on the TUAB and TUEP benchmarks.

Why This Matters

The work suggests that scarce biomedical data can borrow representation strength from abundant data in a different modality, sidestepping the expense of training codecs from scratch on small EEG corpora. It also introduces a practical route to tokenizing continuous brain signals, which is a prerequisite for applying next-token-prediction style sequence models to EEG.

Real-world applications implied by the paper's framing:

• Epilepsy diagnosis, where the authors report 85% classification accuracy on reconstructed signals from the TUEP subset.
• Abnormal EEG screening, via the TUAB subset, though grouped multi-channel decoding underperformed per-channel decoding here (78% vs 81%).
• Sleep staging and cognitive assessment, listed in the paper's introduction as core EEG monitoring applications.
• Compact storage and indexing of clinical EEG, since discrete tokens are easier to store, index, and model than raw continuous recordings.

Industry relevance: the results matter to developers of clinical monitoring pipelines and health-AI foundation models, because they show a viable path to reusing large pretrained checkpoints — including pretrained audio models — rather than collecting orders of magnitude more labeled EEG. The finding that model choices behave differently across tasks (epilepsy favors cross-channel grouping, abnormality favors per-channel decoding) is a caution that task-specific validation is necessary before deployment.

Future Directions

• Resolve the task-dependent channel strategy. Why cross-channel grouping helps epilepsy but hurts abnormality detection is unresolved; the paper offers an explanation rather than a mechanism.
• Stabilize adversarial training for EEG. The authors had to drop GAN losses after observing divergence due to domain mismatch, so a more robust adversarial objective could improve fidelity.
• Close the gap to the baseline. Reconstructed-signal classification still trails the baseline on both tasks (85% vs 84% is an improvement for epilepsy under grouping, but abnormal detection tops out at 83% for DAC-SC against an 82% baseline, with grouped models at 78%), leaving room to better preserve clinical detail.
• Scale to more channels and larger montages. The paper groups channels into subsets of at most 5 because full attention across DAC's 1024-dimensional latent space is computationally expensive; more efficient cross-channel attention is an open engineering problem.

Target Audience

Researchers and practitioners working on EEG modeling, biomedical signal compression, or cross-modal transfer learning will benefit most, particularly those building foundation models over physiological time series. It is also relevant to clinical ML engineers evaluating whether reconstruction-based pipelines are safe enough for diagnostic tasks, and to audio compression researchers curious about out-of-domain generalization of codecs. Readers should be comfortable with autoencoders, vector quantization, and basic spectrogram analysis to get full value from the trade-off experiments.

Authors’ abstract

EEG and audio are inherently distinct modalities, differing in sampling rate, channel structure, and scale. Yet, we show that pretrained neural audio codecs can serve as effective starting points for EEG compression, provided that the data are preprocessed to be suitable to the codec's input constraints. Using DAC, a state-of-the-art neural audio codec as our base, we demonstrate that raw EEG can be mapped into the codec's stride-based framing, enabling direct reuse of the audio-pretrained encoder-decoder. Even without modification, this setup yields stable EEG reconstructions, and fine-tuning on EEG data further improves fidelity and generalization compared to training from scratch. We systematically explore compression-quality trade-offs by varying residual codebook depth, codebook (vocabulary) size, and input sampling rate. To capture spatial dependencies across electrodes, we propose DAC-MC, a multi-channel extension with attention-based cross-channel aggregation and channel-specific decoding, while retaining the audio-pretrained initialization. Evaluations on the TUH Abnormal and Epilepsy datasets show that the adapted codecs preserve clinically relevant information, as reflected in spectrogram-based reconstruction loss and downstream classification accuracy.