A Simple Method to Enhance Pre-trained Language Models with Speech Tokens for Classification

AI summary

Overview

Research area: Multimodal natural language processing — specifically the fusion of speech tokens into text-only pre-trained large language models for classification tasks, applied to argument mining (fallacy detection and classification) and affective computing (sentiment and emotion recognition).

Technical level: Intermediate. The paper assumes familiarity with pre-trained LLMs, subword tokenization, LoRA fine-tuning, self-supervised language modeling, residual vector quantization, and standard classification metrics.

Scope: The paper proposes and evaluates a three-step pipeline that selects a small, task-specific subset of audio tokens and adapts them to a frozen LLM, reporting results on four classification tasks across two datasets.

What This Paper Is About

Audio is produced at roughly 50 Hz while text is produced at approximately 2 Hz, so speech tokenizers emit sequences an order of magnitude longer than their textual counterparts, creating attention sparsity, redundancy, and computational overhead when an LLM tries to process them. Rather than compressing audio sequences during inference, this work identifies, at the level of token types, which audio tokens carry discriminative information for a specific downstream task. The goal is a lightweight recipe that lets any pre-trained text LLM benefit from speech on classification tasks without the multi-stage pre-training required by large SpeechLLM pipelines.

Key Contributions

1. A task-level audio token selection method. The authors apply ℓ1-regularized logistic regression to a multimodal Bag-of-Words representation (textual and audio token histograms concatenated) to select only the audio token types that are discriminative for the target task, reducing the audio vocabulary to less than 1% of its original size prior to any fine-tuning.

2. A minimal two-phase adaptation procedure. The selected tokens are added to the LLM vocabulary as new embeddings, adapted through a causal language modeling objective while all original LLM parameters remain frozen, and then the model is fine-tuned on the downstream task using LoRA and a single linear classification head.

3. Empirical gains on tasks where audio was previously believed unhelpful. The method improves over an unimodal text model, over larger SpeechLMs including a 2.5× larger multimodal model (Qwen2-Audio-7B), and over audio integration via learned representations.

4. An in-depth ablation and analysis. The paper compares acoustic-only versus acoustic-plus-semantic tokens, lasso-based versus random selection, and with versus without audio token pre-training, and examines fine-grained prediction examples.

Main Findings

• Argumentative Fallacy Classification (AFC). On the MM-ArgFallacy2025 shared task test set in Macro-F1, LLaMA-3B fine-tuned on training data reaches 0.523 (text only, no bagging), surpassing the previous state of the art of 0.486 (RoBERTa + Whisper). Adding audio tokens raises this to 0.548; with probability-based bagging over 5 seeds the method reaches 0.592, versus 0.547 for Llama3-3B+BYOL-A and 0.515 for Qwen2-Audio-7B (SFT). The paper states this exceeds the prior state of the art by more than 10 points.

• Bagging helps the multimodal model more than the unimodal one. The paper reports +4.4% for the multimodal configuration versus +1.4% for the unimodal one, with the best results from the multimodal configuration.

• Argumentative Fallacy Detection (AFD). Positive-class F1 without bagging is 0.307 for the proposed method, against 0.285 for the (MM-)RoBERTa + WavLM baseline and 0.285 for Llama3-3B+BYOL-A; with bagging the method reaches 0.317 versus 0.301. The paper notes AFD is more challenging than AFC because of class imbalance.

• Sentiment and emotion on CMU-MOSEI. The method reaches 0.792 Macro-F1 on SENT (against 0.779 for Qwen2-Audio-7B SFT and 0.771 for Llama3-3B+BYOL-A with audio) and 0.742 on EMO (against 0.734 and 0.732 respectively).

• Acoustic-only tokens beat acoustic-plus-semantic tokens. The best AFC configuration uses pre-training, excludes the semantic layer (last 7 layers), and applies ℓ1 selection, reaching 54.8, versus 53.9 when semantic tokens are retained. The authors interpret this as the LLM handling semantics through the text.

• Even random audio token selection helps. The paper reports that using audio tokens always improves results, including when tokens are selected at random, suggesting complementary information is always present in the audio tokens.

• Pre-training is not strictly necessary, but selection is. Without pre-training, the lasso-based filtering lifts performance from 48.7 to 52.7; without filtering, the model reaches a low F1 of .487 with a standard deviation 5 times higher than other configurations.

• Sequence lengths become comparable. The average textual token sequence length is 31.52 and the average speech token sequence length is 35.52 — roughly the same order. The number of selected tokens is 73 for the 7-layer configuration and 80 for the full 8-layer configuration, both under 1% of the initial vocabulary.

• Token redundancy is consistent with prior work. The paper connects its result to Gibier et al. (2025), who found that retaining as few as 25% of audio tokens preserves performance on captioning and question-answering benchmarks, and argues that the vast majority of tokens from fine-grained speech tokenizers can be discarded.

• Fine-grained error patterns. The unimodal model confuses Appeal to Emotion with Ad Hominem when the sentence contains "He" or "She"; the multimodal model reduces confusion between Appeal to Authority and Appeal to Emotion, which the authors associate with voice characteristics (assertive and monotonous for authority, more prosodic with tremor for emotion). On MOSEI, the multimodal model resolves cases where the text is neutral but the voice is happy or sad.

Methodology in Plain English

The method proceeds in three steps.

Step 1 — Token selection. Each audio signal is passed through SpeechTokenizer, an acoustic-semantic encoder-decoder based on Residual Vector Quantization with 8 layers. The first layer captures semantic content; the remaining layers capture paralinguistic content such as prosody, rhythm, and voice quality. Because the first layer's semantic content is already present in the text, the authors optionally drop it and keep the last 7 layers. Each layer has a vocabulary of 1024, giving 8,196 distinct tokens in the initial vocabulary, and one token per layer is emitted every 20 milliseconds. Text and audio are then both converted into Bag-of-Words histograms — for audio, each layer contributes a 1024-dimensional histogram, producing a 1024 × L vector per sample — and concatenated. An ℓ1-regularized logistic regression is trained to predict the task label; the penalty drives irrelevant audio coefficients to zero. The surviving token types form a reduced vocabulary that is less than 1% of the original, and the filter is fit on the training split only.

Step 2 — Audio token embedding pre-training. The reduced vocabulary is added to the LLM by creating new, randomly initialized embeddings of the same dimensionality as the existing token embeddings, with no extra projection layers. A causal language modeling cross-entropy loss adapts these embeddings while every original LLM parameter stays frozen, preserving the pre-trained linguistic representations.

Step 3 — Downstream fine-tuning. A single linear classification head maps the final hidden representation of the last token to the logits for the task's classes (for example, 2 for AFD: "No Fallacy" and "Fallacy"). Only the LoRA adapter parameters and the classification head are updated; the rest of the model remains frozen.

Experimental protocol. Audio token embeddings are trained on MM-USED, fallacy tasks use MM-USED-Fallacy, and sentiment and emotion use CMU-MOSEI. Results on fallacy tasks are reported on the official test set of the MM-ArgFallacy2025 shared task. Metrics are macro-average F1 for AFC and F1 of the positive class for AFD, averaged over 5 seeds with probability-based bagging. Baselines include a text-only Llama3-3B, a Llama3-3B augmented with a 2048-dimensional BYOL-A v2 audio representation projected into the embedding space, Qwen2-Audio-7B in both 3-shot in-context learning and supervised fine-tuning settings, and encoder-based shared-task systems. Experiments used torch 2.1.2 and transformers 4.46.3 on an Nvidia RTX-A6000 GPU with CUDA 12.2.

Why This Matters

Impact on research. The paper offers a counterexample to two common assumptions: that audio is unhelpful or even counterproductive for argumentative fallacy tasks, and that effective speech integration requires massive multi-stage pre-training. It reframes the sequence-length mismatch problem as a vocabulary-selection problem, complementing dynamic, instance-level inference-time pruning with a complementary, task-level approach. It also reports that selection helps even when tokens are chosen randomly, which raises questions about how much fine-grained token identity actually matters.

Real-world applications.

• Detecting fallacious or manipulative rhetoric in political debates, campaign speeches, and public consultations, where tone of voice carries signal that transcripts miss.
• Speech-based emotion and sentiment monitoring, for example in customer service calls, where neutral wording can mask a happy or sad speaker.
• Content moderation and misinformation triage over audio and video platforms, using a lightweight classifier rather than a large multimodal model.
• Assistive and accessibility tooling that augments transcripts with paralinguistic cues without requiring a full SpeechLLM stack.

Industry relevance. The method requires no multi-stage pre-training and can be applied to any pre-trained LLM at minimal computational cost, which matters for teams that have a small task-specific dataset and cannot afford to train or serve a large audio-language model. The paper notes that using all audio tokens without selection is computationally impractical (approximately 4,000 tokens for 10 seconds of speech) and did not yield positive results in preliminary experiments, so the reduction to roughly 73 to 80 token types is not just an efficiency gain but a prerequisite for usable results.

Future Directions

• Broader validation. The paper acknowledges that its evaluation covers four classification tasks across two datasets (MM-USED-Fallacy and CMU-MOSEI), with a single LLM backbone (LLaMA-3.2-3B) and a single speech tokenizer (SpeechTokenizer), and states that additional datasets, languages, task types, architectures, and backbones would strengthen generalizability.
• Extension to generation. The authors suggest moving beyond classification to generation tasks, citing Kim et al. (2025) on gestures and facial expressions as a possible direction.
• Alternative training and regularization configurations. The paper proposes evaluating the method without fine-tuning the audio embeddings to assess their intrinsic generalization, investigating strategies such as updating all parameters instead of only the embedding layers, and using a lower lasso regularization to retain more speech tokens and empirically find the optimal size.
• Interpretability and comparison. The authors call for a correlational study between learned speech tokens and expert acoustic features such as pitch, intensity, spectral shape, or glottal flow features, a deeper attention-flow analysis of how acoustic information is integrated across layers, and inclusion of other SpeechLLM baselines such as Audio-Flamingo2-3B or EmoSLLM and other textual LLMs.

Target Audience

This paper is most useful for NLP and speech researchers working on multimodal fusion, efficient adaptation of pre-trained language models, and argument mining or affective computing. It also suits applied machine learning engineers who need a low-cost way to add speech cues to an existing text model, and graduate students looking for a clear example of task-specific feature selection combined with parameter-efficient fine-tuning. Readers should already be comfortable with tokenization, LoRA, and standard classification evaluation to follow the technical details.

Authors’ abstract

This paper presents a simple method that allows to easily enhance textual pre-trained large language models with speech information, when fine-tuned for a specific classification task. A classical issue with the fusion of many embeddings from audio with text is the large length of the audio sequence compared to the text one. Our method benefits from an existing speech tokenizer trained for Audio Speech Recognition that output long sequences of tokens from a large vocabulary, making it difficult to integrate it at low cost in a large language model. By applying a simple lasso-based feature selection on multimodal Bag-of-Words representation, we retain only the most important audio tokens for the task, and adapt the language model to them with a self-supervised language modeling objective, before fine-tuning it on the downstream task. We show this helps to improve the performances compared to an unimodal model, to a bigger SpeechLM or to integrating audio via a learned representation. We demonstrate its effectiveness on Argumentative Fallacy Detection and Classification tasks where audio was previously believed counterproductive, and affective computing tasks on a widely-used dataset. We also provide an in-depth analysis of the method, showing that even a random audio token selection helps enhancing the unimodal model. Our code is available [online](https://github.com/salocinc/EMNLP26SpeechTokLLM/).