Extending Audio Context for Long-Form Understanding in Large Audio-Language Models

AI summary

Overview

• Research area: Natural Language Processing / multimodal large language models — specifically long-context extension for Large Audio-Language Models (LALMs) that use Rotary Positional Encoding (RoPE).
• Technical level: Advanced. The paper assumes familiarity with RoPE, positional interpolation (PI), YaRN, LoRA fine-tuning, and attention temperature scaling.
• Scope: The paper studies training-free and fine-tuning-based positional extension methods that expand the usable audio context of existing unified-input-space LALMs (SALMONN and Qwen2-Audio) beyond the roughly 30 seconds of audio they were trained on, up to 10-minute inputs.

What This Paper Is About

LALMs such as SALMONN and Qwen2-Audio pair an audio encoder with a text LLM backbone, but they are typically trained on audio segments of 30 seconds or less, so they generalize poorly to longer audio even when their text backbone supports long contexts. Prior work has applied context-extension methods such as Positional Interpolation and YaRN to unimodal LLMs, but their application to LALMs is unexplored. The paper's goal is to extend the audio context of existing LALMs without retraining a new architecture from scratch, by manipulating only the positional encoding of audio tokens, and then to make that extension generalize during training.

Key Contributions

1. Partial YaRN — a training-free, modality-decoupled audio-context extension for LALMs that stretches only the audio token positions while leaving text positions unaltered, in order to preserve the base LLM's text capabilities.
2. Virtual Longform Audio Training (VLAT) — a fine-tuning strategy that repurposes Partial YaRN as a training-time positional augmentation, simulating a wide range of audio lengths so the model generalizes to inputs longer than those seen in training.
3. A comparative study between existing whole-context extension methods designed for unimodal LLMs (Whole PI, Whole YaRN) and the proposed modality-decoupled methods, covering both training-free and fine-tuning settings.
4. A custom long-audio dataset, YODAS2-MCQA, built by segmenting English YODAS2 audio into 1, 2, 5, and 10 minute non-overlapping segments and generating MCQA pairs with Gemini 2.0 Flash; each duration's test set has 750 QA pairs. Code and dataset are released at https://github.com/yophis/partial-yarn.

Main Findings

• No universally superior extension method. Across training-free settings, the best choice between whole-context and modality-decoupled extension is model- and length-dependent. On SALMONN stretched from the 30s window, Partial YaRN scored 57.35% at 1 min and 32.93% at 10 mins, while Whole YaRN scored better at 5 mins (47.20% vs 38.53%).
• Base models retain accuracy up to 2 mins. Both SALMONN and Qwen2-Audio performed consistently up to 2-minute audio (SALMONN Vanilla: 49.01% at 1 min, 46.13% at 2 mins; Qwen2-Audio Vanilla: 72.71% and 75.87%), suggesting their innate audio context is longer than 30s, possibly from multi-audio training.
• Anchoring the extension at the observed context matters more than the method. Stretching from the empirically observed 2-minute window instead of 30s substantially improved long-audio results. On Qwen2-Audio, Partial YaRN at 10 mins rose from 28.53% to 48.00%, which the authors describe as an absolute improvement of almost 20% and a final performance 26% higher than the Vanilla baseline (22.00%).
• Fine-tuning with an extension method yields large gains. Using single-epoch LoRA of rank 8 on Qwen2-Audio, the qkvo setting produced 83.47% (Whole YaRN) and 83.07% (Partial PI) at 10 mins versus 64.93% for Vanilla fine-tuning — roughly a 19% absolute gain.
• VLAT generalizes to unseen lengths. Training on 2-minute YODAS2-MCQA with VLAT raised 10-minute accuracy from 32.76% (vanilla fine-tuning, vanilla inference) to 75.11%.
• VLAT combines with inference-time extension. With Partial PI applied at inference, the VLAT-trained model reached 81.73% at 10 mins versus 75.56% for the vanilla-trained counterpart — the highest overall 10-minute result reported in that table.
• Both components of YaRN matter. Ablations show that removing frequency grouping or attention temperature scaling causes large drops, particularly at high extension ratios. For Qwen2-Audio's Partial YaRN, 10-minute accuracy fell from 28.53% to 28.13% without frequency grouping and to 18.80% without attention temperature scaling; the corresponding Partial PI figure was 18.40%.
• Two-group frequency partitioning beats three-group. For Qwen2-Audio, the proposed 2-group Partial YaRN scored 48.53% at 5 mins and 28.53% at 10 mins, versus 20.00% and 16.53% for a 3-group version — the paper reports accuracy gaps of 28.53% and 12.00% respectively.
• Reference points. Gemini 2.0 Flash scored 94.70 / 95.86 / 92.80 / 89.87 and GPT-4o scored 89.10 / 92.91 / 90.40 / 83.65 across 1, 2, 5, and 10 minutes.

Methodology in Plain English

RoPE encodes position by rotating query and key vectors at an angle that depends on the token's position. Because LALMs were trained only on short audio, longer audio pushes audio tokens into position ranges the model has never seen. The authors' hypothesis is that the model already understands audio and text well — the real bottleneck is unfamiliarity with out-of-range audio positions.

Partial YaRN exploits this by remapping only the audio region of the sequence into a familiar range while leaving text positions untouched. Concretely, the audio positional window [p, p + L_audio) is stretched to a new length L_audio', and the RoPE dimensions are split into two groups: low-frequency dimensions are interpolated (stretched) so they stably cover the longer audio, while high-frequency dimensions are extrapolated to preserve local distances and high-frequency detail. An attention temperature is folded into the magnitudes of the RoPE rotation matrices rather than added as a separate softmax term, which also smoothly handles attention between unaltered text and scaled audio regions. With default hyperparameters (cutoff index 0, temperature 1.0), the audio-only extension reduces to plain positional interpolation, which the authors call Partial PI.

VLAT takes Partial YaRN and uses it during training as a positional augmentation. For each training sample of actual length L_data, a virtual source length L_virt is drawn as the model's default audio context length multiplied by a factor sampled from [1, 5, 10, 15, 20, 25]x, and Partial YaRN is used to stretch or compress that virtual window down to L_data. Unlike prior random position-index downsampling, VLAT uses interpolation to create a dense, continuous space of positions and is bidirectional, teaching through both compressed and stretched contexts.

For experiments, long audio is segmented into non-overlapping 30-second chunks encoded independently by the Whisper encoder. SALMONN produces 88 audio tokens per 30s chunk (with Q-Former), while Qwen2-Audio produces 750 tokens (with 2x downsampling). Baselines are Vanilla, Whole PI, and Whole YaRN. Fine-tuning uses LoRA on Qwen2-Audio with either q-only or qkvo adaptation; VLAT runs 10-epoch qkvo LoRA and results are averaged over 3 random seeds.

Why This Matters

• For research: The work shows that long-audio capability can be improved by manipulating the positional space of existing LALMs rather than designing new architectures or curating large long-audio datasets, and it introduces modality-decoupled positional extension as a distinct alternative to whole-context extension.
• For practitioners: Partial YaRN is described as a lightweight, architecture-agnostic "drop-in" enhancement, and VLAT is presented as a data-efficient fine-tuning path — both usable on already-released open-weights models.

Real-world applications implied by the work:

• Long-form meeting, lecture, or podcast transcription and question answering over 10-minute-plus recordings.
• Audio content moderation or compliance review, where an entire recording must be assessed, not just a 30-second excerpt.
• Assistive and accessibility tools that summarize or answer questions about long recordings.
• Spoken-dialogue and call-center analytics, where context spans far beyond the short training window.

Industry relevance: The paper notes that concurrent long-audio systems such as Audio Flamingo 3 (up to 10 mins), Voxtral (up to 40 mins, with a 32k-context LLM backbone), FastLongSpeech, and Qwen3-Omni (up to 40 mins) achieve their results through new architectures, large long-audio datasets, and extensive training. This work positions itself as complementary and cheaper, extending existing models instead of requiring costly data acquisition and retraining.

Future Directions

• Apply Partial YaRN and VLAT to vision models, given the structural similarity between long video and long audio — the authors explicitly suggest the principles are likely applicable to the video domain.
• Explore controllable generation, such as multi-scale image synthesis or finer-grained decoupling for non-uniform speed control throughout audio or video generation.
• Broaden evaluation beyond multiple-choice QA to open-ended generative and language-modeling tasks and to benchmarks used in other long-audio work, which would more fully test whether preserving text positions matters.
• Reduce the hyperparameter tuning burden, since Partial YaRN requires tuning a cutoff dimension index and an attention temperature rather than relying on YaRN's predefined cutoffs and closed-form temperature; the paper also reports that VLAT performance can be further enhanced by using a narrower range for the virtual length L_virt.

Target Audience

Researchers and engineers working on multimodal LLMs, long-context extension, and positional encoding will get the most from this paper, particularly those who already understand RoPE, YaRN, and LoRA fine-tuning. It is also useful for practitioners building long-audio applications who want a cheap way to extend existing open-weights LALMs, and for anyone tracking the tradeoffs between whole-context and modality-specific context extension. The multiple-choice-only evaluation, reliance on the synthetic YODAS2-MCQA dataset, and the two-hyperparameter tuning requirement are explicit limitations to weigh before adopting the methods.

Authors’ abstract

Large Audio-Language Models (LALMs) are often constrained by short audio context windows, even when their text backbones support long contexts, limiting long-form audio understanding. Prior work has introduced context-extension methods (e.g. YaRN) on unimodal LLMs, yet their application to LALMs remains unexplored. First, building on RoPE-based context extension, we introduce Partial YaRN, a training-free, modality-decoupled extension method that modifies only audio token positions, leaving text positions intact to preserve the base LLM's text capabilities. Second, we propose Virtual Longform Audio Training (VLAT), a training strategy that extends Partial YaRN into a training-time positional augmentation. VLAT simulates diverse audio lengths during training, enabling generalization to inputs far longer than those seen in training. Our experiments on SALMONN and Qwen2-Audio confirm that Partial YaRN outperforms the original models across wide range of settings, and VLAT provides substantial performance improvement on long audio of unseen lengths.