Encode Once, Decode Never: Reusing Audio LM Internals for Efficient Temporal Localization

AI summary

Overview

Research area: Audio language models and efficient temporal localization (word alignment, speaker diarization, audio event localization).

Technical level: Intermediate to Advanced. The core idea is intuitive, but the paper introduces a novel probabilistic loss (inhomogeneous Poisson process) whose derivation uses point-process theory, hazard rates, and the time-rescaling theorem.

Scope: The paper argues that audio LMs should stop generating timestamps as text tokens and instead read timestamps directly off the frame-level audio representations they already compute, showing this is faster, more robust to out-of-distribution audio lengths, and at least as accurate.

What This Paper Is About

Audio language models encode audio into rich frame-level representations, but when asked "when did the dog bark?" they typically answer by generating timestamps as text tokens autoregressively. That decoding process is slow (one token at a time, thousands of tokens for a long podcast), and it hallucinates badly when asked about audio longer than what was seen in training. This paper introduces internal frame-level reuse: attaching a small prediction head to the LM's own decoder outputs so the model emits a probability (or event-intensity) value per audio frame, from which timestamps are extracted with simple deterministic code—no token generation, no external tools.

Key Contributions

1. Internal frame-level reuse: A method that trains an audio LM to reuse its own decoder frame-level representations (including states normally discarded during token generation) for localization, bypassing autoregressive decoding entirely.
2. A novel inhomogeneous Poisson process (IHP) loss: The ground-truth timestamps are modeled as arrival times of a continuous-time point process whose time-varying intensity is predicted from the LM's frame representations. This is claimed to be the first application of point processes to temporal localization with audio LMs.
3. A systematic comparison against token-based finetuning: Identical training conditions across three tasks (word localization, speaker diarization, audio event localization), two model sizes, and two frame-level objectives (binary classifier vs. Poisson), plus a zero-shot baseline sweep over six open and proprietary audio LMs.
4. Evidence of efficiency, robustness, and quality simultaneously: Demonstrated speedups (over 50×), length generalization where token models collapse, and equal-or-better accuracy/MAD across tasks, with an open code release.

Main Findings

• Speedups of 50× or more: Because localization is a single parallel pass over audio frames rather than sequential decoding, Poisson-based inference is over 50× faster than token generation at small batch sizes (up to 60×), and at least about 3× faster at large batch sizes after averaging. Gains grow with the number of timestamps required.

• Token-based models collapse out of distribution: When trained on timestamps in 0–4 seconds and tested on 4–8 seconds, the token model drops to 0.9% / 1.5% accuracy at 20 ms / 40 ms tolerances with a MAD of 2.35 seconds. Training on 0–16 s and testing on 16–20 s produces 0.0% accuracy for the token model.

• Frame-level reuse generalizes to unseen audio lengths: The Poisson loss maintains over 85% accuracy at the 40 ms threshold on out-of-distribution timestamp ranges, with MADs around 0.01–0.07 seconds, because predictions are read from audio frames rather than mimicked from memorized timestamp text.

• Poisson loss matches or beats token baselines on every task: On LibriSpeech word alignment (Qwen2.5-Omni 7B), Poisson reaches 93.3% at 20 ms / 98.1% at 40 ms versus 93.2% / 97.1% for tokens, with lower MAD (0.01 vs. 0.03 s). On LibriCount speaker diarization, 65.9% / 76.5% versus 62.1% / 75.3%. On AudioSet event localization, 42.3% / 58.5% versus 40.1% / 56.9%.

• Binary frame loss is simple but weaker: The reweighted binary cross-entropy head is competitive at loose tolerances (96%+ at 40 ms on LibriSpeech) but consistently trails the Poisson loss on 20 ms accuracy and MAD across tasks, motivating the specialized temporal loss.

• Multi-timestamp prediction works without a token bottleneck: When targets include every word's start time in a transcript, Poisson inference stays comparable to token inference (93.1% vs. 93.2% at 20 ms overall) while retaining the large speed advantage.

• Combining losses does not help: Interpolating token loss and Poisson loss (L_tokens + 0.05 L_Poisson) yields no consistent improvement over the Poisson loss alone.

• Zero-shot temporal localization is essentially unusable: All six tested audio LMs—Audio Flamingo 3, Voxtral 3B/24B, Qwen2.5-Omni 3B/7B, Gemini 2.5 Flash, and GPT-4o Audio—score poorly without task-specific finetuning (Gemini 2.5 Flash is the best at just 7.7% / 13.8% on LibriSpeech).

• No catastrophic forgetting: Full-model finetuning (no LoRA adapters) showed no degradation on the MMAU audio understanding benchmark.

Methodology in Plain English

The audio LM already has an encoder that turns a waveform into a sequence of frame vectors, and a decoder that, conditioned on a text prompt like "When did the dog bark?", produces a matching sequence of frame-level states. Instead of asking the decoder to spell out a timestamp as text, the authors bolt a tiny projection head onto those decoder states—one output per 40 ms frame.

Two ways of training that head are compared. The binary version treats it as a per-frame yes/no classification: is the queried event happening in this frame? Because positive frames are extremely rare, positive examples are upweighted. At inference, the top-k highest-probability frames are taken as the answer.

The Poisson version treats ground-truth timestamps as arrivals of an event process with a time-varying rate. Each frame's representation is projected to a non-negative "intensity" value, and the model is trained to maximize the negative log-likelihood of the true arrival times given the total accumulated intensity. This means the model is not scoring frames independently but learning a probability density over continuous time. At inference, the most likely timestamp is found by evaluating the density at frame knots and the distribution's mode, using the time-rescaling theorem to convert the complex process into a standard one.

Evaluation spans three tasks: word timestamps on LibriSpeech (ground truth from the Montreal Forced Aligner), speaker start times on the synthetic LibriCount cocktail-party set, and acoustic event timestamps on the human-sounds subset of AudioSet. Accuracy is measured as whether the prediction is within 20/40/100 ms of ground truth, alongside mean absolute deviation.

Why This Matters

Impact on research: The paper challenges a default assumption in multimodal modeling—that anything expressible as text should be generated as text. It shows that for a task fundamentally about pointing at positions in an input, reusing internal representations beats generation on speed, robustness, and accuracy simultaneously. It also introduces point processes as a principled temporal loss for audio LMs, connecting modern audio modeling back to the frame-level objectives (CTC, RNN-T) that long defined speech recognition.

Real-world applications:

• Podcast, video, and film editing: Instant speaker turns, laughter, applause, and music-boundary markers on hour-long files without tens of thousands of sequential decoding steps.
• Meeting and call analytics: Fast diarization and event tagging for compliance review, summarization, or speaker-attributed transcripts.
• Accessibility and subtitles: Precise word-level timing for captions and karaoke-style highlighting, including for audio longer than any training example.
• Audio search and archival indexing: Making large media archives queryable by timestamped sound events rather than by transcribed speech alone.

Industry relevance: Any product built on transcription, diarization, or audio understanding pays the autoregressive cost per timestamp. A 3–60× latency reduction with better OOD behavior is directly relevant to ASR vendors, media-production tools, contact-center analytics, and any team deploying audio LMs where audio length varies widely in production.

Future Directions

• Video temporal localization: The authors explicitly flag extending frame-level reuse to moment retrieval and event grounding in video, where the same internal-representation argument applies.
• Removing the known-event-count assumption: The method requires knowing k (how many timestamps to extract). The paper suggests the LM can supply this cheaply, but a fully joint count-and-localize objective remains open.
• Handling overlapping and hierarchical events: The Poisson formulation currently models a single event stream; extending it to simultaneous events (overlapping speech, layered soundscapes) or nested structures is unresolved.
• Hybrid and reinforcement-learning objectives: Since interpolating token and Poisson losses did not help, other combinations—such as interval-overlap rewards used in video work—may be worth testing.
• Streaming and latency-critical settings: All results are on complete recordings; online localization from partially observed audio is a natural but untested extension.

Target Audience

Speech and audio ML researchers working on alignment, diarization, and event detection; engineers building production transcription or media-indexing systems who care about inference latency and robustness to variable audio length; and graduate students interested in how multimodal LMs can internalize localization tasks rather than outsourcing them to tools or external models. Readers should be comfortable with language model architectures and basic probability; the Poisson-process derivation in Section 2.2 rewards familiarity with point processes but is self-contained enough to follow.

Authors’ abstract

Audio language models process input audio into rich frame-level representations, but the standard approach to temporal localization generates timestamps as sequences of text tokens, which discards the frame-level representations in favor of autoregressive decoding. However, generating timestamps as tokens is slow and not parallelizable, and tends to hallucinate when producing timestamps outside the training distribution. We propose internal frame-level reuse, a method that trains audio LMs to reuse their own internal audio representations for temporal localization directly, bypassing token generation altogether. We introduce a lightweight prediction head trained via different frame-level objectives: a binary frame classifier and a novel inhomogeneous Poisson process (IHP) loss that models temporal event intensity. Across word localization, speaker diarization, and event localization tasks, our approach can achieve a >50x inference speedup over token-based generation and demonstrates robust length generalization, maintaining high accuracy on out-of-distribution audio durations where token-based models collapse completely. We find that reusing audio frame-level representations yields localization accuracies comparable to (and often better than) finetuned token-based baselines.