Reusing Latent Speech Representations for Query-Conditioned Topic Localization in Transcripts

AI summary

Overview

• Research area: Natural language processing / spoken language processing — specifically query-conditioned span localization inside speech transcripts, combining automatic speech recognition (ASR) representations with text embeddings.
• Technical level: Intermediate. The paper assumes familiarity with ASR encoder architectures (Whisper, Canary), sentence embeddings, and span-prediction models (VSLNet, QMSum-Pointer), but the core idea is explained conceptually.
• Scope: The paper proposes an audio-text framework that reuses frozen ASR encoder states as sentence-level representations and fuses them with textual sentence embeddings to improve query-conditioned topic localization, evaluated with two locators, two ASR models, and four datasets.

What This Paper Is About

Long speech transcripts are expensive inputs for downstream NLP systems such as retrieval, question answering, and summarization, and they often contain context irrelevant to a user's information need. Fixed-size retrieval windows ignore topical structure, and conventional topic segmentation produces query-independent segments that may not match the user's query. This paper studies query-conditioned topic localization: given a topic title as query and a transcript, predict the contiguous sentence span that best addresses that query — and it does so by reusing the internal encoder states of the ASR model that already transcribed the audio, rather than running a separate audio encoder.

Key Contributions

1. Method: An audio-text framework for query-conditioned topic localization that reuses ASR encoder states as sentence-level acoustic representations, pooled from aligned time intervals and fused with textual sentence embeddings. The localization models themselves are unchanged; only the input representation is augmented.
2. Experiments: Evaluation with two span locators (VSLNet and QMSum-Pointer), two ASR models (Whisper large-v3 and Canary canary-1b-v2), and four public datasets (Euronews, YTSeg, AMI, Videoaula) spanning different domains and languages.
3. Analysis: Ablations on encoder layers, temporal pooling and fusion strategies, text encoder choice, cross-dataset transfer against retrieval baselines, linear probing of boundary roles in embeddings, and a qualitative single-author inspection of acoustic boundary cues.
4. Resources: Release of evaluation code and checkpoints for the localization models at https://github.com/steffrs/speech-topic-localization, plus a script for extracting Euro­news chapter titles from YouTube video descriptions.

Main Findings

• Text+audio beats text-only: Across both datasets and both localizers, the fused T+A representation (gated fusion) gives the best results. For VSLNet on Euronews, EM rises from 45.26 to 69.11, and the same trend holds for all R@1 thresholds. Gains are larger on Euronews than on YTSeg.
• Gains concentrate on strict boundary matching: The largest relative gains appear for Exact Match and R@1 at IoU ≥ 0.7, shrinking as the IoU threshold relaxes — suggesting ASR-derived representations help locate exact start and end positions rather than just the broader relevant region.
• Audio-only performance is dataset-dependent: On YTSeg, text-only clearly outperforms audio-only (e.g., VSLNet EM 13.88 vs. 5.97). On Euronews, audio-only is competitive and often stronger than text-only (VSLNet EM 58.28 vs. 45.26), which the authors link to the structured production style of news videos.
• Both ASR backbones help: Adding ASR-derived representations increases EM by 23.85 points with Whisper and 18.23 points with Canary on the Euronews VSLNet ablation, so the benefit is not specific to one ASR model.
• Reuse beats a dedicated audio encoder: A dedicated wav2vec 2.0 audio encoder reaches 67.00 EM / 77.73 R@1 (IoU ≥ 0.5) with VSLNet on Euronews, below the 69.11 / 79.66 obtained from reused Whisper states, while requiring an extra encoder pass.
• Pooling and fusion differences are small but consistent: Mean^ext pooling is best for Whisper and remains competitive for Canary; for Canary, Boundary gives slightly higher EM while Mean+Std gives slightly higher R@1. Gated fusion gives the best EM for both ASR models.
• Deeper encoder layers work better: Performance generally improves in deeper Whisper encoder layers, with the final layer (layer 32) giving the best result; the authors use layer 32 for remaining experiments.
• Benefits extend across text encoders: With E5-large and Qwen3-Embedding-4B added to MiniLM, T+A improves EM in all 12 comparisons and R@1 in 10 of 12, although it does not improve every metric.
• Cross-dataset transfer depends on domain: Training on the combined Euronews and YTSeg train sets, VSLNet T+A achieves the highest R@1 on all four datasets (AMI 15.63, Videoaula 21.22, Euronews 83.39, YTSeg 49.74) and the highest EM on three of four. Gains are clearest on structured or semi-structured content; AMI, with overlapping spontaneous meeting speech, remains difficult for all models.
• Localization beats retrieval over fixed windows or predicted segments: Fixed-size windows score near-zero EM on AMI (0.00–0.05) and Videoaula (0.13–0.27), and predicted-segment retrieval also trails the localizers, supporting fine-grained span prediction over ranking segments.
• ASR-derived embeddings encode boundary roles: Linear probes for start, end, and boundary classification show ASR-derived audio embeddings outperform text embeddings on AUC-ROC and F1 across all datasets and tasks — for example, boundary AUC on Euronews is 95.05 for audio vs. 71.69 for text; on AMI the gap is small (68.75 vs. 65.76).
• Cue patterns explain domain differences: A qualitative inspection of 20 randomly sampled recordings per dataset found the clearest boundary cues in Euronews (production cues such as swoosh sounds and background changes, pauses, speaker changes), weaker or mixed cues in YTSeg and Videoaula, and few boundary-specific cues in AMI.
• Computation trade-off: Text+audio increases offline FLOPs (pooling and fusion of ASR states), but for a given locator T and T+A have the same query-time computation and stored context size, since both reduce to a precomputed context matrix.

Methodology in Plain English

The pipeline has two stages. Offline, each recording is transcribed once. Audio is segmented with voice activity detection and chunked into at most 30-second pieces for ASR. The transcript is split into sentences with Punkt, and each sentence is linked to its aligned ASR encoder frames via timestamps. Those frames are pooled into one acoustic vector per sentence using one of four strategies: averaging all frames (Mean), averaging frames from an interval extended by 1.28 seconds on both sides (Mean^ext), concatenating the mean with the per-dimension standard deviation (Mean+Std), or averaging two 2.56-second windows at the sentence start and end (Boundary). Text and audio embeddings are each projected into a shared hidden size of 128 and combined by summation, concatenation, or gated fusion. The combined context is encoded once per recording with a QANet-style encoder (convolutions plus multi-head self-attention), producing a query-independent context matrix that can be reused for many queries.

Online, the query text is encoded at token level by the same text embedding model, projected, and passed through the same encoder. Two unchanged span locators consume the context and query: VSLNet (context-query attention with a span head and query-guided highlighting) and a QMSum-style pointer model (a start pointer over sentence positions, then an end pointer conditioned on the selected start). Evaluation uses Exact Match and R@1 at IoU thresholds of 0.3, 0.5, and 0.7.

Training uses AdamW with weight decay 0.01, an initial learning rate of 3×10⁻⁴, a cosine schedule with linear warmup ratio 0.05, batch size 32, up to 20 epochs, and early stopping on the dev set with patience 2. Only the projection, fusion, context/query encoding, and localization layers are trained; all ASR and text embedding models are frozen. Separate models are trained for the text-only, audio-only, and text+audio settings. The main text encoder is paraphrase-multilingual-MiniLM-L12-v2, with multilingual-e5-large and Qwen3-Embedding-4B used in an ablation.

Why This Matters

Impact on research. The paper shows that the encoder states produced during transcription — normally discarded — are a usable and effective signal for downstream query-conditioned localization, without any extra audio encoder at query time. It connects three lines of work (span localization, ASR representation reuse, and audio cues for topic segmentation) and provides evidence that acoustic information particularly sharpens boundary precision rather than coarse region selection. The cross-dataset results also delineate where the approach does and does not transfer, which is useful framing for anyone building on spoken-topic datasets.

Real-world applications:

• Searching news bulletins or podcasts by topic title and jumping directly to the relevant sentence range.
• Meeting and lecture navigation, where users want the part of a recording that addresses a specific topic.
• Two-stage retrieval pipelines where a recording is fetched first and within-document localization narrows the context passed to a summarizer or question-answering model.
• Efficient long-transcript preprocessing: reducing the amount of text sent to expensive downstream models by selecting a focused, query-relevant span.

Industry relevance. The setup matches a common production pattern: transcribe once, index offline, and serve many queries cheaply. Because text-only and text+audio configurations share the same stored context size and query-time computation for a given locator, and because QMSum-Pointer is cheaper online while VSLNet is more accurate, the paper offers a concrete accuracy-versus-cost choice for teams integrating speech content into search or summarization products. The authors also flag privacy considerations: ASR encoder states may carry speaker identity, accent, speaking style, emotion, or recording conditions, and prior work has shown self-supervised speech models can leak membership information, so such embeddings should be stored securely and not published when recordings contain private speech.

Future Directions

• End-to-end retrieval: The current method assumes the relevant recording is already known and performs only intra-document localization; combining document retrieval with within-document localization into a single pipeline is left to future work.
• Robustness to realistic queries: Topic titles are concise, normalized queries. A controlled perturbation experiment in Appendix E replaces titles with natural-language templates or single-typo variants, and all perturbations reduce performance, with the largest degradation for typo queries — so handling noisier, more ambiguous user queries remains open.
• Multi-span localization: The models always output one contiguous span, matching the annotation structure, but some information needs require multiple spans or evidence spread across a recording.
• Better generalization to spontaneous speech: Results on AMI are much lower for all models, and the paper notes that larger, more diverse benchmarks with audio, boundaries, and titles are needed to assess cross-domain generalization more completely.
• Multimodal extension: Visual information is not used, partly because visual streams are not consistently available across the datasets considered; incorporating video representations is suggested as an interesting direction.

Target Audience

Researchers and practitioners working on spoken language processing, long-document retrieval, and query-conditioned span localization will benefit most, particularly those interested in reusing frozen ASR representations rather than training dedicated audio encoders. It is also relevant to engineers building search, summarization, or question-answering systems over podcasts, news bulletins, lectures, and meetings, and to readers interested in the limits of speech-based augmentation on spontaneous multi-speaker audio.

Authors’ abstract

Long transcripts are costly inputs for downstream NLP systems and often contain irrelevant context. We study query-conditioned topic localization: predicting the sentence span in a transcript that best addresses a topic-title query. To improve span localization, we reuse ASR encoder states as sentence-level representations and fuse them with textual embeddings. This lets lightweight span locators exploit speech information without running a separate audio encoder. Experiments on two public datasets show consistent gains over text-only baselines, especially under strict boundary-matching criteria. Cross-dataset experiments further indicate that the benefits are strongest for structured or semi-structured speech, while gains on spontaneous speech are limited and mixed.