Towards Multi-Level Transcript Segmentation: LoRA Fine-Tuning for Table-of-Contents Generation

AI summary

Overview

• Research area: Natural Language Processing and speech technology — specifically hierarchical topic segmentation of spoken-language transcripts and automatic table-of-contents (ToC) generation.
• Technical level: Intermediate. The core ideas (segmentation, tables of contents, prompting vs. fine-tuning) are accessible, but the paper assumes familiarity with LLM fine-tuning concepts such as LoRA, quantization, and boundary-similarity metrics.
• Scope: The paper proposes generating multi-level tables of contents from speech transcripts using LLMs (zero-shot prompting vs. LoRA fine-tuning, with optional pause-duration cues) and introduces a metric that evaluates all hierarchy levels at once.

What This Paper Is About

Automatically transcribed speech — lectures, meetings, seminars — has no chapters or headings, so finding specific content in it is hard and downstream systems like retrieval-augmented generation have nothing meaningful to index against. Most prior work segments transcripts along a single flat timeline, but real topics nest: a broad theme like "Project Requirements" contains subtopics like "User Interface" and "Testing Strategies." The paper's goal is to make an LLM produce a nested, multi-level table of contents directly from a transcript, and to measure how well that hierarchy matches human annotations.

Key Contributions

1. A new formulation of hierarchical topic segmentation as LLM-based table-of-contents generation, where the model outputs a labelled outline (dotted numbering such as 2.2.1) rather than only boundary positions.
2. Demonstration that LoRA-fine-tuned LLMs outperform established segmentation baselines (unsupervised TextTiling-BERT, supervised MiniSeg and CS-BERT, and prompt-based SegmentLLM) on three datasets, covering English, Portuguese, and German.
3. A hierarchical evaluation wrapper that aligns reference levels to hypothesis levels under a non-decreasing constraint via dynamic programming, collapsing a multi-level comparison into a single scalar score instead of forcing evaluation at one flat level.
4. An exploration of high-level acoustic cues — inter-sentence pause durations injected as text annotations — showing they help fine-tuned models but hurt zero-shot ones.

Main Findings

• Fine-tuned ToC models lead across the board: ToC-Nemo + Pause achieved the best fine-tuned results on all three datasets: F1 30.34 / B 24.81 on AMI, F1 67.34 / B 55.18 on Videoaula, and F1 58.89 / B 49.77 on LectureDE. For comparison, CS-BERT scored 20.50 / 15.77 on AMI and 47.77 / 34.22 on Videoaula.
• ToC beats SegmentLLM under matched conditions: When both approaches used the same Nemo model, the table-of-contents format consistently outperformed SegmentLLM's flat segment-index output. SegmentLLM with ChatGPT was the strongest zero-shot method overall, but could not be run on LectureDE due to privacy restrictions on cloud processing.
• Pause cues are training-dependent: Adding inter-sentence pause durations improved every fine-tuned ToC-Nemo result but reduced scores for the zero-shot variant, indicating the model must be trained to interpret these cues usefully.
• Hierarchy helps most on deeply nested data: On LectureDE (5 levels), multi-level methods scored above their own linear B scores — ToC-Nemo + Pause gained +5.94, ToC-Nemo +4.52 — while the linear-only SegmentLLM Nemo dropped −2.65. On AMI, whose annotations are relatively shallow, the gap was negligible across all methods.
• Level matching is sensible: Per-level B-score analysis on Videoaula shows the highest agreement when comparing reference and hypothesis levels of similar granularity, indicating the model captures both broad and fine topic scales rather than collapsing to one.
• Zero-shot remains weak: Prompted models without fine-tuning still lag substantially behind supervised ones, leaving a large gap for approaches that avoid target-domain or target-language training data.

Methodology in Plain English

The researchers fed transcripts to instruction-tuned LLMs one sentence per line, each labelled with an index, and asked the model in a system prompt to output a table of contents with dotted numbering, topic titles, and the sentence indices where each section begins and ends. This forces the model to produce the whole hierarchy rather than just marking change points.

For acoustic information, they optionally appended pause durations (e.g., 42 (pause=0.62s):) to each sentence line. Because the audio was already force-aligned to the transcript, these gaps came for free without extra audio processing.

Two multilingual models were used: Mistral Nemo (12B parameters, called ToC-Nemo) as the primary model and Qwen2.5-7B (ToC-Qwen) for comparison. Both were loaded in 4-bit quantized form via the unsloth library, and the authors compared a zero-shot setting against LoRA fine-tuning, which freezes the base weights and trains small low-rank adapters. Prompts stayed identical across settings to keep the comparison fair.

Evaluation used three datasets: the English AMI meeting corpus (139 sessions, ~73 hours, 3 levels), Videoaula Portuguese computer-science lectures (34 recordings, ~23 hours, 4 levels), and an internal German lecture collection, LectureDE (96 recordings, ~27 hours, 5 levels). All transcripts were produced by Whisper large-v2, aligned with the Montreal Forced Aligner, and sentence-split with SpaCy to mimic a realistic automatic pipeline. Linear results used the finest level and were measured with F1 and boundary similarity B; the new hierarchical metric then averaged B across optimally matched levels.

Why This Matters

Impact on research: The paper reframes segmentation as structured outline generation rather than boundary detection, and its hierarchical metric exposes a blind spot in standard evaluation — a method can score well on flat boundaries while failing to produce nested structure, or vice versa. Releasing code and prompts lowers the barrier for follow-up work.

Real-world applications:

• Accessible lecture and seminar recordings: transcripts with clear multi-level headings as a primary information channel for deaf and hard-of-hearing users.
• Meeting intelligence tools: automatically chaptering long multi-speaker recordings so participants can jump to the relevant discussion.
• Retrieval-augmented generation and information retrieval: topic-aligned chunks give retrieval systems semantically coherent units instead of arbitrary token windows.
• Podcast and video chaptering: extending single-level chapter generators like PODTILE to full hierarchical outlines.

Industry relevance: Any platform hosting long-form spoken content — education technology, enterprise meeting software, podcast and video hosting, media archives — can use this approach without manual annotation, and the LoRA + 4-bit quantization setup keeps training and inference feasible on modest hardware. The demonstrated privacy limitation of cloud APIs for sensitive recordings (LectureDE) also points to locally deployable models as the practical path.

Future Directions

• Closing the zero-shot gap so hierarchical ToC generation works on unseen domains and languages without target-domain supervision.
• Investigating finer or continuous acoustic features beyond coarse inter-sentence pauses, since pauses only helped once the model was fine-tuned.
• Testing whether the hierarchical boundary-similarity wrapper generalizes as a general-purpose metric for other nested segmentation tasks.
• Extending beyond topic structure — for example, speaker roles, argumentative structure, or cross-document linking — and validating at larger scale on datasets with deeper annotation hierarchies.

Target Audience

Speech and language processing researchers working on topic segmentation, spoken document understanding, or long-context LLM applications; practitioners building meeting transcription, lecture indexing, or podcast chaptering systems; and accessibility engineers who need structured written representations of audio content. Readers looking for a practical template for LoRA fine-tuning a small multilingual LLM on a structured generation task will also find the setup and released prompts directly reusable.

Authors’ abstract

Segmenting speech transcripts into thematic sections benefits both downstream processing and users who depend on written text for accessibility. We introduce a novel approach to hierarchical topic segmentation in transcripts, generating multi-level tables of contents that capture both topic and subtopic boundaries. We compare zero-shot prompting and LoRA fine-tuning on large language models, while also exploring the integration of high-level speech pause features. Evaluations on English meeting recordings and multilingual lecture transcripts (Portuguese, German) show significant improvements over established topic segmentation baselines. Additionally, we adapt a common evaluation measure for multi-level segmentation, taking into account all hierarchical levels within one metric.