G-STAR: End-to-End Global Speaker-Tracking Attributed Recognition

AI summary

Overview

• Research area: Speaker-attributed automatic speech recognition (SA-ASR) for long-form, multi-party meeting audio, combining speech LLMs with speaker diarization and tracking.
• Technical level: Advanced (requires familiarity with Speech-LLM architectures, serialized output training, and diarization metrics such as cpWER and DER).
• Scope: The paper introduces G-STAR, an end-to-end framework that couples a cache-conditioned Sortformer-style speaker tracker with a Speech-LLM transcription backbone to produce timestamped, speaker-labeled transcripts with meeting-level global speaker identity consistency under chunk-wise inference.

What This Paper Is About

When a long meeting recording is processed in chunks, a system must not only transcribe overlapping multi-party speech but also keep the same real-world speaker labeled with the same identity across the entire recording, while emitting accurate timestamps. Prior Speech-LLM systems tend to handle either local diarization within a chunk or global labeling, but not both simultaneously. G-STAR aims to solve both problems jointly by feeding structured speaker-tracking cues directly into an LLM that generates attributed text.

Key Contributions

1. An end-to-end Speech-LLM SA-ASR system for multi-party meetings that produces timestamped, speaker-attributed transcripts with meeting-level global speaker identity consistency under chunk-wise long-form inference. The authors state they will release the model and code.
2. Dual-protocol evaluation of G-STAR in both local (oracle-segmented, up to 20-second utterances) and global (full-meeting) settings on meeting benchmarks, compared against representative Speech-LLM systems and strong conventional cascades under clearly specified VAD and collar protocols.
3. Analyses and ablations identifying key design factors behind the gains, including speaker cue fusion strategies, the fusion (interleave) ratio for injecting speaker cues into LLM generation, and the impact of hierarchical cross-entropy objectives.
4. A cache-conditioned tracking design that integrates Sortformer-style arrival-order modeling with a persistent Arrival-Order Speaker Cache (AOSC) so that speaker identity becomes stateful across chunks rather than re-indexed per chunk.

Main Findings

• Local setting (Table 1, oracle VAD/segmentation, up to 20 s, collar 0): G-STAR achieves cpWER/DER of 24.86/19.00 on AMI, 10.29/8.18 on Fisher, 13.90/6.49 on MLC, and 14.54/17.56 on Candor. It improves both metrics over VibeVoice-ASR (30.51/31.99, 15.18/17.68, 21.74/14.01, 22.12/30.89) and MOSS-Diarizen (25.13/32.20, 11.69/21.61, 14.16/10.58, 16.38/31.76) on all datasets.
• Comparison to the NVIDIA cascade: Sortformer reports DER of 29.87 (AMI), 18.33 (Fisher), 17.76 (MLC), 30.92 (Candor); Parakeet reports cpWER of 24.62 (AMI), 27.73 (Fisher), 25.90 (MLC), 27.44 (Candor). G-STAR reduces DER relative to Sortformer, is comparable to Parakeet on AMI, and achieves lower cpWER on Fisher, MLC, and Candor.
• Global (meeting-level) setting (Table 2, collar 0): G-STAR reaches cpWER/DER of 16.44/16.85 on Fisher, 17.15/14.25 on MLC, 15.17/24.89 on Candor, and 30.85/32.23 on AMI, the best meeting-level cpWER across those datasets. The controlled late-fusion cascade scored 21.01/23.41, 23.18/21.38, 17.62/17.67, and 39.52/37.63, indicating the gain comes from tracking-conditioned generation rather than stronger components alone.
• Diarization trade-off: The paper notes that dedicated diarization systems can still be stronger for diarization purity, while G-STAR prioritizes speaker-attributed transcription with cache-consistent global labels.
• Internal out-of-domain test set (Table 3, collar 0.5): G-STAR achieves cpWER/DER of 10.42/4.86 on two-speaker audio and 38.85/28.59 on three-to-four-speaker audio, averaging 34.37/24.88, outperforming the cluster-based pipeline (23.56/14.20, 41.14/30.76, average 38.37/28.15) and VibeVoice-ASR (11.10/14.83, 54.48/38.33, average 47.64/34.62).
• Ablation on fusion and loss (Table 4, local setting): Interleave fusion alone gives 26.33/21.06 (AMI), 10.88/10.24 (Fisher), 14.97/20.21 (Candor); hierarchical CE alone gives 28.63/21.28, 14.23/9.02, 18.30/18.10; both together give 24.86/19.00, 10.29/8.18, 14.54/17.56. The two components are described as complementary rather than redundant.
• Different roles for each component: Interleave temporal fusion reduces both cpWER and DER with a larger gain on cpWER, while hierarchical cross-entropy improves DER more clearly and leaves cpWER largely unchanged.
• Interleave ratio matters: Figure 3 shows that overly dense speaker-cue injection can interfere with lexical modeling, while overly sparse injection provides insufficient tracking guidance; a moderate ratio gives the best balance on MLC local SA-ASR performance.
• Limitations reported by the authors: A strictly streaming deployment was not fully evaluated, and the current data scale may limit robustness across acoustic conditions, speaking styles, languages, and speaker configurations.

Methodology in Plain English

The system processes a long recording as a sequence of chunks. Each chunk goes through two branches. The first branch is an acoustic encoder plus a projector that turns audio into embeddings the language model can read. The second branch is a speaker-tracking module, initialized from a streaming Sortformer model, which looks at the current chunk together with a persistent memory called the Arrival-Order Speaker Cache. This cache stores speakers in the order they first appeared, so a speaker who returns later is matched back to their original slot instead of receiving a new one, which keeps identities stable across chunks.

The speaker cues from the tracking branch are projected and then interleaved into the acoustic embedding stream at a fixed stride (the interleave ratio K), producing one time-ordered sequence that periodically carries explicit speaker evidence. The LLM (Qwen2-7B-Instruct with LoRA weights inherited from FireRed-LLM) then generates a single serialized output stream in a Serialized Output Training style, containing start-time, words, end-time, and a global speaker-ID token such as <spk=k>. Timestamps and speaker labels are added as special LLM tokens.

Training proceeds in three stages: meeting-style ASR pre-training, local SA-ASR training on segmented utterances (up to 20 seconds), and global meeting-level SA-ASR training with 90-second chunks used to tune the Sortformer module. The generation objective uses hierarchical cross-entropy with a 1.5x loss weight on timestamp tokens and a 2x loss weight on speaker-label tokens. The ASR projector, SD projector, and LoRA adapters are the trainable modules; the SD projector is a 1D convolutional downsampling module with stride 5 followed by a two-layer MLP, randomly initialized. Training used packs of up to 12,000 tokens, LoRA rank 64 with scaling factor 16 and dropout 0.05, AdamW with weight decay 0.01, beta1 = 0.9 and beta2 = 0.95, and gradient clipping. The first two stages used a warmup ratio of 0.01 to a peak learning rate of 5e-5 with cosine annealing for 20,000 steps each; the global stage reduced the peak learning rate to 2e-5 and trained for 5,000 steps. Sortformer tuning used AdamW at 1e-4, batch size 4, and 5 epochs.

Experimental data included MLC (English subset only), AMI, Fisher, and Candor (audio modality only, excluding video), plus an internal Chinese conversational corpus dominated by two-speaker conversations, and the open-source AISHELL-4 and AliMeeting datasets used for the internal model. A controlled late-fusion cascade baseline was built using FireRedASR-LLM, a silero-vad VAD front-end, an mms-300m forced aligner, and the same Sortformer-style front-end.

Why This Matters

• Research impact: The paper reframes speaker attribution as a conditioning signal during generation rather than a post-processing decision made after ASR, offering an alternative to late-fusion cascades and post-hoc global clustering in Speech-LLM systems.
• Real-world applications:
  • Meeting transcription and minute-taking where each sentence must be attributed to the right participant across a long session.
  • Compliance, legal, and medical documentation requiring timestamped, speaker-labeled records of multi-party conversations.
  • Conference, panel, and broadcast captioning where overlapping speech and recurring speakers are common.
  • Voice-assistant and conversational analytics systems that need to track who is speaking over long interactions without re-identifying speakers per segment.
• Industry relevance: The modular design supports component-wise optimization and joint end-to-end training, which the authors present as useful under heterogeneous supervision, dataset imbalance, and domain shift. The use of Qwen2-7B-Instruct with LoRA and initialization from FireRed-LLM and Streaming Sortformer means the approach builds on existing open components, and the authors state they will release the model and code.

Future Directions

• Extend the system to real-time streaming deployment and analyze latency, memory usage, and cache update stability under online constraints (explicitly listed as future work).
• Scale both the Speech-LLM training data and the speaker-tracking supervision to improve robustness across acoustic conditions, speaking styles, languages, and speaker configurations.
• Investigate the interleave ratio further, since the paper shows performance is sensitive to how frequently speaker cues are injected.
• Improve diarization purity, since dedicated diarization systems can still outperform G-STAR on DER even when G-STAR achieves better speaker-attributed transcription.

Target Audience

Researchers and engineers working on multi-party speech recognition, speaker diarization, and Speech-LLM systems will benefit most. It is also relevant to practitioners building meeting transcription products who need to weigh end-to-end tracking-conditioned generation against cascaded ASR-plus-diarization pipelines, and to students with background in speech processing and LLM-based sequence generation who want a concrete example of unifying diarization and recognition in a single generative framework.

Authors’ abstract

We study timestamped speaker-attributed ASR for long-form, multi-party speech with overlap, where chunk-wise inference must preserve meeting-level speaker identity consistency while producing time-stamped, speaker-labeled transcripts. Previous Speech-LLM systems tend to prioritize either local diarization or global labeling, but often lack the ability to capture fine-grained temporal boundaries or robust cross-chunk identity linking. We propose G-STAR, an end-to-end system that couples a time-aware speaker-tracking module with a Speech-LLM transcription backbone. The tracker provides structured speaker cues with temporal grounding, and the LLM generates attributed text conditioned on these cues. G-STAR supports both component-wise optimization and joint end-to-end training, enabling flexible learning under heterogeneous supervision and domain shift. Experiments analyze cue fusion, local versus long-context trade-offs and hierarchical objectives.