TurnTurk-GPT: Text-Based Turn-Taking Prediction for Turkish

AI summary

Overview

Research area: Natural Language Processing — spoken dialogue systems, turn-taking prediction, and Turkish language modelling.

Technical level: Intermediate. The paper assumes familiarity with transformer language models, fine-tuning, and information-retrieval style metrics such as average precision and bootstrap significance testing, but each component is described in enough detail to follow without prior work on turn-taking.

Scope (one sentence): The paper builds and evaluates TurnTurk-GPT, a Turkish text-based end-of-turn predictor fine-tuned from a 124M-parameter Turkish GPT-2, comparing it against rule baselines, a zero-shot reading of the same backbone, and the LiveKit multilingual turn detector on real Turkish dyadic conversation.

What This Paper Is About

A spoken dialogue system has to decide, moment by moment, whether the user has finished speaking or is merely pausing; answering too early interrupts, answering too late feels sluggish. Most turn-taking research and benchmarks cover English, and although deployed multilingual end-of-turn detectors list Turkish as supported, their behaviour on real Turkish conversation has not been reported under a public protocol. This paper trains a text-only predictor for Turkish (a verb-final, agglutinative language whose completion cues sit at the end of the utterance) and evaluates it carefully on a small real corpus, including the harder case where the test speakers never appeared in training.

Key Contributions

1. A Turkish turn-taking model and protocol. TurnTurk-GPT is trained by fine-tuning turkish-gpt2 (124M parameters) with the TurnGPT objective — a <ts> turn-shift token plus two speaker tokens — and released publicly on Hugging Face along with the evaluation protocol.
2. A rigorous, leakage-aware evaluation design for Turkish. The protocol uses word-level decision points on unpunctuated, lower-cased text, average precision as the primary metric, recording-disjoint (leave-one-recording-out) and speaker-disjoint folds, nested threshold selection, and paired cluster bootstrap tests over 2,000 recording-level replicates.
3. A comparison against three kinds of baseline. Rule baselines (random, turn-length prior), a zero-shot reading of the untuned backbone in four ways, and the LiveKit multilingual turn detector used as released at every word and at pause moments.
4. Turkish-specific analyses. Morphological ablation on the turn-final word, a tokeniser/backbone comparison, an analysis of synthetic-only training, and an evaluation restricted to the 633 pre-pause negatives where a deployed system would actually query.

Main Findings

• TurnTurk-GPT beats the multilingual detector on both splits. On the recording-disjoint split, the best configuration (S2R — synthetic pre-training then real fine-tuning) reaches AP 0.148 with 95% CI [.132, .167], against 0.100 [.081, .118] for LiveKit, a paired difference of +.048 [+.034, +.060]. On unseen speakers S2R reaches AP 0.120 and still beats LiveKit by +.020 [+.009, +.034].
• The zero-shot newline baseline is strong. Reading the untuned backbone's probability of a line break gives AP 0.130 [.103, .154], which itself beats LiveKit (paired difference −.029 [−.039, −.016] for LiveKit minus LM newline). S2R beats it by +.019 [+.002, +.037] on the recording-disjoint split only.
• Speaker familiarity, not model capacity, is the limiting factor. On unseen speakers S2R versus the zero-shot newline model is not significant (−.010 [−.025, +.011]), and the speaker-disjoint split costs 0.028 AP relative to recording-disjoint.
• Synthetic dialogue alone is the weakest trained model. The synthetic-only configuration (S) reaches AP 0.082, below the zero-shot model, and has the highest pre-pause false alarm rate of all systems at 33.5%.
• Synthetic pre-training helps calibration, not ranking. S2R beats the mixed SR configuration by +.028 [+.013, +.044], but the difference from real-only R (+.032 [−.008, +.063]) is not significant. The threshold needed to reach 70% recall is 0.00000 for R (probabilities below numerical precision) versus 0.016 for S2R.
• The end-of-turn cue lives in the suffixes of the final word. Stemming the turn-final word to four characters drops AP by 32% for R (0.116 → 0.079) and 44% for S2R (0.148 → 0.083), while stemming a random sentence-internal word changes AP by only 0.2% and 2.2%.
• The tokeniser matters substantially. The English GPT-2 tokeniser splits Turkish into 2.94 tokens per word versus 1.17 for the Turkish one; with the English backbone, R's AP falls from 0.116 to 0.042, near the random baseline of 0.037.
• At practical operating points nobody is deployable alone. At 70% recall, S2R produces 9.6 false alarms per correct detection and the zero-shot model 9.3, while LiveKit produces 16.0; at 90% recall the zero-shot model is slightly ahead of S2R (.062 vs .059 precision).
• The published multilingual threshold does not transfer. At its recommended Turkish threshold of 0.0045, LiveKit predicts a turn end at most positions: recall 0.881, precision 0.045, and 85.5% false alarms on pre-pause words.
• When the speaker pauses, text alone nearly fails. On the balanced pause-only task (578 turn ends vs 633 turn-internal pauses, random AP 0.477), the zero-shot newline baseline reaches AP 0.590 and LiveKit 0.530 — both close to chance.
• Fine-tuning does not damage the backbone. Held-out perplexity falls from 282.1 to 272.8.

Methodology in Plain English

Start with a pre-trained Turkish language model and add three new tokens: one that marks a turn shift and two that mark who is speaking. Serialise dialogue as turns, each closed by the turn-shift token, and train with the ordinary next-token language modelling loss. Because the score for a word is just the probability of the turn-shift token at that word's last subword piece, only preceding context is used — the score can be computed incrementally as words arrive, exactly the setting a streaming recogniser produces.

Train four ways: synthetic only (S), real only (R), both in one stage (SR), and synthetic first then real (S2R). Because Syn-TurnTurk is two orders of magnitude larger than Real-TurnTurk, all four use a fixed step budget, and each is run with three seeds. Feed the model lower-cased, punctuation-stripped text to approximate recogniser output.

For evaluation, treat every word in the real corpus as a decision point: 15,522 points, of which 578 are turn ends. Score with average precision, where a random predictor gets 0.037. Choose precision/recall/F1 thresholds on the other folds rather than the test fold. Use two splits — leave one recording out, and speaker-disjoint (where training shrinks to two or three recordings). Resample recordings rather than decision points in the 2,000-replicate bootstrap, so the confidence intervals reflect variation between conversations, and use paired resampling to compare two systems on the same decision points.

Compare against: a random baseline, a turn-length prior estimated on training folds, the untuned backbone read four ways (probability of a period/question mark/exclamation, of end-of-text, of a newline, and their sum), and the LiveKit multilingual turn detector run through its own inference code with its own normalisation. Finally, report false alarm rates on the 633 word positions that are followed by a turn-internal pause, since those are the moments a real voice agent queries its model.

Why This Matters

Impact on research. The paper shows that for Turkish a plain zero-shot newline probability from an untuned backbone is already a strong baseline — stronger than a deployed multilingual detector — and that a weaker reading of the same model would overstate the benefit of fine-tuning roughly twofold. It argues explicitly that future Turkish turn-taking work should report the zero-shot newline baseline. It also quantifies a commonly suspected but rarely measured effect: with six conversations of training data, apparent gains are largely speaker familiarity, and the speaker-disjoint split is what exposes this.

Real-world applications.

• Turkish voice agents and IVR systems that need to decide when to start responding to a caller.
• Speech-recognition front-ends that must endpoint an utterance without cutting off a speaker who hesitates mid-sentence.
• Meeting and call-centre transcription, where turn boundaries drive speaker attribution and segment formatting.
• Multilingual assistant platforms that currently inherit thresholds from another language; the paper shows the Turkish threshold of 0.0045 raises false alarms on 85.5% of pre-pause words.

Industry relevance. The concrete recommendation is that thresholds be selected on in-domain data and that systems be compared at matched recall, not at each system's own optimum. The paper is candid that text-only prediction produces roughly ten false alarms per correct detection at 70% recall, so it should complement rather than replace silence-based endpointing. Both the model and corpus links are released, giving practitioners a reproducible Turkish baseline.

Future Directions

1. Collect conversations from more speakers. The authors identify this as the most direct route to improved generalisation, and note that no open spontaneous two-party Turkish corpus currently exists, so such data would benefit work beyond this paper.
2. Generate synthetic dialogue that behaves like real dialogue — longer turns, sentences spanning turns, and turns that pause and then resume — so the synthetic corpus is useful for more than calibration.
3. Add prosodic and visual cues. Real-TurnTurk has separate audio channels and facial features, making it possible to test whether these resolve the pause case, where text-only systems sit near chance.
4. Move from offline final transcripts to streaming partial hypotheses and measure latency, which the authors flag as a natural extension of the current offline setting.

Target Audience

Researchers and engineers working on spoken dialogue systems, end-of-turn detection, and incremental language modelling — especially those working on Turkish or other morphologically rich, verb-final languages where the completion cue is concentrated in a single word's suffixes. It is also a useful methodological read for anyone evaluating small, multi-speaker conversational corpora: the speaker-disjoint split, nested threshold selection, and paired cluster bootstrap form a template that transfers directly to other low-resource languages and to any setting where trained models are compared against zero-shot language model baselines.

Authors’ abstract

This paper presents TurnTurk-GPT, a text-based turn-taking predictor for Turkish obtained by fine-tuning a Turkish GPT-2 model with the TurnGPT objective on synthetic and real dialogue, and evaluates it on real Turkish conversation against rule baselines, a zero-shot Turkish language model and a widely used multilingual end-of-turn detector. Evaluation uses word-level decision points on unpunctuated, lower-cased text approximating the output of streaming speech recognisers, recording-disjoint and speaker-disjoint folds, nested threshold selection and paired cluster bootstrap tests. TurnTurk-GPT outperforms the multilingual detector under both recording-disjoint and speaker-disjoint evaluation. It outperforms the zero-shot language model only when the test speakers occur in training; on unseen speakers the two are statistically indistinguishable. Removing the suffixes of the turn-final word reduces AP by up to 44%, against 2% for a sentence-internal word, locating the Turkish end-of-turn cue in verb-final morphology. Synthetic dialogue alone yields the weakest trained model, and when a speaker pauses the zero-shot and multilingual baselines remain close to chance. The model and the evaluation protocol are publicly released.