Reading Between the Lines: The One-Sided Conversation Problem

AI summary

Overview

Research area: Natural Language Processing / dialogue systems, spanning conversational AI, text infilling, speech transcription, and privacy-aware computing.

Technical level: Intermediate. The paper is readable without deep NLP specialization, but assumes familiarity with large language models, prompting versus finetuning, and evaluation-by-judge setups.

One-sentence scope: The paper formalizes the "one-sided conversation" (1SC) problem, where only one speaker's utterances are available, and evaluates whether language models can reconstruct the missing speaker's turns and summarize such conversations across four dialogue datasets.

What This Paper Is About

Many real-world conversational AI settings can capture only one side of a dialogue, either because of recording laws and consent requirements or because devices like smart glasses and in-ear assistants are designed to record only the wearer. This leaves the other speaker's turns entirely absent, not merely missing a few words.

The authors define this as the one-sided conversation problem (1SC): inferring and learning from a dialogue when only one speaker's utterances are observed. They study two tasks under this constraint — reconstructing the masked speaker's turns online, and generating faithful summaries from one-sided transcripts — and ask whether current models can do either reliably without fabricating information.

Key Contributions

1. Formalization of the 1SC problem. The authors define 1SC as a distinct setting from text infilling, because the entire other side of the dialogue is absent rather than short spans, and frame it explicitly as a privacy-motivated problem rather than a role-play task. They state that to their knowledge 1SC has not previously been studied in the literature.

2. Two task formulations with controlled context ablations. For reconstruction they compare a baseline full prior context, adding Turn N+1, adding turn lengths, and a limited three-turn local context (Turn N-1, Turn N, Turn N+1). For summarization they compare summaries generated directly from masked transcripts against summaries built from reconstructed turns, benchmarked against an oracle summary produced from the full two-sided conversation.

3. An evaluation framework combining human and automated judgment. They use GPT-4o as a judge with a five-criterion rubric (Semantic Similarity, Intent Preservation, Specific Information Hallucination, Contextual Appropriateness, Summary Alignment) plus precision-recall metrics based on extracted details. This is paired with A/B testing using 16 human judges for reconstruction and 5 human judges for summarization.

4. Empirical results across text and speech data. They evaluate on MultiWOZ, DailyDialog, SpokenWOZ, and Candor, and verify that their reported scores are not the product of dataset contamination. Code and data are released at https://github.com/ebertv/onesided.

Main Findings

• Additional context improves reconstruction. Access to the turn immediately following a masked turn, and information about the length of the masked turn, both improve reconstruction over using preceding context alone. In the reported rubric scores, adding both Turn N+1 and turn lengths produced the highest scores in each dataset's full-prior-context setting.

• Big models work out of the box; small models do not catch up. Claude-4-Sonnet generated plausible reconstructions with prompting alone, while Llama-3.2-1B-Instruct and a finetuned Llama-3.2-1B scored substantially lower even under the same local context. For example, on MultiWOZ, finetuned Llama scored 1.98 (1.12) on Semantic Similarity, 3.13 (1.42) on Intent Preservation, 2.39 (1.17) on Contextual Appropriateness, 1.98 (1.11) on Summary Alignment, and 2.46 (1.46) on Anti-Hallucination, versus 2.96 (1.20), 4.06 (1.17), 4.07 (0.98), 3.00 (1.24), and 4.74 (0.75) for Claude in the full-setting ablation.

• Human judges did not clearly prefer ground truth. With 16 human judges on three-turn contexts from DailyDialog (n=50; Claude n=25, Llama n=25) and MultiWOZ (n=50; Claude n=25, Llama n=25), Claude reconstructions were preferred over the ground-truth response in 32% of DailyDialog cases and 56% of MultiWOZ cases. Finetuned Llama reconstructions were heavily dispreferred (ground truth selected in 68% of DailyDialog cases and 40% of MultiWOZ cases).

• High-quality summaries do not require reconstructing missing turns. Masked-dialogue summaries were consistently ranked above reconstructed-dialogue summaries by the GPT-4o judge and had higher precision with comparable recall, evaluated on DailyDialog (n=1000), MultiWOZ (n=1313), SpokenWOZ (n=100), and Candor (n=137).

• Plausible reconstructions can hurt summarization. The authors explain the apparent contradiction by task goals: reconstruction rewards local plausibility, while summarization rewards faithfulness, so invented details that help a turn-level reconstruction can reduce factual alignment when propagated into a summary. Across all ablations, Semantic Similarity to ground truth scored lower than Contextual Appropriateness.

• Speech transcripts are harder to reconstruct but not harder to summarize. Comparing SpokenWOZ to MultiWOZ, reconstruction quality dropped on audio transcripts; comparing SpokenWOZ to Candor, task-oriented dialogue was easier to recreate than open-domain conversation. On the human A/B test, SpokenWOZ raters saw no difference between ground-truth and reconstructed utterances, and Candor reconstructions were often preferred over the true conversation.

• One dataset reversed the summarization finding. On the full Candor dialogues, predicting the missing utterances was helpful for summaries. The authors conjecture this is caused by dialogue length (Candor dialogues are at least ten times longer than the other datasets on average); after splitting Candor into excerpts of approximately 25 turns, results reverted to the original pattern.

• Turn-by-turn prediction beats whole-conversation prediction. Reconstructing entire conversations at once scored lower across the rubric than single-turn prediction on both DailyDialog and MultiWOZ full test splits.

• Placeholder prompting matters. Removing the instruction to replace unavailable specific information with "xxxx" degraded performance across all measurements.

• Human summary raters split on quality dimensions. Judges perceived dialogue flow and detail balance as better in summaries from predicted conversations, while information accuracy was higher in summaries from masked conversations, especially for DailyDialog compared to MultiWOZ.

Methodology in Plain English

The setup is a two-party dialogue where one speaker is designated the user (observed) and the other the masked speaker (to be inferred). For reconstruction, the model is asked to predict one masked turn at a time, under four context conditions: the full conversation up to the target turn; the full prior context plus the user's next utterance; the full prior context plus word counts as a proxy for timing; and a restricted three-turn window. Models are instructed to output "xxxx" as a placeholder when specific information is not available, and future turns are provided only as context, not as knowledge the masked speaker could have had.

Finetuning used Llama-3.2-1B autoregressively on the limited-context setting, with SODA used only for training and validation and never for evaluation. Prompting used Claude-4-Sonnet (in both complete-context and three-turn settings) and Llama-3.2-1B-Instruct (in the three-turn setting only, to match the finetuned setup). Few-shot examples and reiterated instructions were included.

Evaluation used GPT-4o as a separate judge, either producing 1–5 ordinal scores on the rubric criteria or extracting important details from predictions and ground truth to compute precision and recall. Placeholders were treated as equivalent to ground truth to avoid penalizing anti-hallucination behavior. Human A/B testing showed judges three-turn contexts and two candidate responses, with a sparingly used "neither" option; human summary evaluation used a blind review of three summaries (one-sided, reconstructed, and two-sided oracle) rated 1–5 on five dimensions, with session order randomized and roughly 20 comparisons per session on average.

Candor required preprocessing: anything said by the second speaker during the duration of the user's utterances was excluded, mimicking a recording where the other speaker is detectable only as silence. Data details reported include MultiWOZ (61k train, 6.4k validation, 6.4k test, average 24 turns), DailyDialog (32.3k train, 3.9k validation, 4.5k test, average 8 turns, split 80/10/10), and downsampled test sets for SpokenWOZ (100 conversations) and Candor (8 conversations), each just over 2000 masked turns.

Why This Matters

Impact on research. The paper opens a problem setting distinct from text infilling, dialogue state tracking, and user simulation, all of which typically assume access to structured annotations, domain constraints, or both speakers' turns. It supplies task formulations, context ablations, and a reusable evaluation framework, positioning 1SC as a benchmarkable challenge and arguing that reconstruction and summarization should be evaluated by different criteria.

Real-world applications:

• Proactive assistance devices such as smart glasses and in-ear assistants, where only the wearer's speech is captured. The authors describe a workflow where the agent observes turn t, infers the interlocutor's turn, and evaluates the user's actual response at t+1 to detect missed questions or constraints.
• Call centers and telemedicine, where compliance rules often mean only the agent's or patient's transcript is retained.
• Training and role-play, such as doctor-patient interactions for residents or reconstructed conversations from experienced call center agents for trainees, where multiple plausible reconstructions are useful even if imperfect.
• Forensic settings, analogous to lip-reading techniques that produce several candidate transcripts rather than a single reading.
• Additional uses named in the paper include memory augmentation, personalized social coaching, meeting summarization, support for neurodiverse individuals interpreting social cues, sales calls and negotiations, interviews, and cross-cultural communication.

Industry relevance. The motivation is legal as much as technical. The paper notes that U.S. recording law varies between one-party and all-party consent states (California, Delaware, Florida, Illinois, Maryland, Massachusetts, Montana, Nevada, New Hampshire, Pennsylvania, and Washington at time of publication), that U.S. federal law (18 U.S.C. §2511) prohibits intentional interception except as authorized, and that EU organizations must establish a lawful basis, often explicit consent, under GDPR. Systems that improve performance under one-sided input therefore align with constraints that already govern deployed products.

Future Directions

1. Post-training for 1SC specifically. The authors propose RLHF and RLAIF targeted at reconstruction and summarization of one-sided conversations, which may close the gap between large and small models that finetuning alone did not.

2. Multi-turn and autoregressive prediction. They suggest having a model predict turn 2 and then use that prediction, together with the one-sided transcript, as context for predicting turn 4, and they list multi-turn prediction as an open direction.

3. Conversations with more than two parties. The current formulation is limited to two-party dialogue, leaving multi-party settings unaddressed.

4. Deployment safeguards. The limitations section calls for transparent communication of model uncertainty, protections for the storage and sharing of derived summaries, visible indicators such as an LED on smart glasses signaling 1SC operation, and clear labeling of AI-generated turns or summaries. The authors also flag that using a third-party LM API with even one side of a conversation may pose a privacy risk, and suggest local deployment of an open-weight model instead.

Target Audience

Researchers working on dialogue systems, conversational AI, and text infilling will find the task formulations and context ablations most directly useful, as will NLP practitioners interested in evaluation methodology combining LLM-as-a-judge rubrics with human A/B testing. The paper is also aimed at HCI and privacy researchers, since its motivating scenarios involve augmented-reality devices, in-ear assistants, and recording law. Product and policy teams building assistants for call centers, telemedicine, or wearable hardware will benefit from the finding that useful summaries can be produced from one-sided input without reconstructing missing turns. Readers seeking a benchmark to extend will find the released code and data at the URL given in the paper.

Authors’ abstract

Conversational AI is constrained in many real-world settings where only one side of a dialogue can be recorded, such as telemedicine, call centers, and smart glasses. We formalize this as the one-sided conversation problem (1SC): inferring and learning from one side of a conversation. We study two tasks: (1) reconstructing the missing speaker's turns for real-time use cases, and (2) generating summaries from one-sided transcripts. Evaluating prompting and finetuned models on MultiWOZ, DailyDialog, and Candor with both human A/B testing and LLM-as-a-judge metrics, we find that access to one future turn and information about utterance length improves reconstruction, placeholder prompting helps to mitigate hallucination, and while large models generate promising reconstructions with prompting, smaller models require finetuning. Further, high-quality summaries can be generated without reconstructing missing turns. We present 1SC as a novel challenge and report promising results that mark a step toward privacy-aware conversational AI.