SAUCE: Summary Analysis Using Conversation Entailment

AI summary

Overview

Research area: Natural Language Processing — reference-free evaluation of summarization, specifically cross-lingual conversational speech summarization produced from cascaded Automatic Speech Recognition (ASR) and Machine Translation (MT) pipelines. Work from RTX BBN Technologies (Cambridge, MA, USA).

Technical level: Intermediate. Readers should be comfortable with summarization metrics (ROUGE, BERTScore, SummaC), Natural Language Inference / textual entailment, and the basics of ASR/MT error propagation.

Scope (one sentence): The paper introduces SAUCE, a two-metric, reference-free pipeline that scores summaries of noisy conversational speech by extracting atomic facts with an LLM and checking entailment with a dedicated NLI model, producing an interpretable fact-by-fact ledger.

What This Paper Is About

Evaluating a summary is hard because there is no single correct summary for a document — many different summaries can be valid, and user preferences vary in both style and content. The problem is worse for everyday conversations, which have no clear end goal, contain disfluencies and speaker overlaps, and in the cross-lingual setting are further degraded by ASR and MT errors. The authors build SAUCE (Summary Analysis Using Conversation Entailment), a reference-free pipeline that measures whether a summary's facts are supported by the conversation (faithfulness) and whether the conversation's important facts appear in the summary (coverage), while exposing the exact facts behind each score.

Key Contributions

1. A two-metric, reference-free evaluation pipeline for cross-lingual conversational speech summarization. SAUCE Faithful is the percentage of summary facts entailed by the conversation; SAUCE Coverage is the percentage of conversation facts covered by the summary. Both are computed without human reference summaries.
2. A transparent "fact ledger" as a by-product of scoring. Rather than returning one opaque float, the system surfaces which atomic facts were entailed and which were contradicted or unsupported, letting users distinguish hallucinations (unentailed summary facts) from omissions (unentailed conversation facts).
3. Demonstration that the pipeline tracks information loss from ASR and MT errors where NLI-based baselines (SummaC Conv, SummaC ZS, FENICE) show inconsistent or negligible degradation.
4. A diagnostic use case for model selection: the authors profile four off-the-shelf LLMs (GPT-4o, Mistral7B-instruct-v0.3, DeepSeek-R1, DeepSeek-R1-Distill-Qwen-14B) and show SAUCE explains why a model scores as it does (abstractive over-compression versus extractive hallucination), plus a human evaluation correlating the metrics with human judgments.

Main Findings

• SAUCE Coverage detects ASR/MT damage; SAUCE Faithful stays high. With both ASR and MT errors introduced, SAUCE Coverage is described as "nearly cut in half," while SAUCE Faithful and FENICE remain at 90+% regardless of input quality. The authors explain this as GPT-4o retreating to generic but accurate statements when the transcript degrades.
• FENICE is blind to omissions. Because FENICE has no coverage metric, it shows very limited degradation under errors, especially on Spanish.
• SummaC behaves erratically across languages. SummaC Conv shows a 40–60% absolute difference between Spanish and the other three languages, which the authors attribute to linguistic structure differences and SummaC's design for written text.
• LLM-as-judge metrics degrade but remain black boxes. G-EVAL and FineSurE show consistent degradation under MT and ASR errors except for G-EVAL's consistency in Arabic and Spanish. FineSurE also fails a sanity check: evaluating a ground-truth summary against key facts extracted from that same summary does not yield 100% faithfulness, and its 100% completeness across all languages "seems unrealistic."
• Rank correlation favors SAUCE Coverage. Kendall tau between metric score and ground-truth-versus-ASR+MT summary pairs: SAUCE Faithful 0.153, SAUCE Coverage 0.677, SummaC Conv −0.266, SummaC ZS 0.008.
• Broken summaries are caught. On CallHome Spanish broken summaries (from an over-compressed Mistral7B with td2 quantization), SAUCE Faithful drops to 0.356 and SAUCE Coverage to 0.000, versus 0.941 and 0.358 on normal Spanish summaries. SummaC Conv only moves from 0.641 to 0.598 — still above the 0.221 it gives normal Arabic summaries.
• Model profiles differ in a diagnosable way. GPT-4o achieves the highest SAUCE Faithful, DeepSeek-R1 the highest SAUCE Coverage, and DeepSeek-R1's SAUCE Faithful is the lowest — lower even than its distilled 14B variant. GPT-4o's SAUCE Coverage is below a much smaller Mistral-7B and sometimes only comparable to DeepSeek-14B in Arabic and Mandarin.
• Metrics align with human judgment. With 12 participants producing 132 rank pairs, SAUCE Faithful has the highest correlation with human accuracy rankings (tau 0.259) and SAUCE Coverage the highest with informativeness (tau 0.475). Participant-to-participant agreement was only 60%, underscoring task difficulty.
• Conversation entailment is hard even for the entailment model. On the MSU dataset, entailed pairs score much higher than non-entailed pairs; most errors involve swapped tenses or swapped speaker identities, such as confusing the speaker with the person referred to as "he."

Methodology in Plain English

SAUCE runs two stages, both built from off-the-shelf models.

Stage 1 — Fact extraction. GPT-4o (gpt-4o-2024-08-06) reads a conversation or a summary and writes out "atomic facts" — short, checkable statements. The authors report that their internal evaluation found the extracted facts to be over 98% accurate.

Stage 2 — Entailment. DeBERTa-v3 (deberta-v3-large-mnli) decides whether a fact is supported by a piece of text.

For SAUCE Faithful, the pipeline extracts all facts from the summary and checks each against chunks of the conversation, using 5 lines of conversation per chunk. If a fact-to-chunk score exceeds a threshold, the fact counts as entailed; the metric is the proportion of matched facts. The authors tried checking summary facts against facts extracted from the conversation instead, but found the fact-to-chunk approach significantly better.

For SAUCE Coverage, the LLM is asked to extract only the N most important conversation facts, where N is set by conversation length and a desired compression rate. They do not extract everything, because the LLM will often produce over 100 facts from a 10-minute recording. To handle LLM nondeterminism, the top-N extraction is run 5 separate times and the lists concatenated, on the assumption that repeatedly selected facts are more likely important. Each fact is then checked against each line of the summary.

Experimental setup. The authors use conversational telephone speech in Arabic, Spanish, Mandarin and Russian (CallHome for Arabic; CallHome and Fisher for Spanish; HUB5 for Mandarin; Mixer 3 Speech for Russian), averaging 10 minutes per conversation. English translations are crowd-sourced for Arabic and Spanish and in-house for Mandarin and Russian. GPT-4o writes 200-word summaries; the authors tried 50, 100, 200 and 500 words and chose 200 as a balance between conversation length and readability. The ASR+MT cascade uses Whisper-large-v3 for ASR and NLLB-200-distilled-1.3B for MT, both fine-tuned on each language's training set. N is set to 20 for 200-word summaries. Baselines are SummaC Conv, SummaC ZS, FENICE, G-EVAL and FineSurE.

Threshold selection. On the MSU conversation entailment dataset (statements and supporting Switchboard chunks), they tested thresholds of 0.9, 0.95 and 0.98 on both selected chunks and full conversations, and chose 0.9 because correctly detecting entailed pairs was deemed more important.

Why This Matters

Summary evaluation is the bottleneck for deploying LLM summarization on messy, real-world speech. The paper argues that existing tools fail in two complementary ways: traditional NLI metrics built on structured written text break down on disfluencies, overlaps and translation noise, while LLM-as-a-judge systems are expensive black boxes subject to verbosity bias, self-preference bias, and a reported overlap bias favoring AI-generated summaries — and they often overlook granular factual contradictions in favor of surface fluency. SAUCE's contribution is that every score can be traced to discrete, verifiable facts.

Real-world applications:

• Meeting notes and call transcripts: checking LLM-generated notes for hallucinated entities versus genuinely omitted topics.
• Customer-service and call-center QA: auditing summaries of noisy multilingual calls where transcription and translation are unreliable.
• Cross-lingual monitoring workflows: detecting when translation or recognition degradation silently strips detail out of downstream summaries.
• Model selection for product teams: choosing between an abstractive model with high faithfulness and an extractive model with high coverage, based on the application's tolerance for hallucination versus omission.

Industry relevance: The pipeline composes existing commercial/open components (GPT-4o, DeBERTa-v3, Whisper, NLLB) and produces an artifact that end users can inspect, which is a practical requirement for deploying summarization in audited or high-stakes settings.

Future Directions

• Fine-tuning the entailment model for conversation. DeBERTa-v3 was pretrained on general text and no dedicated conversational entailment model currently exists; fine-tuning requires substantial annotated data that is presently unavailable.
• Replacing or diversifying the fact extractor. The work relied solely on GPT-4o, which requires API access and prevents offline deployment; the authors plan to evaluate other fact extractors and their biases, and to distill smaller LLMs from high-performing teacher models.
• Improving fact-extraction consistency to mitigate fluctuation and bias in the scores.
• Strengthening human validation. The human evaluation sample was limited by resource constraints, and annotators reported that reviewing long transcripts against summaries is exceptionally demanding, leaving open how to systematically cross-validate such pipelines.

Target Audience

Researchers and engineers working on summarization evaluation, speech summarization, and multilingual ASR+MT pipelines; practitioners who need an interpretable, reference-free way to audit LLM summaries; and developers comparing off-the-shelf LLMs for a summarization product. Readers focused on purely text-based summarization benchmarks will find the conversation-and-noise framing the most novel part.

Authors’ abstract

With the growing need for evaluating Large Language Models (LLMs) and their applications to speech, challenges persist in summarizing and evaluating conversations that lack a clear end goal. We introduce SAUCE -- a reference-free, fact-based evaluation pipeline for cross-lingual conversational speech summarization. It measures the accuracy and the fact coverage of a summary through the entailment between conversation and text. We compare SAUCE against several popular summarization metrics and demonstrate the effectiveness of capturing information loss due to transcription and translation error and identifying broken summaries. Crucially, unlike black-box LLM evaluators or dense embedding metrics, SAUCE is inherently explainable: it maps summary scores to discrete, verifiable facts, allowing users to pinpoint exact hallucinations or omissions. We illustrate how this interpretability helps developers systematically profile LLM behaviors and gives end-users an actionable tool to verify summary accuracy in noisy, real-world conditions. Preliminary investigations show SAUCE strongly align with human judgment.