Summarization and Content Condensation

Different readers need different condensations of the same incident report. Nothing about the source decides which of them is right. The purpose does, and a system that has not been told the purpose is not compressing. It is guessing.

Everything that follows is measured. A national standards body fixed the length budgets and the no-addition rule in writing, decades ago. A national agency ran the benchmarks that scored condensation for twenty years. And two shipped consumer systems condensed real news into claims their sources never made, in public, with dates.

The requested output should specify audience, purpose, scope, and update behavior. Two of these five families are precise enough to have a public record attached. One was written into an agency benchmark. The other has a company post-mortem.

Update summarization is a defined task, not a label. NIST ran it as a scored evaluation: “Piloted in DUC 2007, the TAC 2008 Update Summarization task is to generate short (~100 words) fluent multi-document summaries of news articles under the assumption that the user has already read a set of earlier articles.” The guidelines describe a reader of whom NIST writes that “he would like to read summaries that only talk about what's new or different”. The test set was about 48 topics. Each topic held 20 AQUAINT-2 news articles, split into a chronologically earlier Set A and a later Set B of 10 documents each, and Set A is stipulated as already read. That single stipulation is what turns temporal alignment from a good intention into a scored requirement. A fact correctly extracted from Set B is still a failure if Set A already carried it. Content was not scored by n-gram overlap at all. It was scored against multiple human model summaries with the Pyramid method, which weights Summarization Content Units by how many model summaries express them. Nenkova and Passonneau built that method in 2004 on the premise “that no single best model summary for a collection of documents exists”.

The multi-document row's warning that source authority varies has a public failure attached to it. In May 2024 Google's AI Overviews told people to put glue on pizza, and the screenshots going round were not fakes. Google said so itself on 30 May: “But some odd, inaccurate or unhelpful AI Overviews certainly did show up.” — Elizabeth Reid, VP, Search at Google. The explanation is the uncomfortable one for anyone who treats faithfulness and correctness as the same property. Some overviews had condensed satirical and forum sources accurately. They were wrong because the source was: “Forums are often a great source of authentic, first-hand information, but in some cases can lead to less-than-helpful advice, like using glue to get cheese to stick to pizza.” The advice to eat a rock a day came out of the same mechanism — a data void, filled from whatever occupied it. Google reported more than a dozen technical fixes and disclosed a rate: “We found a content policy violation on less than one in every 7 million unique queries on which AI Overviews appeared.” BBC News, reporting the episode, recorded that Google described the results as “isolated examples”, and named both the “non-toxic glue” pizza answer and the one-rock-per-day answer. A multi-document summarizer that weights every source alike inherits the least reliable one it read.

Separating selection from realization makes summary errors easier to diagnose. This is not the lesson's own advice, and it is not new. Condensation has had a written standard since before neural summarizers existed.

The standard is ANSI/NISO Z39.14-1997, Guidelines for Abstracts, reaffirmed in 2015. In thirty pages it performs most of steps 1 to 4. It separates informative from indicative abstracts — a scope decision taken before a word is written. It names the content elements to be selected: purpose, methodology, results and conclusions. It sets the length budget by document type rather than by taste: a maximum of 250 words for papers and articles, 100 words for notes and short communications, 30 words for editorials and letters, and 300 words for monographs and theses. And it states step 4's whole constraint in one sentence: “Do not include information or claims not contained in the document itself.”

The standard also governs the part of selection that a length budget cannot reach: “Retain the balance and emphasis of the original documents, except in a slanted abstract.” The exception is the interesting half. A condensation written for a declared purpose is permitted to re-weight its source — that is what query-focused summarization is. The standard's answer is that re-weighting must be declared rather than performed silently. The Oregon Department of Transportation distributes the same text under the earlier 2010 reaffirmation, with the same no-addition sentence and the same length table.

Each failure family suggests a different data or system intervention, and two of them now have a counted rate rather than a definition.

BBC journalists spent a month reviewing answers from ChatGPT, Copilot, Gemini and Perplexity, each given access to BBC News articles. 51% of all the AI answers about the news carried significant issues of some form. Distortion was counted separately: 19% of the answers citing BBC content introduced factual errors in statements, numbers or dates. So was misattribution, and it is the figure to remember for any system that promises quotation: “13% of the quotes sourced from BBC articles were either altered or didn’t actually exist in that article.” The BBC published those numbers on 11 February 2025. Pete Archer's summary of the finding was that the assistants “can produce responses to questions about key news events that are distorted, factually incorrect or misleading”.

The rates move, and they do not move to zero. The EBU/BBC study re-measured the BBC subset eight months later: “The share of responses with significant issues of any type improved from 51% to 37%”. A fourteen-point improvement still leaves more than a third of answers as the case the review process has to catch.

ROUGE compares n-gram or sequence overlap with reference summaries, and it remains useful for reproducible comparison. A valid paraphrase can score lower. A high-overlap summary can preserve a source error or omit a crucial fact. That is not a defect discovered later — it is the definition. Chin-Yew Lin introduced ROUGE-N, ROUGE-L, ROUGE-W and ROUGE-S in 2004 as counts of overlapping n-grams, word sequences and word pairs between a system summary and human reference summaries. Nothing in a count of shared units inspects attribution, causation or truth.

The field's standard benchmark took the metric at exactly that value. At NIST's Document Understanding Conference in 2004 the length budget was hard, and measured in bytes rather than words: “The maximum target length for very short summaries was 75 bytes. For short summaries it was 665 bytes.” Anything longer was truncated before scoring, with no credit for coming in under. And the scoring ran on one channel: “The summaries in tasks 1-4 were evaluated solely by means of ROUGE (ISI's Recall-Oriented Understudy for Gisting Evaluation, alias RED) automatic (n-gram) matching.” The official measures were 1-gram, 2-gram, 3-gram, 4-gram and longest common substring. A system could lead that table with 665 bytes of well-overlapping words in which every quotation was attributed to the wrong speaker. By TAC 2008 the same agency was scoring content manually with the Pyramid method, and computing ROUGE and BE scores for every submitted run alongside it, precisely to track how well the automatic measures correlated with the manual ones.

Complement overlap with semantic, source-grounded, question-based, factuality, human, and task-based evaluation; validate automated evaluators against domain judgments before scaling them.

Fabbri and colleagues took that second channel seriously in Transactions of the ACL. They “re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion”. They also “consistently benchmark 23 recent summarization models”. The work uses model outputs “along with expert and crowd-sourced human annotations”. The study exists because the question was open, in their own words. “The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress.”

The evaluation should reveal what is lost or invented under each compression setting. Step 4 — score with multiple channels — is the step usually left as a list of nouns. Here is what it looks like carried out at full size.

In October 2025, 22 public service media organisations in 18 countries and 14 languages ran the same test at once, coordinated by the EBU and the BBC. Journalists — fourteen NPR editorial staff among them — rated responses from ChatGPT, Copilot, Gemini and Perplexity against a multi-criterion rubric: accuracy, sourcing, opinion versus fact, editorialization, and context. The methodology appendix records 2,709 of a potential 2,760 core responses actually evaluated. The headline sits in the foreword: “Overall, 45% of responses contained at least one significant issue of any type. Sourcing is the single biggest cause of significant issues (31%).” That foreword is signed by Pete Archer of the BBC and Jean Philip De Tender of the EBU.

Separate criteria are what make the result usable. 20% of responses had significant accuracy issues, and on that criterion the four assistants sat close together, between 18% and 22% — a difference no product decision should rest on. Sourcing is where they came apart. Gemini had a significant sourcing issue in 72% of its responses, while every other assistant stayed below 25%. One blended quality score would have averaged 72% against under 25% into a single unremarkable number, on the criterion that produced most of the failures in the whole study. The report's own conclusion is that “AI assistants are still not a reliable way to access and consume news”.

Take one meeting transcript and identify decisions, owners, deadlines, rejected proposals, uncertainty, and unresolved questions. Compare these with a generated summary at two length budgets. Borrow the budgets from a standard rather than inventing them: ANSI/NISO Z39.14 allows 250 words for a paper and 100 for a short communication, and the gap between those two is where selection starts to hurt.

Mark every summary proposition as supported, distorted, unsupported, or ambiguous. Then score the summary on the five criteria the EBU/BBC reviewers used — accuracy, sourcing, opinion versus fact, editorialization, context — rather than on one impression of quality. That separation is what let their study report 72% on one criterion and 20% on another instead of a single flat verdict. Check the quotations separately and verbatim, since 13% of the quotes drawn from BBC articles in the February 2025 study had been altered or did not exist. Then ask a participant whether the summary enables the intended follow-up without hiding disagreement.