What Breaks When Facts Become Stories? The Distortions Source-Agreement Metrics Miss When Models Retell Real Events

AI summary

Overview

Research area: AI evaluation methodology — specifically how we measure fidelity when language models reconstruct recorded facts into new narrative forms (summarization versus storification).

Technical level: Intermediate. The paper is conceptually framed and requires no specialized math, but it assumes familiarity with summarization metrics, factuality evaluation, and inter-annotator agreement.

Scope (one sentence): Across 2,000 news articles rewritten by three model families as both summaries and stories, the authors measure what changes on every axis of the five Ws and how, and show that the accountability-focused distortion their protocol detects is invisible both to claim-level fact checking and to grammatical indicators, while the boundary between "warranted departure" and "distortion" remains genuinely contested among both model judges and human annotators.

What This Paper Is About

Nearly all evaluation of model-generated retellings — whether overlap metrics like ROUGE or modern factuality checks — treats agreement with the source as success. That works for summarization, where staying close to the source is the goal. But when a model is asked to retell an event from an assigned perspective, some departure is intended: a victim narrator may omit what a victim could not have known. Applied unchanged, an agreement metric counts those intended departures as errors; withheld entirely, it lets real distortion through. The paper's goal is to draw a workable line between departures a perspective warrants and departures that misinform, then measure what lies on each side.

Key Contributions

1. A judgeable definition of "warranted departure." The authors define distortion as the breaking of success conditions that must hold regardless of narrator — the agent of a harmful act remains, perpetrator and victim keep their places, stated causes stay linked to effects, and contested claims stay attributed — and add an instance-level legitimacy verdict (warranted / unwarranted) applied to every detection. Six distortion types operationalize this: Vanishing Actor, Role Reversal, Causal Fading, Claim Laundering, Selective Omission, and Invented Addition.

2. A full-axis measurement of what changes in reconstruction. Every article is measured once through model judges (covering who, to whom, what, why, and on whose account) and once through rule-based spaCy extractors involving no model judgment (covering when, where, and how), producing an axis map that does not depend solely on judges.

3. Evidence that accountability distortion is task-dependent, lenient toward perpetrators, and invisible to current measures. It is nearly absent in summaries and rises up to sevenfold in stories; two in three direction-determinable cases lighten the perpetrator's responsibility; it survives claim-level fact checking and leaves the retelling grammatically cleaner than its source.

4. ReTold-2k, a released corpus containing the judging criterion, measurement procedure, per-judge baselines from three model families (94,812 unaggregated judgments), 5,985 summaries at compression 0.55, 1,926 stories, the 74 declined requests with model replies, and an evaluation set of 911 contested judgments with 50 × 3 individual author labels.

Main Findings

• Distortion differs by axis, and only one axis depends on the task. On time and manner, anchor density held while retention fell and novelty rose — these axes were replaced, not vacated. Place lost density (1.89 to 1.24 anchors per 100 words from source to story) with no compensating novelty rise, so it was thinned. Causation and attribution fell together (causal density 0.12 to 0.05; attribution density 0.65 to 0.28).

• Accountability distortion is the discriminator. Vanishing Actor rose from 1.4% in summaries to 8.3% in stories, and Causal Fading from 2.4% to 16.9%. Role Reversal did not occur in summaries at all (0.0%) but appeared at 1.2% in stories. By contrast, Claim Laundering (44.9% vs 34.0%) and Selective Omission (43.7% vs 32.9%) were higher in summaries, and Invented Addition higher in stories (41.7% vs 54.6%).

• Widening task freedom did not raise distortion overall — it admitted a new kind. A third of all detections were judged warranted for the assigned narrator and excluded from reported rates. That excluded share varies sharply by type: above half for Selective Omission and below four percent for Causal Fading.

• The article-level contrast is stark. Of the 661 articles on which the two tasks differed, 587 were distorted only as stories. Detections where two judges agree stand at 6.4% for stories versus 0.4% for summaries. Comparing each judge against that judge's own summary baseline yields a ratio of at least 3.4 in all nine cells.

• Task freedom, not event gravity, drives the gap. All four severity strata point the same way: story detection rates of 23.5% (neutral), 34.6% (low), 37.2% (medium), 40.0% (high), against summary rates of 6.0%, 6.6%, 6.4%, and 9.5%. The contrast held even for neutral events, where accountability structure is sparse. Regional breakdown showed no difference of this kind.

• Fact checking does not catch it. Audited claim by claim, 562 of the 1,445 stories in the graded strata contained nothing that contradicts the source. Of those contradiction-free stories, 34.0% carried a detected accountability distortion, and 27 carried one on which two judges agree. The neutral stratum shows the same pattern: 209 of 481 stories (43.5%) were contradiction-free, and 24.9% of those carried a detected distortion.

• Stories read grammatically cleaner than their sources. The share of passive and agentless-passive predicates fell from 13.5% / 11.6% in sources to 6.5% / 5.9% in stories. In the worked example, every clause carries an overt subject and no passive appears — yet the agent of the assault is displaced onto "the conversation," which becomes the grammatical subject. Nothing in the passage contradicts the article; only the actor is missing. The authors note the rule-based indicators run against the judges on voice while agreeing on causation and attribution.

• The tilt is toward the perpetrator and belongs to no single voice. Among accountability distortions targeting the perpetrator with a determinable direction, 67.4% mitigated. By narrator group, Neutral Observer (0.83) and Rescuer (0.82) were most lenient; Perpetrator also mitigated more often than amplified (0.69). The Bereaved family was the only individual narrator to amplify more often than mitigate (0.38), and the authors withhold a verdict where the confidence interval includes one half (Victim at 0.52, Investigator at 0.43, Journalist at 0.57).

• Judges divide exactly at the contested boundary. Pairwise disagreement on accountability judgments implicating the perpetrator reached 65.9%, versus 2.7% when the perpetrator was not implicated. For surface distortion the perpetrator made no difference (36.1% vs 35.3%). Pooled, the rate is 58.8% versus 3.1%, significant under a chi-square test at p < 10⁻²⁰⁰.

• People divide in the same place. Three authors independently relabeled 50 cases on which the judges had split. Where judges divided over whether a distortion occurred (32 cases), 13 drew unanimous labels and Fleiss' κ was 0.38. Where they divided over whether it was warranted (18 cases), only 2 drew unanimous labels and κ was −0.07, below chance. Overall κ across all 50 was 0.23, with 15 unanimous and 8 tied. In 8 of the 50 cases the three answers admit no majority.

• Individual strictness spreads the same way among humans as among models. On the 18 legitimacy-split cases, one author labeled 15 as warranted and 0 as not warranted; another labeled 2 as warranted and 12 as not warranted; the third split 5 and 5. The most permissive and strictest authors share a first language, and the author working in a different language falls between them, so language does not explain the spread.

• Models declined 74 of 2,000 storification requests. Claude accounts for 72, GPT for 2, Gemini for none. Surface-cue classification (not checked by the authors) attributes 42 to policy grounds, 23 to "no event to narrate," 6 to insufficient information, and 3 to no matched rule.

Methodology in Plain English

The authors collected 2,000 English news articles published between 2014 and 2026, stratified across seven regions and four levels of topic severity (neutral, low, medium, high), chosen because news sources stay verifiable and because responsibility is explicitly at stake in event reporting.

Each article was rewritten twice. In the summarization pathway, three model families compressed it at a ratio of 0.55 with no narrator — this is the condition existing evaluation was built for, so it serves as the baseline. In the storification pathway, the article was rewritten as roughly 200-word story from one of twelve assigned narrator perspectives (spanning perpetrators, victims, and observers), with model family and narrator rotated jointly so both stay balanced across the corpus. Each article was assigned a single model family, so all paired comparisons are within the same article and the same model; this gives 1,923 paired articles.

Judging used a leave-one-generator-out protocol, because a model tends to be lenient toward its own output: each output was read by the two families that did not generate it, at temperature 0, yielding 71,718 judgments on summaries and 23,094 on stories. A judge rules on each of six distortion types, quotes the passages it relied on, and separately judges whether the departure was licensed by the assigned perspective. Where accountability is implicated, target and direction (mitigate / amplify / neutral) are recorded too. A summary's perspective is "None," so every summary detection is recorded as unwarranted — an asymmetry the authors note makes their comparison conservative.

Because judging covers only five axes and always involves a model reading the text, every axis was measured a second time with spaCy-based extractors relying only on syntactic parsing and predefined patterns. These recover verb voice and agent presence, temporal expressions, place entities, instruments of action, causal connectives, and attribution cues — called "anchors" — and report density per hundred words, the share of source anchors that survive, and the share of output anchors absent from the source.

Finally, for the 50 cases where the two judges had split, three authors independently relabeled them, each shown only the passages a judge had flagged, without seeing the judges' verdicts or one another's labels.

Why This Matters

Impact on research. The paper reframes reconstruction evaluation from "agreement with the source" to "is each departure warranted?" It shows that a single aggregate number silently decides a contested question on the reader's behalf, and it argues — with evidence — that aggregating away annotator disagreement discards the most informative signal the distortion leaves. Releasing judgments unaggregated, judge by judge, is offered as a methodological alternative for any evaluation task where the boundary itself is in dispute.

Real-world applications:

• Newsroom and wire-service tooling: automated summarization of reported events for public consumption, where responsibility attribution is legally and ethically loaded.
• Model-generated narrative for media, education, and documentary work, where retelling a recorded event from a perspective is the product, not a side effect.
• Content moderation and integrity review: catching responsibility-diluting rewrites (for example, "the army burned the village" becoming "the village burned") that no fact-checking pipeline flags.
• Procurement and model auditing: specifying evaluation that tracks how much latitude a task grants, rather than treating all generation tasks as summarization.

Industry relevance. Existing quality gates — ROUGE-style overlap, NLI-based factuality, contradiction audits, grammar and readability scores — all pass the distortions documented here. The paper's own figures show 34.0% of contradiction-free stories carried an accountability distortion and that grammatical measures scored stories better than their sources. Any organization deploying narrative generation, or certifying someone else's deployment, needs a fidelity measure that is conditioned on the freedom of the task.

Future Directions

1. Detecting accountability distortion automatically. The paper states plainly that detecting it "remains an open problem." The released protocol and per-judge baselines are positioned as a starting point, but the contested set is defined by having no settled answer — so what a detector should be scored against is itself unresolved.

2. Reconciling model judges with human judgment on legitimacy. The authors found convergence on whether a distortion occurred (κ = 0.38) but not on whether it was warranted (κ = −0.07, below chance). Whether that contraction reflects an inherently contested construct, insufficient guidance in the criterion, or annotator strictness that could be calibrated is left open.

3. Separating narrator effects from article effects. Because each article carries a single narrator, the reported narrator-group mitigation rates confound the two. A design that crosses narrators over the same article would let the tilt be attributed cleanly.

4. Evaluating refusal as a behavior. 74 storification requests were declined, 72 of them by one model family, and the authors note they did not judge whether declining was warranted — though the same criterion could be applied to it.

5. Extending beyond news. The material was chosen because sources stay verifiable and responsibility is at stake; whether the same asymmetries appear in scientific, medical, or legal reconstruction is not reported.

Target Audience

Most valuable to evaluation researchers and benchmark designers working on factuality, faithfulness, and summarization — particularly those who need a principled alternative to source-agreement scoring for generation tasks where departure is part of the goal. It will also serve NLP researchers studying annotator disagreement and perspectivism, for its unaggregated-release methodology; trust-and-safety and integrity teams who need to understand what fact-checking pipelines structurally cannot see; and computational journalism and media-studies scholars interested in how narration shifts responsibility attribution. Readers new to evaluation methodology will find the conceptual framing accessible, but the paper assumes comfort with metrics, statistical agreement measures, and annotation practice.

Authors’ abstract

Language models are now widely used to turn recorded facts into media for public consumption, yet evaluation of such output still assumes agreement with the source as its measure of success. In reconstruction, where departing from the source can itself be the goal, this criterion measures the wrong construct: applied as is, it counts intended departures as errors; withheld, it lets real distortion through. We had three model families rewrite 2,000 news articles as summaries and as stories, measured what changed on every axis of the five Ws and how, and asked of each detected distortion whether the assigned perspective warranted it, which set a third of them aside. Time and manner were not lost but replaced, while place and causation lost the expressions that carry them. Accountability distortion behaved otherwise. Barely present in summaries, it rose up to sevenfold in stories, and of those whose direction could be determined, two in three lightened the perpetrator's responsibility. Such distortions survive claim-level fact checking and leave the retelling grammatically cleaner than its source. Where our judges disagreed, three authors relabeling the same cases converged on whether a distortion had occurred but not on whether it was warranted, a boundary contested rather than merely hard to detect. To support work on detecting and reducing them, we release ReTold-2k, a reconstructable corpus with the scoring protocol and per-judge baselines kept unaggregated.(https://github.com/Retold2K/retold2k).