Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for Nonlinear Narrative Generation

AI summary

Overview

Research area: Natural Language Processing — computational narrative / story generation and evaluation of large language models.

Technical level: Intermediate. The paper uses formal notation (permutations, Kendall's τ, LIS) and statistical testing, but the core idea — separating the order events happen in from the order they are told in — is intuitive.

Scope: A corpus of 83 published narratives with paired chronological/telling orders and story graphs, plus a five-task benchmark on which 19 LLMs are tested for their ability to produce, read, and be judged on nonlinear narrative structure, validated against 3,528 blind judgments from 71 human readers.

What This Paper Is About

Every story has two orders: the order events actually happen (the fabula) and the order a teller presents them (the syuzhet). When these differ, the story is nonlinear — think Memento or Citizen Kane — and much of the effect comes from when the reader is allowed to learn things. Prior story-generation work plans plots and revises drafts, but the telling order is handled tacitly and rarely recorded, so it is hard to even state what makes a story nonlinear. This paper annotates both orders for 83 published works and asks whether LLMs can deliberately tell stories out of order, whether they can read such tellings, whether a structured "story graph" helps them, and whether automated judges agree with human readers about the results.

Key Contributions

1. A paired-order corpus. 83 published works across seven narrative shapes (73 nonlinear, 10 predominantly chronological), each annotated with one shared list of 12–20 event sentences in both chronological and published telling order, a structural label, and a hand-written story graph. Totals: 1,433 event nodes, 1,221 scenes, 857 characters, 3,478 typed edges (CAUSES 1,330; SHARES 1,397; FORESHADOWS 444; REVEALS 307), 657 reader-question threads, 1,459 pre-conditions and 1,688 post-conditions, and 2,504 character/reader knowledge claims. Mean τ of published telling orders is 0.456.

2. A five-task benchmark (T1 cued ordering, T2 free ordering, T3 reconstruction of chronology, T4 shape classification, T5a/T5b 550–750-word story writing with and without a shape cue), with model-free baselines, a "no-story floor" control for reconstruction, and evaluation of 19 models.

3. A story-graph scaffolding study with leave-one-layer-out ablations (G0 no graph, G1 full graph, G2 supplied told order), run on 14 works, two per shape.

4. A human validation study: 3,528 blind judgments from 71 readers across 294 sittings, testing an automated judge (Gemini 3.1 Pro) and the authors' own structural metrics against reader preference.

Main Findings

• Left alone, models tell it straight. Given events with no structural instruction (T2), the pooled told order sits at τ = 0.638 and LIS = 0.796, against a human corpus mean of 0.456. All 19 models are more chronological than the human tellings of the same works, with per-model values from 0.480 to 0.808 (sign test p = 3.8 × 10⁻⁶). Prose is more chronological still (τ = 0.989), and 3 of 8 prose models write every story in chronological order. Only 1.8% of free outputs land in the band of the source work's own shape.

• Asked for a shape, models over-comply. Cued ordering (T1) lands in the human band 47.9% of the time pooled (28.9–60.2% by model, best Gemini 3.1 Pro) versus 8.4% for a random permutation; format is not the bottleneck (99.5% valid permutations, 97.6% scene contiguity). The failures concentrate in two shapes: retrograde scores 1.1% and chronicle 5.3%, because 93.6% of retrograde outputs are the exact reversal (τ = −1) and 88.9% of chronicle outputs the exact identity. Published works of those shapes stop short (τ ∈ [−0.92, −0.43] and [0.62, 0.97]). Published retrograde tellings reverse 94.0% of cross-scene event pairs but 0.0% of within-scene pairs; cued models reverse 98.8% and 98.0%. A model-free baseline — reverse the scenes, play each forward — hits the retrograde band for 6 of 10 works, while the identity hits the chronicle band in 0 of 10. Excluding those two shapes, pooled band hit is 62.1%; judgment-dependent shapes (zigzag, prolepsis, analepsis) reach the band 68–83% of the time, though zigzag has the lowest agreement with the gold telling (0.396).

• Reconstruction survives without the story. T3 pooled τ_rec = 0.743 (0.409–0.892 per model). With the telling deleted, three strong solvers reach τ = 0.849 versus 0.884 with the full telling — the telling is worth only +0.035 τ. Only 7 of 19 models clear that floor, and per-work correlation between conditions is r = 0.837. De-dating events (cutting time anchors from 6.3% to 0.7% of events) moved the floor by only −0.005 τ. The floor accounts for 96.1% of the τ score but only 42.1% of exact reconstructions; exact match rises from 8.5% to 20.2% when the telling is present. No model reconstructs a chronicle exactly (0%).

• Naming the shape is a separate skill. T4 pooled accuracy is 50.2% (best 79.5%, Gemini 3.6 Flash) against a 14.3% seven-way random baseline and 26.5% majority-class baseline. Errors are systematic: 5 of the 6 other classes send 29–46% of their items to analepsis, and achrony is recognized 7.1% of the time. Naming a shape correlates with both writing ability (ρ = 0.717 with T1 band hit) and reading ability (ρ = 0.747 with T3 τ_rec), while those two barely correlate with each other (ρ = 0.088).

• A story graph helps, but barely and mostly via one model. G1 beats G0 under the judge 31–14 with 11 ties (p = 0.016); PDS rises on 5 of 5 components (3.850 to 4.096) and band hit rises from 41.1% to 51.8%. Agreement with the gold telling does not move (0.554 to 0.554), so models do not use the ordering constraints the REVEALS edges fix (5.5% of told-order pairs). The structural gain comes almost entirely from Claude Sonnet 5 (+0.250 band hit; Gemini 3.6 Flash −0.035; the other two none). The preference gain disappears after length adjustment.

• Readers prefer the default to the instructed story. In paired cued-versus-free comparisons, free tellings win 68–19 with 24 ties (n = 111, p ≈ 1.2 × 10⁻⁷); all 8 models lose, and PDS agrees more mildly (3.74 vs. 3.60). Readers chose the free version 76% of the time, called the instructed one harder to follow 81% of the time, and split evenly on endings (0.460). The penalty is not only nonlinearity: cued stories also lose when the requested shape is chronicle (0.32, n = 37, uncorrected p = 0.047) and draw level on retrograde (0.48). Length favours the instructed texts (747 vs. 711 words) and the longer text won 57% of the time; among length-matched pairs the instructed telling wins 30%. Of 186 written comments, 50 name order or difficulty following.

• Readers prefer graph-guided stories to order-constrained ones, and the judge misses it. Readers chose the full-graph story over the supplied told order 72% of the time (181–70, p < 0.0001), with no cost in difficulty (p = 0.69) and a smaller advantage on endings (0.571). 53 of 68 readers, 12 of 14 stories and 3 of 4 generators agree; the length-adjusted rate is 71%. The judge splits that comparison 23–22, even though G2 leads on structure (98.2% band hit). The two conditions differ in more than the graph: G1 fixes 5.5% of told-order pairs while G2 fixes all of them (gold-telling agreement 0.554 vs. 0.998).

• The graph-over-event-list win does not survive length control. Raw preference is 59% (p = 0.0028), but graph texts average 649 words vs. 603, readers chose the longer text 61% of the time, and the word-count coefficient is the largest in the study (+0.67 per hundred words, p = 0.0003). Adjusted for length the rate is 52% (p = 0.61), with the length-matched subset at 54%.

• Every one-layer ablation beats the full graph with readers. Pooled over layers, readers preferred the reduced-graph story 69% of the time (p < 0.0001) and called the full graph's story harder to follow 62% of the time (p = 0.0004); the full graph wins between 0.271 and 0.333 of decisive judgments against each reduction. Under the judge, all five ablations match the full graph on preference (p from 0.50 to 0.83). Caveat: Claude Sonnet 5's full graph wins only 26 of 147 decisive judgments against its own reduced output (18%), while Gemini 3.6 Flash is at chance (47%).

• The judge is a weak proxy. Pooled over the four comparisons, judge and reader majority agree on 0.662 of items where both are decisive (κ = +0.306, [+0.17, +0.43]), and the judge agrees with individual readers on 0.679 of 779 judgments. Item-level agreement is highest on graph versus event list (κ = +0.625) — the comparison with the smallest reader effect — and indistinguishable from none on layer removals (κ = −0.030). On graph versus told order the judge sits at 0.511 and readers at 0.721 (κ = +0.242). Judge and reader PDS on the same 193 texts correlate at r = 0.43, but the judge rates uniformly high (+0.28 above readers), so a PDS level from one judge is not reportable.

• Chronology predicts reading difficulty; band hit works against preference. Across 918 decisive judgments, the text readers called harder to follow is the less chronological of the pair by Δτ = −0.171 (p < 0.0001, item-level r = −0.34). In the 381 decisive judgments where exactly one text falls inside the human band, readers chose that in-band text 34% of the time (p < 0.0001); the effect holds with condition-pair fixed effects (band hit β = −0.58, τ β = +0.70).

• Reader reliability and early disclosure. Readers agree on "better story" (Krippendorff's α = 0.415) but not on the other two questions (0.112 and 0.073), and reproduce their own decisive verdict on unannounced repeats 93% of the time (76% over all repeats). Premature disclosure of the latest-placed reveal occurs in 8.3% of stories without a graph, 8.5% with one, and none when the model is handed the told order (p = 0.125).

Methodology in Plain English

The authors start from a simple formalism: take the events of a story in the order they happen, then represent the published telling as a permutation of those events. Strict chronology is the identity permutation; anything else is nonlinear. A "scene" is a group of events that stays contiguous in both orders, so structural labels describe scene order while the permutation records event order.

They built the corpus by selecting works for structural shape — starting with films and adding novels, plays, musicals and a short story until each of seven shapes had at least ten examples (so the sample measures coverage of the shapes, not their prevalence). For each work they wrote one self-contained event sentence per event, so the same sentences can be read in either order; they then recorded both orders, a label with a rationale, and a story graph whose typed edges encode causal dependencies, world-state constraints, character knowledge, and reader questions. Machine checks reject works that violate constraints, and models see anonymized versions with identifying details replaced.

The benchmark gives models the event list and asks them to reorder it, either with a named shape (T1) or without one (T2); or gives them the published order and asks for the underlying chronology (T3) or the shape's name (T4); or asks for an actual 550–750-word story with (T5a) or without (T5b) a shape cue. Ordering uses temperature 0 and low reasoning effort, prose 0.7. Scoring uses Kendall's τ against chronology (+1 chronological, −1 exact reversal) and a normalized longest increasing subsequence, with a "band hit" awarded when both fall inside the min–max range of the corpus's human tellings for that shape. Generated prose is aligned back to the event list by first mention so the same structural scores apply.

To check whether reconstruction scores come from the telling or from the event descriptions, they also run the reconstruction with the events shuffled into random order (the "no-story floor") and after rewriting the events to strip explicit time cues. To test scaffold value they compare no graph, full graph, and supplied told order, plus five leave-one-layer-out ablations. Finally, they compare a model-written story's two versions pairwise using an LLM judge (each pair judged twice with positions swapped, disagreement recorded as a tie) and the same pairs shown blind to 71 readers, who answer the judge's overall question plus "which was harder to follow" and "whose ending tied things together better," with one repeated swapped pair per sitting to measure self-consistency.

Why This Matters

Impact on research. The paper shows that three things the field tends to conflate — following a structural instruction, resembling published practice, and pleasing a reader — can come apart. A model can satisfy an ordering rule exactly while producing something no published work does, and can land inside the human structural band while readers prefer a plainer telling. The reconstruction control is a methodological warning in its own right: a task that looks like reading comprehension keeps 96.1% of its τ score with the story deleted. The paper argues that reporting only one of these measures can certify a failure as a success, and it offers six protocol practices that need no new annotation.

Real-world applications:

• Creative writing assistants that offer "tell this out of order" controls, where the finding that instructed reorderings are rated less engaging and harder to follow is directly actionable.
• Interactive fiction and games, where controlling when a player learns something is a core design lever and premature disclosure breaks suspense.
• Screenwriting and editorial tools that check whether a planned reveal lands after the event it explains.
• Evaluation of LLM judges in any subjective-quality setting, since the judge here missed reader preferences on two of four comparisons and its ties were not evidence of no difference.

Industry relevance. Any team shipping long-form generation (fiction, marketing narrative, educational content) needs to know that models default to chronology and that instruction-following on structure carries a quality cost. The result that the graph's structural gain came almost entirely from one of four generators is a caution for anyone assuming a prompting scaffold generalizes across model families; the result that Anthropic's Claude Sonnet 5 both gained most from the graph and lost most to its ablations is a caution for anyone reading the aggregate.

Future Directions

• Move the specification from events to scenes. The authors identify this as the most germane open question: published reverse chronology is a rule applied to scenes, whereas their prompts stated it on event ranks, and they did not test whether models find the scene-level shape once scene boundaries are marked.
• Separate entailment from memorization in reconstruction. Events can imply their own order (a man is shot before he bleeds to death), but these works are also recognizable, and de-dating leaves prior knowledge intact. Readers ordered the events of a generated story at only τ = 0.537 where the floor on the same 14 works is 0.833, so something beyond textual entailment is doing work.
• Read graph value against a length-matched order-supplying condition. A scaffold that encodes discourse relations encodes part of the answer, so the authors argue its value can only be read against a condition that supplies the order directly, at matched output length — which their run did not hold fixed.
• Test more judges and more model families. The judge study is one judge and one cohort; the second-model check validated alignment, not preference. The structural gains concentrate in one generator of four, and the per-layer tests are underpowered, so no single graph layer is either singled out or shown to be inert.

Target Audience

Researchers in computational narrative and story generation; NLP evaluation scientists interested in LLM-as-judge validity and human-study design; practitioners building creative-writing or interactive-fiction systems on top of LLMs; and cognitive narratology or

Authors’ abstract

Every story has two orders: the order in which its events happen and the order in which they are told. A story is nonlinear when the two differ. We introduce a benchmark of 83 published works that measures whether language models can tell nonlinear stories. For each work, we annotate one list of events in both orders, together with a story graph of causal dependencies, character knowledge, and the questions the telling raises and resolves. Given events in chronological order, all 19 models tell them, on average, more chronologically than the published works do; asked for a specific structure, they land within the human range about half the time. We then compare a model's stories of the same events under different conditions, using both an automated judge and 71 human readers. Both prefer the chronological story a model writes by default to the nonlinear one it writes on request, which readers find harder to follow in 81% of decisive judgments. Readers also prefer stories written from the story graph to stories constrained to the published telling order (72% of decisive judgments), a difference the judge does not detect. Our results suggest that following an instruction to reorder a story is not yet the same as telling it well out of order.