Evaluating LLM Story Generation through Large-scale Network Analysis of Social Structures

AI summary

Overview

Research area: Natural Language Processing — evaluation of large language model creativity, specifically quantitative analysis of narrative social structure using signed character networks.

Technical level: Intermediate. The paper is readable without deep graph-theory background, but familiarity with network metrics (density, clustering coefficient, assortativity) and basic statistics helps.

Scope: The paper proposes and demonstrates a scalable, automated method for evaluating LLM story generation by extracting character co-occurrence networks from short stories and comparing their structural properties against human-written fiction.

What This Paper Is About

Evaluating how creative or "human-like" an LLM's storytelling is normally requires slow, expensive human judgment, which does not scale to thousands of outputs. The authors ask whether the underlying social structure of a story — who interacts with whom, and whether those interactions are positive or negative — can be captured as a graph and used as an automatic proxy for narrative quality. They then compare over 1,200 LLM-generated and human-written science-fiction stories to see whether measurable differences emerge.

Key Contributions

1. A scalable framework for quantitative network analysis of AI-generated narratives, which surfaces underlying tendencies in LLM story generation without human annotation.
2. The first application of character network analysis to LLM-generated stories, according to the authors; prior character-network work focused on human-written novels, legends, movies, and comics.
3. Large-scale comparative evidence from more than 1,200 stories that LLM-generated narratives build positive-biased social structures relative to human writing.
4. A validation result: the automated network metrics reproduce conclusions previously obtained through semi-manual plot analysis and human expert creativity assessments, supporting the method as a substitute for some human evaluation.

Main Findings

• LLM outputs cluster together; human writing spreads out. Violin plots and Wasserstein distances show LLM score distributions occupying a similar range, while Project Gutenberg stories diverge. Human stories have the greatest Wasserstein distances from all LLM stories in almost every metric, whereas LLMs maintain smaller distances with each other. The authors note that human stories have the second largest distances from one another in assortativity mixing.

• LLM stories are denser. Mean density is 0.372 (GPT 4o), 0.470 (GPT 4o Mini), 0.378 (Gemini 1.5 Pro), and 0.338 (Gemini 1.5 Flash), against 0.182 for humans. More characters co-occur in the same narrative units.

• LLMs show a positivity bias. Humans are the only writer with a negative mean average edge weight, at −0.061, versus 0.659 (GPT 4o), 0.235 (GPT 4o Mini), 0.312 (Gemini 1.5 Pro), and 0.374 (Gemini 1.5 Flash).

• Tightly knit character groups. Mean average clustering coefficient is 0.665 (GPT 4o), 0.766 (GPT 4o Mini), 0.709 (Gemini 1.5 Pro), and 0.623 (Gemini 1.5 Flash), compared with 0.485 for humans.

• Assortativity is mildly positive for most models. Mean assortativity mixing stays around 0.05 for most LLMs (GPT 4o: 0.047; Gemini 1.5 Pro: 0.052; Gemini 1.5 Flash: 0.044; humans: 0.012). GPT 4o Mini is the exception at −0.072, which the authors describe as a tendency toward slightly non-homogeneous networks.

• Positive subgraphs are denser than negative ones for LLMs, but the reverse holds for humans. In positive networks, LLM density ranges from 0.315 (Gemini 1.5 Flash) to 0.395 (GPT 4o Mini) with average clustering between 0.531 and 0.589; in negative networks, LLM density is 0.222 to 0.261 with average clustering 0.072 to 0.212. Humans show the opposite pattern: positive-network density 0.294 and clustering 0.259, versus negative-network density 0.313 and clustering 0.395.

• Amiable characters form small worlds. The high clustering of LLM positive networks implies groups of friendly characters repeatedly appear together in positive narrative units, which the authors argue inhibits suspenseful or dramatic plot progression, such as a protagonist group simply progressing through a dungeon.

• Statistical tests separate humans from models, but not always models from each other. Welch's t-tests yield high p-values for several model pairs, including density (Gemini Pro and GPT 4o: p = 0.520), average clustering of positive networks (Gemini Pro and GPT 4o Mini: p = 0.792; GPT 4o and GPT 4o Mini: p = 0.116), and average clustering of negative networks (Gemini Flash and Pro: p = 0.840). Aside from assortativity mixing, p-values for human pairs are consistently p < 0.01 in almost every metric, the exception being density of positive networks with Gemini 1.5 Flash (p = 0.070). Assortativity mixing is the metric where some human pairs also show high p-values (Gemini Flash and GPT 4o: p = 0.852; Gemini Flash and Pro: p = 0.607; Gemini Pro and GPT 4o: p = 0.736; Gemini Flash and Humans: p = 0.165; GPT 4o and Humans: p = 0.122).

• Headline conclusion: in this science-fiction corpus, LLMs generate stories biased toward positive relationships and devoid of dramatic dynamics compared to humans, aligning with prior human-annotated findings about plot progression and creativity.

Methodology in Plain English

Story generation. Four models — OpenAI GPT-4o and GPT-4o mini, and Google Gemini 1.5 Pro and Gemini 1.5 Flash — each produced roughly 250 science-fiction short stories. A prompt template first asked for a 10-chapter plot and a cast of 19 characters with the chapters in which they appear; the character count was chosen as the average character count in 255 human stories. Each chapter was then requested at 800 words. Sampling used temperature 1, top_p 0.95, and top_k 40, with default settings for models that reject those parameters.

Human comparison corpus. 255 short stories were drawn from a Kaggle dataset of 1,002 Project Gutenberg stories. Genres were classified with Gemini 2.0 Flash, and only science fiction was kept as the most frequent genre. Stories outside roughly 3,000 to 15,000 words were filtered out to match LLM story length. The final analysis used 168 human stories.

Building the networks. Characters are vertices; an edge exists if two characters co-occur in a narrative unit, defined as ⌊0.01 × N⌋ sentences where N is the total sentence count. A narrative unit in LLM stories averages roughly 83 tokens. Each unit is labeled positive or negative by a RoBERTa-based sentiment classifier trained on 15 diverse datasets, with average accuracy 0.93 across those datasets. When two characters co-occur in several units, the logits are averaged and passed through a sigmoid. Edges are binary: −1 for negative, 1 for positive.

Cleaning up characters. Transformer-based named entity recognition (SpaCy 3.8.0, precision/recall/F-score of 0.90) finds character names. Genders are estimated from titles and lists of 2,940 male and 4,987 female names. Nickname lists and a customized python-nameparser generate possible referents. Vertices merge when genders do not conflict, when each name appears in the other's referent list, and when titles do not conflict. Ambiguous vertices merge into the more frequent name, and the edge between merged vertices is dropped.

Metrics. Four measures are computed with NetworkX and custom functions: density, average edge weight, average clustering coefficient, and assortativity mixing, the last using a modified weighted average neighbor degree that rewards being the enemy of an enemy. Each network is also split into a positive subgraph and a negative subgraph for density and clustering. Networks with fewer than 10 nodes or density below 0.1 are excluded, leaving 251 GPT 4o, 249 GPT 4o Mini, 252 Gemini 1.5 Pro, 249 Gemini 1.5 Flash, and 168 human networks. Comparisons use Wasserstein distances (scipy.stats.wasserstein_distance) and Welch's t-tests (scipy.stats.ttest_ind).

Why This Matters

Research impact. The paper offers a route around the scalability bottleneck in evaluating creative LLM output. Because the network metrics reproduce conclusions previously reached through human annotation, the approach can be used as an automated, reproducible proxy for at least some qualitative narrative judgments, and it opens a new lens — structural social dynamics — for studying model behavior.

Real-world applications.

• Automated quality and diversity checks in AI writing assistants, flagging output that is uniformly positive or lacks conflict.
• Benchmarking suites for creative-writing models, complementing perplexity, human preference, and creativity tests.
• Narrative design tooling for games, film, and interactive fiction, where character relationship graphs can diagnose flat or conflict-free story arcs.
• Media and entertainment pipelines that screen large volumes of generated scripts or drafts before human editorial review.

Industry relevance. Any organization deploying LLMs for content generation — publishing, entertainment, education, marketing — can use this kind of structural profiling to detect systematic stylistic limitations, such as a bias toward harmonious, low-tension narratives, without commissioning costly human evaluations for every model update.

Future Directions

• Alternative edge definitions. Replace co-occurrence with conversation, mention, or direct-action networks, optionally using relation extraction models, to test whether conclusions hold.
• Beyond science fiction. Apply the method to other genres, and to interactive narratives, where human input shapes the social structure in real time.
• Larger and longer texts. Analyze more extensive human story datasets and longer LLM outputs, which would yield larger networks suitable for community detection and robustness analysis.
• Temporal networks. Track character networks at multiple points in a plot to study coherence and time-varying social dynamics.
• Methodological fixes. Incorporate coreference resolution to catch characters referred to only by pronoun, and consider neutral labels or continuous edge weights instead of binary weights. The authors also leave open how results shift with narrative unit sizes other than ⌊0.01 × N⌋ sentences, though they expect the ratio of density and clustering scores across writers to stay similar.

The paper lists its limitations explicitly: named entity recognition misses characters never named in a narrative unit, and the authors observe that names appear more often in AI-generated stories (for example, "Mike said…" rather than "He said…"), which may contribute to the higher density of AI original networks. They argue their central conclusion is unaffected because it rests on relationship polarity and on positive/negative subgraph density and clustering, not on original-network density and clustering.

Target Audience

Researchers in computational linguistics, AI evaluation, and computational narratology; practitioners building or benchmarking creative-writing systems; and data scientists interested in applying graph analysis to text. Readers with a general understanding of LLMs and basic network concepts will get the most from it; no deep mathematics is required to follow the argument, though the metric definitions reward some graph-theory familiarity.

Authors’ abstract

Evaluating the creative capabilities of large language models (LLMs) in complex tasks often requires human assessments that are difficult to scale. We introduce a novel, scalable methodology for evaluating LLM story generation by analyzing underlying social structures in narratives as signed character networks. To demonstrate its effectiveness, we conduct a large-scale comparative analysis using networks from over 1,200 stories, generated by four leading LLMs (GPT-4o, GPT-4o mini, Gemini 1.5 Pro, and Gemini 1.5 Flash) and a human-written corpus. Our findings, based on network properties like density, clustering, and signed edge weights, show that LLM-generated stories consistently exhibit a strong bias toward tightly-knit, positive relationships, which aligns with findings from prior research using human assessment. Our proposed approach provides a valuable tool for evaluating limitations and tendencies in the creative storytelling of current and future LLMs.