Social Story Frames: Contextual Reasoning about Narrative Intent and Reception

AI summary

Overview

• Research area: Natural Language Processing, specifically computational narrative understanding, commonsense reasoning about reader response, and computational social science applied to online communities.
• Technical level: Intermediate. The paper assumes familiarity with supervised finetuning, model distillation, taxonomy-based annotation, and evaluation metrics like micro/macro F-1 and inter-annotator agreement.
• Scope: The paper introduces SocialStoryFrames (SSF), a 10-dimension formalism plus a two-stage generation-and-classification modeling pipeline for reasoning about the intent and reception of social media stories, validated with human surveys and applied to a 6,140-story Reddit corpus.

What This Paper Is About

When people read stories online, they infer the author's intent, explain and predict events, emotionally respond, and make value judgments about characters and themes. Existing computational work is caught in a tradeoff: deep qualitative studies of single communities do not generalize, while large-scale analyses rely on coarse measures such as storytelling frequency or structural prototypes. The authors build a general formalism, taxonomy, and pair of models to characterize these "extra-textual" reader responses at scale in their community and conversational context.

Key Contributions

1. The SSF-Taxonomy and formalism: A 10-dimension taxonomy of reader response for social media storytelling—overall goal, narrative intent, author emotional response, causal explanation, prediction, character appraisal, moral, stance, narrative feeling, and aesthetic feeling—each refined into fine-grained subdimensions, grounded in narrative theory, linguistic pragmatics, and psychology, and designed to account for community and conversational context.
2. Two distilled models and two tasks: SSF-Generator (Llama3.1-8B-Instruct finetuned with LoRA on GPT-4o outputs) for inference generation, and SSF-Classifier (Llama3.1-8B-Instruct finetuned on GPT-4.1 outputs) for multi-label inference classification, with generation validated by large-scale human surveys and classification validated by expert annotation.
3. SSF-Corpus: A curated dataset of 6,140 Reddit stories with preceding context posts and comments, sampled from a curated subset of the reddit-corpus-small dataset from ConvoKit, in two variants—SSF-Split-Corpus (N=1,778) with train/validation/test splits and SSF-Stratified-Corpus (N=2,250) with 45 stories from each of 50 subreddits.
4. Pilot analyses at scale: Distributional and association analyses of narrative intents, plus a new narrative similarity measure, ssf-sim, used to compare communities and to characterize author-centric versus reader-centric diversity in narrative practices.

Main Findings

• Generated inferences are plausible to human readers: Across survey ratings, ≥94% of ratings were deemed plausible and ≥78% were deemed very or somewhat likely, indicating SSF-Generator effectively infers probable reader responses across diverse contexts.
• Survey scale and filtering: The main survey recruited N=300 U.S. adults via Prolific; 4,239 ratings from N=278 annotators passed quality filtering, and a smaller validation survey for SSF-Classifier yielded 1,665 ratings from N=104 annotators (totaling N=382 participants across the paper).
• The distilled classifier approaches GPT-4.1: SSF-Classifier exceeds, matches, or is within 0.05 F-1 points of k-shot GPT-4.1 for 7/10 dimensions on Micro F-1 and 6/10 on Macro F-1, with all dimensions within 0.1 points of their GPT-4.1 counterparts.
• Best and weakest classification dimensions: Character appraisal and stance both reached 1.00 Micro and Macro F-1 in validation annotation (test: character appraisal 0.99 Micro/0.99 Macro, N=259; stance 1.00/1.00, N=242), while moral showed the weakest test performance (SSF-Classifier 0.65 Micro / 0.62 Macro, versus k-shot GPT-4.1 0.75 Micro / 0.72 Macro, N=100).
• Annotation reliability: Two authors annotating N=50 test examples per dimension produced strong multi-label agreement, with mean Jaccard Index 0.732 and minimum ≥0.517.
• Most common narrative intents: Justifying or challenging a belief was the most common narrative intent at 40%, followed by clarification (14%), emotional release (14%), showing one's identity (10%), and entertaining (10%).
• Goal–intent association: Providing emotional support was strongly associated with the narrative intent of conveying a similar experience (NPMI: 0.35), casting narrative as a mechanism for empathy.
• Narrative similarity differs from semantic similarity: ssf-sim aligned with human judgments of story similarity 74% of the time versus 52% for a Sentence-BERT all-MiniLM-L6-v2 semantic similarity baseline (200 stories; N=50 pairs of pairs; agreement on a subset of N=20 was κ=0.5098).
• Topically different communities can share narrative practices: Despite surface-level topic differences, ssf-sim identified strong similarity between pairs such as r/MakeupAddiction–r/buildapc and r/apple–r/books, while both metrics agreed on dissimilarity between pairs such as r/Christianity–r/buildapc and r/buildapc–r/politics, and on similarity between r/Android–r/apple and r/AskReddit–r/tifu.
• Topical overlap does not imply functional overlap: r/funny frequently engages with content from r/news and r/politics but with a lighthearted rather than persuasive orientation, yielding low ssf-sim values despite topical overlap.
• Communities differ in narrative diversity: Using normalized entropy across dimensions split into author-centric (overall goal, narrative intent, author emotional response) and reader-centric groups, r/Frugal and r/techsupport show low diversity on both axes, r/Fitness shows high author but low reader diversity, and r/worldnews, r/politics, and r/offbeat show low author but high reader diversity; support, gaming, and hobby communities skew toward author diversity while sports, entertainment, and news communities show greater reader diversity.

Methodology in Plain English

The authors start from theory: narrative theory, discourse processing, and psychology suggest distinct kinds of responses readers have to stories, and they turn those into a written taxonomy of 10 dimensions with finer subdimensions. They then sample real social media stories from Reddit, curating 6,140 stories with their surrounding context. For each story, they summarize two kinds of context using GPT-4o: conversational context (the initial post plus up to 5 ancestral parent comments and 5 prior peer comments, approximating what a Reddit reader would likely have seen) and community context (short summaries of each subreddit's stated purpose and norms and values, drawn from a dataset of public subreddit self-descriptions and community guidelines). Two authors checked the consistency and relevance of at least 30 summaries per source type on a 5-point Likert scale, with mean scores of at least 4.

For inference generation, they prompt GPT-4o to produce up to three independent inferences per story–dimension pair, each following a dimension-specific template, and then distill that behavior into a smaller open-weight student model (SSF-Generator) using LoRA finetuning of Llama3.1-8B-Instruct. To check whether the inferences are believable, they run crowd surveys in which participants see a story in context and rate one random inference per dimension on a 4-point Likert scale, with known-implausible inferences mixed in as a quality filter.

For inference classification, zero-shot performance was poor, so they write annotation guidelines, develop a k-shot prompting strategy that samples similar yet diverse examples via maximum marginal relevance, and distill GPT-4.1's outputs into SSF-Classifier, also based on Llama3.1-8B-Instruct. Two authors measure agreement on a subset of 50 test examples per dimension before the first author annotates 100 validation and at least 100 test examples per dimension under a protocol that adjusts counts for label skew and removes extremely rare or nonexistent labels. Finally, they apply both models to the corpus for distributional, association, similarity, and diversity analyses.

Why This Matters

• Impact on research: The paper moves computational narrative work past the depth-versus-breadth tradeoff by pairing a generic, theory-grounded taxonomy with context-sensitive models, and it offers a validated, reusable resource (code, data, and models) for studying reader response rather than only story content or structure.
• Community comparison: ssf-sim shows that narrative practice similarity is not reducible to semantic or topical similarity, which matters for how researchers draw boundaries between online communities.
• Real-world applications:
  • Moderation and community-health analysis, by characterizing how communities frame intent and respond to stories.
  • Platform design research, by revealing which communities show predictable versus highly variable reader reactions.
  • Mental-health and support-community research, given the finding that providing emotional support is strongly associated with conveying a similar experience.
  • Content and marketing analysis, by surfacing how audiences interpret authorial intent and stance rather than only what a text is about.
• Industry relevance: The pipeline shows that small open-weight models distilled from proprietary teachers can approach teacher-level performance at lower inference cost and compute, which is directly relevant to teams that need scalable, reproducible narrative analytics. The authors also gate model access on HuggingFace by requiring users to acknowledge generalizability limitations.

Future Directions

1. How narrative reception practices of communities and their members co-evolve over time.
2. How individuals belonging to multiple interpretive communities navigate potentially contradictory commitments.
3. Which storytelling dynamics foster prosocial outcomes in conversations, such as perspective-taking and learning.
4. Personalization of models to individuals or communities, for example via Direct Preference Optimization on pairwise preference data for inference plausibility.

The paper also flags open questions: the taxonomy assumes each dimension is independent and that readers generate at least one inference per dimension per story, whereas the authors note there is no consensus on interdependencies across reader response dimensions, and theoretical or empirical work on those dependencies could motivate joint, structured, or recursive modeling. They also note the framework's restriction to English conversations, weaker expected generalization to niche or highly idiosyncratic communities, and the need for broader global construct validation of ssf-sim.

Target Audience

Researchers in computational linguistics and NLP working on narrative understanding, reader response, and commonsense reasoning; computational social scientists studying online communities and Reddit-scale discourse; scholars in empirical literary studies and reception theory interested in scalable operationalizations of interpretive communities; and applied practitioners who need validated, context-aware tools for analyzing how audiences interpret stories rather than merely classifying their topics.

Authors’ abstract

Reading stories evokes rich interpretive, affective, and evaluative responses, such as inferences about narrative intent or judgments about characters. Yet, computational models of reader response are limited, preventing nuanced analyses. To address this gap, we introduce SocialStoryFrames, a formalism for distilling plausible inferences about reader response, such as perceived author intent, explanatory and predictive reasoning, affective responses, and value judgments, using conversational context and a taxonomy grounded in narrative theory, linguistic pragmatics, and psychology. We develop two models, SSF-Generator and SSF-Classifier, validated through human surveys (N=382 participants) and expert annotations, respectively. We conduct pilot analyses to showcase the utility of the formalism for studying storytelling at scale. Specifically, applying our models to SSF-Corpus, a curated dataset of 6,140 social media stories from diverse contexts, we characterize the frequency and interdependence of storytelling intents, and we compare and contrast narrative practices (and their diversity) across communities. By linking fine-grained, context-sensitive modeling with a generic taxonomy of reader responses, SocialStoryFrames enable new research into storytelling in online communities.