Telling Speculative Stories to Help Humans Imagine the Harms of Healthcare AI

AI summary

Overview

Research area: Natural Language Processing, specifically LLM-based story generation applied to human-centered AI ethics and healthcare AI safety.

Technical level: Intermediate. The reader needs basic familiarity with large language models, prompting, and evaluation protocols (win rates, LLM-as-a-judge, entropy), but the framework itself is described procedurally and the user study is qualitative.

Scope: The paper proposes a three-stage pipeline that turns consumer-health AI concepts into speculative stories about benefits and harms, then tests in a 45-participant user study whether reading and discussing those stories broadens the harms and benefits people can imagine.

What This Paper Is About

Healthcare AI tools such as stress monitors, wellness trackers, and mental health chatbots are being built faster than they can be regulated, which raises risks of bias, privacy violations, and unequal access. Most existing ethical tooling automates risk detection (for example, generating risk summaries from model cards), which can sideline human judgment about how harms arise and who they affect. This paper instead asks whether automatically generated speculative stories — read alone or discussed in a red-team session with simulated expert personas — help people independently imagine a wider range of potential benefits and harms before a system is deployed.

Key Contributions

1. A human-centered story generation framework. The authors introduce a method that automatically generates context-sensitive user stories grounded in users' identities, behaviors, and needs, rather than generating harms directly or using plot-planning approaches.
2. A three-stage generation pipeline. AI concepts are mapped to structured use-case scenarios, those scenarios are expanded through role-playing and environment trajectory simulation, and the resulting logs are rephrased into short stories.
3. A pairwise evaluation of story quality. The method is compared against a plot-planning baseline across five criteria (Creativity, Coherence, Engagement, Relevance, Likelihood of Harm or Benefit) using both an LLM judge and human annotators.
4. A user study on ethical reasoning. A 45-participant, between-subjects study measures how reading and discussing stories changes the distribution of harms and benefits participants write into a speculative model card.

Main Findings

• Story-driven discussions broaden harm recognition. In the user study, the Control group concentrated on a small set of harm categories — primarily diminished health or well-being (32.3%), service or benefit loss (24.2%), and privacy violations (22.6%) — while both the Story-only and Story conditions distributed responses more evenly across the 17 harm subtypes defined by Shelby et al. (2023). The abstract states that participants who did not read stories focused primarily on privacy and well-being, at 79.1%.
• Entropy increases across conditions. Shannon entropy over harm types rose from 2.329 (Control) to 2.927 (Story-only) to 3.701 (Story). Bootstrap t-tests confirmed Story-only and Story were both significantly higher than Control (p < .001), and Story was significantly higher than Story-only (p < .001). Benefit entropy followed the same pattern: 2.407 (Control) to 3.242 (Story-only) to 3.868 (Story), all significant (p < .001).
• Some categories appeared only with stories. Harm subtypes including cultural harms, political and civic harms, and tech-facilitated violence appeared only in the Story condition. Benefit subtypes including accessibility & disability support, clinician workload relief, and transparency & trust also appeared only in the Story condition.
• The storytelling method wins pairwise comparisons. Against the plot-planning baseline, Storytelling with Gemma reached 89.45% (Creativity), 92.15% (Coherence), 92.75% (Engagement), 85.65% (Relevance), and 96.05% (Likelihood), for an overall average of 91.21%. Storytelling with Llama3 reached an overall average of 89.24%, and with GPT-4o, 65.19%. Baseline averages were 72.76% (Gemma), 69.71% (Llama3), and 50.00% (GPT-4o, the reference point).
• Human judgments align with the LLM judge. Two graduate student annotators evaluated 100 story pairs per model and method. The Storytelling method was preferred over baselines at 88% with Llama3 and 76% with Gemma. Inter-annotator agreement (Cohen's kappa) ranged from 0.619 to 0.729 across models and methods.
• Both pipeline components matter. Removing the environment trajectory mechanism (role-playing without modeling event progression) produced the largest drop; for Gemma, coherence fell by 17.7 and relevance by 12.2. Removing role-playing (sequential event prediction without character perspectives) reduced creativity by 10.0 and engagement by 8.8.
• Outputs are also more diverse. With Llama3, Storytelling reached the highest DistinctL-3 through DistinctL-5 scores (6.104, 6.158, 6.174). Gemma reached 5.863 on DistinctL-2. The environment-trajectory ablation attained the highest Diverse Verbs score (0.988) but lower DistinctL-n scores.
• Participants described the stories as clarifying. Quoted participant responses include "It helped me to understand more" (P9), "The story provides a concrete example of how AI can be harmful" (P7), and "I could not think of [risks] really, but the story shifted my focus to the negative aspect of things which we usually ignore" (P4). Others noted the stories surfaced "the lack of cultural context" (P6) and "masking of feelings" (P3).
• Qualitative coding agreement. Two annotators achieved moderate agreement at Cohen's kappa = 0.4368 for harms and kappa = 0.3968 for benefits.

Methodology in Plain English

The authors built a three-step pipeline.

Step 1 — From AI idea to scenario. They manually collected 38 consumer-health AI concepts from Wired articles, industry product descriptions, and PubMed research papers, spanning mental health, chronic illness management, elderly care, and public health. GPT-4o turned each into a structured model specification, then into ethically sensitive use-case scenarios represented as a 7-tuple: AI capability, intended user, subject, input/usage context, expected benefit, potential harm, and failure trajectory. For each AI concept, ten variations were generated covering different user roles (doctor, nurse, caregiver), settings (rural clinic, hospital, home), patient profiles (adolescent, older adult, multicultural family), and contextual conditions.

Step 2 — Simulating the interaction. The system extended Solo Performance Prompting, in which one LLM internally plays multiple expert roles within a single prompt, by adding a "world agent" that keeps the environment coherent and handles non-dialogue events such as movement, tool use, or sensor activation. Each role produced per turn a thought in brackets, an action in parentheses, and dialogue in plain text. The world agent responded to actions with simulated system output — for example, returning a stress score of 7.3/10 with a cultural flag — and then updated simulation state such as the patient's emotional profile or alert level.

Step 3 — From log to story. An LLM rephrased the simulation logs into a concise five-sentence narrative preserving the main events, role dynamics, and emotional flow.

Evaluation. The baseline was a traditional plot-planning approach in which the model first outlines a plot and then writes the story in one step from the same ethically sensitive seed, using a template that covers the AI's purpose, the people involved, the use context, ethical risks, and how user identity affects harm. Stories were compared pairwise on five criteria using the arena-hard-auto method, with GPT-4o as judge at temperature 0.1 and with randomized pair ordering to remove positional bias. Story diversity was measured with DistinctL-n and Diverse Verbs.

User study. A between-subjects study with 45 IRB-approved participants used a speculative model card task with three conditions: Control (complete the model card directly), Story-Only (read benefit and harm stories first, no discussion), and Story (engage in a red-team discussion with simulated expert personas before completing the model card). Sessions had a pre-survey, the model card task, and a post-survey on usefulness, trust, and engagement. The authors describe it as an exploratory qualitative study focused on recurring themes rather than statistical power.

Why This Matters

Impact on research. The paper argues for a shift from reactive to proactive safety evaluation and positions human speculation, not automated risk detection, as the source of ethical insight. It provides an alternative to fully automated pipelines such as RiskRAG that summarize risks from model cards and incident data but may reduce opportunities for human reflection. It also reframes story generation as language-based world imagination, connecting narrative systems to work on LLMs as text-based world models.

Real-world applications:

• Pre-deployment review of consumer health apps, wearables, and mental health chatbots, where teams could run story-based red-team sessions before release.
• Model card and documentation workflows, where speculative model cards generated after reading stories could complement standard model cards and Risk Cards.
• Participatory design with patients, caregivers, and non-experts, since participants reported that minimal prior expertise was needed to engage with the narratives.
• Regulatory and compliance foresight, given that the authors note the EU AI Act and a U.S. Executive Order emphasize transparency and accountability while regulation remains slow and fragmented.

Industry relevance. Low-barrier prompting approaches such as "vibe coding" let non-experts prototype AI applications rapidly, including through platforms like CareYaya. The paper targets exactly that gap: teams that can build fast but lack structured ways to anticipate harms. The framework is designed for early-stage exploration and is not yet validated for regulated clinical decision support.

Future Directions

• Extending beyond consumer health. The authors explicitly state their scenarios focus on consumer health and were not tested in regulated domains such as clinical decision support, finance, or law, and that the work should be read as demonstrating methodological feasibility rather than direct applicability to clinicians or patients.
• Larger, more diverse human studies. The user study is small and includes mostly participants with technical backgrounds, and it measures only short-term reflection rather than long-term impact on design decisions.
• Stronger evaluation. The pairwise comparison relies on a single LLM judge, which may favor certain writing styles; the authors call for evaluation with multiple judge models to strengthen validation.
• Replacing simulated personas with real stakeholders. The red-team discussions use predefined personas rather than real experts, which enables rapid iteration but does not capture the full range of stakeholder perspectives.

Target Audience

This paper is most useful to HCI and NLP researchers working on human-centered AI, AI ethics and safety practitioners who build risk-assessment tooling, and product or design teams developing consumer health AI who want a lightweight way to surface harms before deployment. It is also relevant to researchers studying automated story generation, since it contributes a world-agent extension to multi-role prompting and reports diversity and pairwise quality metrics. Clinicians, regulators, and policymakers may find the framing useful, but the authors caution that the study does not yet validate the method for regulated or clinical environments.

Authors’ abstract

Artificial intelligence (AI) is rapidly transforming healthcare, enabling fast development of tools like stress monitors, wellness trackers, and mental health chatbots. However, rapid and low-barrier development can introduce risks of bias, privacy violations, and unequal access, especially when systems ignore real-world contexts and diverse user needs. Many recent methods use AI to detect risks automatically, but this can reduce human engagement in understanding how harms arise and who they affect. We present a human-centered framework that generates user stories and supports multi-agent discussions to help people think creatively about potential benefits and harms before deployment. In a user study, participants who read stories recognized a broader range of harms, distributing their responses more evenly across all 17 harm types. In contrast, those who did not read stories focused primarily on privacy and well-being (79.1%). Our findings show that storytelling helped participants speculate about a broader range of harms and benefits and think more creatively about AI's impact on users.