Collective Narrative Grounding: Community-Coordinated Data Contributions to Improve Local AI Systems

AI summary

Overview

Research area: Natural Language Processing / Human-Computer Interaction — specifically local-knowledge question answering, participatory design, and community data governance for AI systems.

Technical level: Intermediate. The paper combines accessible qualitative methods (community mapping workshops, storytelling) with a technical pipeline (schema design, retrieval-augmented generation, vector indexes, knowledge graphs), so readers need some familiarity with how retrieval-augmented QA systems are built.

One-sentence scope: The paper introduces a participatory "Narrative Grounding" protocol that turns community stories into structured, governed "narrative units" that can ground local QA systems, supported by a failure taxonomy audit of a county-level benchmark and a small participatory QA baseline.

What This Paper Is About

Large language model question-answering systems answer broad questions well but fail on hyper-local knowledge — a neighborhood's history, a local official, a community festival — creating "knowledge blind spots" that the authors frame as epistemic injustice against marginalized or peripheral communities. The paper's goal is to design and pilot a protocol through which community members themselves collect, structure, validate, and govern the local narratives that these systems need, rather than relying on scraping existing (often sparse or biased) local text. The authors pair this protocol with a quantitative audit of where and how LLMs actually fail on local questions, to show that the errors they target are the dominant ones.

Key Contributions

1. A quantified failure taxonomy for local knowledge gaps. The authors audit model failures on a county-level benchmark and identify eight mutually exclusive error categories, showing that the top four — Factual Knowledge Gap, Cultural Misunderstanding, Geographic Confusion, and Temporal Misalignment — account for 76.7% of all errors and are primarily addressable with locally grounded narratives and better source selection.
2. The Narrative Grounding Protocol. A participatory protocol, developed with N=24 community members across 3 workshops in Atlanta, that specifies how to elicit, transcribe, segment, structure, validate, and provenance-control community stories so they can enter an AI system's knowledge pipeline.
3. The narrative unit schema and system architecture. A defined data schema (narrative_id, author_pseudonym, timestamp, geocode, narrative_text, embedded_claims[], media_links[], verification_status, community_flags[], relationships[]) plus a three-part architecture of Elicitation, Structuring, and Governance that powers retrieval-augmented local QA with inline provenance and a community moderation dashboard.
4. Design tensions and governance implications. The paper articulates three persistent tensions — representation and power, governance and control, and privacy, consent and agency — and draws concrete requirements for retrieval-first, provenance-visible, locally governed QA systems.

Main Findings

• Local knowledge gaps dominate benchmark failures: On LocalBench, a county-level benchmark of 14,782 validated QA pairs from 526 U.S. counties, an audit of 1,000 model failures found that the top four of eight mutually exclusive error categories account for 76.7% of errors: Factual Knowledge Gap (31.8%), Cultural Misunderstanding (23.4%), Geographic Confusion (12.4%), and Temporal Misalignment (9.1%). These failures were stratified by task type, data source, and model, and labeled by two trained judges with consensus adjudication (raw agreement = 87%, Cohen's κ = 0.852).
• Models answer very few participatory local questions without context: On a participatory QA set derived from the workshops, GPT-5 answered 4/20 questions fully correctly, 12/20 partially correctly or vaguely (missing locally salient specifics), and 3/20 incorrectly or with hallucination. The abstract summarizes this as the state-of-the-art LLM answering fewer than 21% of questions correctly without added context. Errors typically involved misidentifying local officials, conflating nearby places, or giving outdated event details. Two annotators — a community validator with local knowledge and a research annotator — labeled independently with adjudication (raw agreement = 84.2%, Cohen's κ = 0.812).
• The missing facts were already in the narratives: In a majority of non-correct cases, the missing facts were already present in the workshop narratives, suggesting that supplying this material should close the dominant error modes for narrative items.
• A stated misinformation vulnerability table: The paper presents a Misinformation Vulnerability Analysis listing Knowledge Gaps (67.3%), Cultural Disconnect (23.4%), Temporal Lag (15.8%), and Source Absence (12.1%), each paired with a way failures are exploited (false claims filling information voids, exploited authenticity deficits, spread of outdated narratives, and "data void" conditions).
• Workshop frictions motivated explicit design principles: The authors report reluctance or hesitation to share stories in workshops, stemming from a mismatch between community and "technocrat" goals and values, and respond with four principles: Explicit Expert-Framing, Physical Scaffolding, Asset-Based Framing, and Ethical Engagement.
• De-identification and utility are in direct tension: The paper states that the more specific a story is, the more useful — and the more re-identifiable — it becomes, and that consent is temporal and situational.

Methodology in Plain English

The authors ran three in-person participatory mapping workshops with 24 residents (ages 18–55) of a single Atlanta community, held at a local university. The centerpiece was a large-scale satellite projection of the neighborhood on a paper-covered table; participants traced the map and marked places of interest, then shared place-based stories in response to story prompts (for example, "Tell me about the last time you visited this area, and what did you experience?") rather than abstract survey questions. Facilitators were framed as limited participants while residents were treated as the "neighborhood expert."

Recorded audio and video were transcribed verbatim, broken into individual narrative units through NLP pipelines, verified by both researchers and community members, and populated into a structured schema capturing entities, geocodes, timestamps, extracted claims, media links, verification status, community flags, and relationships to other units. Units were then returned to the community for validation and consent.

To establish that this work targets the right problem, the authors separately audited model errors on LocalBench, which contains 14,782 validated QA pairs from 526 U.S. counties drawn from structured county indicators (e.g., ACS/USDA/NRHP), local news, and online discussion threads; questions span narrative (non-numerical) and numerical tasks aligned to a Localness framework (physical, cognitive, relational domains), and models were run in closed-book and web-augmented settings, scored with exact/semantic matches, a strict numeric-accuracy rule, an LLM-judge binary correctness metric, and an answer rate for selective refusal. They then built a small 20-question participatory QA set from the workshops and tested GPT-5 on it without added context.

Why This Matters

For research, the paper reframes local QA failure as a data-provenance and participation problem rather than purely a modeling problem, and supplies both a failure taxonomy and a reusable elicitation-and-schema design for studying community-grounded AI.

Real-world applications:

• Local news and civic information systems that can answer questions about a specific neighborhood's events, officials, and history with cited community sources rather than generic or outdated answers.
• Public services and navigation for underrepresented places, where the paper notes rural and low-income communities are underrepresented in information resources used to train AI tools.
• Community archives and oral history projects, which gain a schema and governance model (verification status, dispute, retraction, community flags) for making stories machine-usable without losing provenance.
• Misinformation mitigation, since the paper argues that model failures create "data void" conditions where false claims can fill information gaps.

For industry, the design implication is concrete: retrieval-first, provenance-visible QA with hyper-local citations and community-controlled moderation, rather than fine-tuning on sparse local text or relying on closed-book model knowledge. The authors also describe DAO-inspired consensus mechanisms such as quorum-based approval or reputation-weighted votes, recorded in an append-only log, as a possible governance layer.

Future Directions

• Extending beyond a single-community pilot: The protocol was developed with one Atlanta community and 3 workshops; the authors explicitly note that the design tensions look different across rural vs. urban, multi-ethnic coalitions, Indigenous sovereignty, and immigrant/diaspora contexts.
• Resolving cross-community and internal conflict: Open questions include protocols for coordination before, during, and after workshops, mechanisms for resolving conflicting narratives, and where the boundaries of "the community" begin and end for cross-neighborhood, cross-jurisdiction, or diasporic settings.
• Balancing specificity against re-identification: How to preserve the utility of detailed stories while preventing re-identification, and how to handle consent that is temporal and situational once narratives are indexed, retrieved, and recontextualized by models — especially for groups facing surveillance or stigma.
• Validating the expected retrieval gains: The paper projects higher answer rates, incorrect-to-partial and partial-to-correct conversions on narrative items, and reduced retrieval hallucination from narrative grounding as retrieval-first context with hyper-local provenance filtering, but presents this as an expectation rather than a measured outcome.

Target Audience

This paper is most valuable to HCI and NLP researchers working on local, place-based, or culturally grounded AI; practitioners building retrieval-augmented QA over community or civic data; community organizers, librarians, archivists, and local government staff interested in contributing data under their own governance; and policy-oriented readers concerned with epistemic injustice, data voids, and the power asymmetries of community data collection.

Authors’ abstract

Large language model (LLM) question-answering systems often fail on community-specific queries, creating "knowledge blind spots" that marginalize local voices and reinforce epistemic injustice. We present Collective Narrative Grounding, a participatory protocol that transforms community stories into structured narrative units and integrates them into AI systems under community governance. Learning from three participatory mapping workshops with N=24 community members, we designed elicitation methods and a schema that retain narrative richness while enabling entity, time, and place extraction, validation, and provenance control. To scope the problem, we audit a county-level benchmark of 14,782 local information QA pairs, where factual gaps, cultural misunderstandings, geographic confusions, and temporal misalignments account for 76.7% of errors. On a participatory QA set derived from our workshops, a state-of-the-art LLM answered fewer than 21% of questions correctly without added context, underscoring the need for local grounding. The missing facts often appear in the collected narratives, suggesting a direct path to closing the dominant error modes for narrative items. Beyond the protocol and pilot, we articulate key design tensions, such as representation and power, governance and control, and privacy and consent, providing concrete requirements for retrieval-first, provenance-visible, locally governed QA systems. Together, our taxonomy, protocol, and participatory evaluation offer a rigorous foundation for building community-grounded AI that better answers local questions.