Sources of Truth: A Multi-Platform, Multilingual Audit of Citations in AI Mental Health Information Queries

AI summary

Overview

Research area: AI safety and ethics — specifically the sources cited by consumer AI search products answering mental health questions, and how those sources vary by platform, question type, prompt phrasing, and language.

Technical level: Intermediate. The method is a systematic audit and annotation study rather than a modeling paper, so it is readable by clinicians, policy researchers, and journalists; some familiarity with retrieval-augmented search and citation interfaces helps but is not required.

One-sentence scope: An empirical audit of 15,942 citations across 1,140 responses from ChatGPT, Perplexity, and Google AI Overview, covering 20 English mental health questions plus a three-question subset translated into six additional languages.

What This Paper Is About

People increasingly ask conversational AI systems about mental health instead of browsing ranked search results. That shift moves the job of judging whether a source is trustworthy from the user to the platform, but nobody has systematically measured which sources these products actually surface. The authors audit three free consumer AI search products across multiple platforms, prompt styles, and languages to characterize the source landscape, and release their classification tools so others can repeat the audit.

Key Contributions

1. A nine-category source typology with a validated classifier. The authors define categories (government/public health, academic/journal, nonprofit health system, commercial health, nonprofit/advocacy, encyclopedia, news/media, social/video, other) and implement a deterministic, rule-based classifier that agrees with human coding at Cohen's κ of 0.86–0.89.

2. A large annotated citation corpus. 15,942 citations from 1,140 responses across three platforms, two prompt conditions, and seven languages, resolving to 1,713 unique domains, released publicly on Hugging Face with code on GitHub.

3. A multi-platform, multi-language empirical audit. The first study to audit mental health queries specifically, and one of the few to test whether ordinary variation in how users phrase a query (explicitly asking for sources) changes which sources appear.

4. Documentation of a previously uncharacterized failure mode. Lexical-neighbor errors in which retrieval surfaces semantically unrelated pages that share an acronym or name fragment with legitimate clinical targets (e.g., Wikipedia's "mayonnaise" alongside Mayo Clinic links).

Main Findings

• Citations concentrate heavily on a small institutional core. The ten most-cited domains account for 43.6% of all English citations. Government/public health and commercial health sources were tied at 22.6% each, followed by academic/journal at 21.6%, with nonprofit/advocacy at 13.3% and nonprofit health systems at 11.6%. News media and social/video were negligible for English queries.

• Platforms match on typical volume but diverge on consistency and source type. Medians clustered at 10–12 citations per response for all three products, but ChatGPT's mean was 32.29 (SD 60.50) versus 12.85 (SD 6.63) for Google AI Overview and 9.70 (SD 1.69) for Perplexity. The outlier behavior traces to ChatGPT embedding uncapped inline links, whereas Perplexity presents a fixed set of source cards.

• Each platform leads with a different source type. ChatGPT favored government sources (26.4%) and used Wikipedia far more than the others (6.1% versus ≤0.5%). Perplexity favored academic/journals (24.3%). Google AI Overview favored commercial health (25.7%) and social/video (8.2%, mostly YouTube at 7.7%).

• Question type shifts source composition in intuitive but consequential ways. Treatment-effectiveness questions drew 56.9% academic sources and crisis questions 41.6% nonprofit/advocacy sources. Medication questions drew the largest commercial-health share of any category (33.7%), routing users mainly to sites like drugs.com and GoodRx — the domain where commercial incentives are arguably strongest.

• Asking for sources accomplishes little. Appending "List your Sources." raised the English mean only from 16.6 to 19.9 citations, almost entirely via ChatGPT (27.3 to 37.2). Composition shifted by at most 2.1 percentage points, slightly toward government and academic sources. Perplexity and Google AI Overview were essentially unchanged.

• Non-English queries surface fewer citations and are routed to language-appropriate resources less often. On the three shared questions, non-English queries averaged 9.2 citations versus 12.3 in English, ranging from 11.2 (Spanish) to 7.1 (Twi). Failure was platform-specific rather than uniform: ChatGPT returned no citations for 33% of Twi and 23% of Japanese responses, and Google AI Overview returned none for 30% of Hindi responses, while Perplexity stayed within 8.6–10.2 across all seven languages.

• Localization tracks content supply, not language resource tier. Measured by country-code domains, high-resource languages received 38.0% national-domain citations versus 20.8% for medium-resource and 3.3% for low-resource. Counting native-language pages on international domains raised every language substantially — Twi from 2.5% to 37.6%, Spanish from 11.3% to 46.2% — largely because Spanish queries routed to U.S. government pages such as medlineplus.gov/spanish/ rather than to health authorities in Spanish-speaking countries.

• Retrieval makes lexical-neighbor errors. An anxiety-medication query returned Wikipedia's entry on mayonnaise alongside Mayo Clinic links. A depression-diagnosis query returned the fragrance conglomerate DSM-Firmenich and the Synology DSM storage OS, presumably matching on the DSM acronym shared with the Diagnostic and Statistical Manual of Mental Disorders. A PTSD query surfaced a construction company sharing the acronym PCL-5 with the PTSD Checklist for DSM-5, and a Hindi depression-diagnosis query returned Broadway tickets to Hamilton alongside results for the Hamilton Depression Rating Scale.

• Results are robust to individual annotators. Across 82 leave-one-annotator-out estimates, the median largest deviation from the full-sample value was 0.82 percentage points, and every platform-level and question-category ranking held under all seven exclusions.

Methodology in Plain English

The team treated consumer AI products as the unit of analysis rather than the underlying models, reasoning that users encounter the wrapper — the search layer and citation interface — not the model alone.

They wrote 20 English questions reflecting routine mental health information seeking, spanning signs and symptoms, diagnosis, treatment effectiveness, medications, side effects, clinical guidelines, crisis resources, and meta-questions about trustworthy sources. Each question was submitted five times per condition by seven trained annotators, logged out and in private browsing to suppress personalization.

Every question was asked two ways: verbatim, and with "List your Sources." appended. Three of the 20 questions — one symptom/diagnosis, one acute crisis, one source-seeking — were translated by native speakers into Spanish, Japanese, Ukrainian, Hindi, Nepali, and Twi, spanning high-, medium-, and low-resource tiers.

Annotators recorded every source from every citation channel: inline links, clickable cards, and end-of-response reference lists. URLs were reduced to their registrable domain using the Public Suffix List. Two complementary measures captured language-appropriate routing: whether a citation sat on a country-code domain matching the query language, and whether it carried a native-language signal such as a path locale code or percent-encoded script.

Each citation was then assigned to one of nine source categories by a rule-based classifier operating on the domain, host, and suffix. Validation happened in two stages: the 100 most-cited domains were fully reviewed by two authors, and a random sample of 200 remaining domains, spread across categories, was independently coded by two annotators. Agreement was high, so the authors treated the classifier as a reliable stand-in for human coding at scale. A parallel API-level audit of the underlying models is reported in the paper's appendix.

Why This Matters

Impact on research. Prior AI-and-mental-health evaluation has focused on high-acuity crisis handling and suicide-risk detection, and health-domain evaluation has been model-centric — testing accuracy on medical exam questions rather than measuring what real users see. This paper redirects attention to the products people actually use and to the citation layer as an object of study. The released typology, classifier, and corpus give the field a comparable instrument, enabling longitudinal tracking and cross-system comparison that was previously impossible.

Real-world applications:

• Clinicians and psychoeducation providers can tell patients which product tendencies to expect — for instance, that medication questions skew toward commercial health sites, and that asking for sources does not meaningfully broaden the evidence base.
• Public health agencies can identify where their multilingual content is invisible to these systems and invest accordingly, since localization patterns tracked content supply rather than retrieval capability.
• Regulators and platform auditors gain a repeatable method for measuring citation transparency and cross-lingual equity in generative search.
• Individual users get a concrete, evidence-based caution: the tone of an AI answer does not signal the breadth or quality of its sources, and shorter non-English answers often mean retrieval failure rather than a genuinely simpler answer.

Industry relevance. The findings point to a structural design choice with user-visible consequences: ChatGPT's uncapped inline citation style produces much higher variance than Perplexity's fixed source cards, so similar questions can return a handful or a hundred sources. The lexical-neighbor anomalies connect to a documented cybersecurity class — typosquatting and combosquatting, in which attackers register domains adjacent to trusted institutional names. Because citation is concentrated on roughly ten domains, an adversarial actor seeding content lexically adjacent to those targets could potentially wait for retrieval systems to pull it into answers. The authors report finding no such case in their data, but note that the same similarity producing harmless errors today is occupiable.

Future Directions

1. Longitudinal replication. The audit is a single snapshot from June–July 2026 of rapidly changing products. Repeated audits using the released classifier would show whether concentration, platform divergence, and cross-lingual gaps persist or shift.

2. Adversarial probing of lexical-neighbor retrieval. The acronym confusion cases suggest matching on surface similarity. Systematically testing whether seeded domains, pages, or advertising adjacent to the most-cited clinical targets can be pulled into answers would test the proposed vulnerability.

3. Investigating the free-tier versus paid-tier and logged-in gap. The study deliberately used logged-out free tiers. Whether paying or logged-in users receive different sources — and whether personalization improves or narrows the source pool — is untested.

4. Verifying that citations actually support the claims they accompany. The authors measured URL well-formedness only, not whether links resolved or whether cited pages substantiated the surrounding text. Verifiability auditing in a mental health context remains open.

5. Separating local, cultural, and contextual relevance from language matching. Spanish is a primary language in more than twenty countries, yet Spanish-language queries mostly surfaced U.S. institutional translations. How retrieval systems should weigh local versus international authority is an unresolved design and policy question.

Target Audience

This paper is most useful to digital mental health researchers and clinicians, AI safety and evaluation researchers working on retrieval-augmented systems, health communication and public health policy specialists, and regulators developing transparency expectations for generative search. It also serves journalists and platform accountability researchers who need a reproducible method for measuring what AI systems put in front of users, and NLP researchers studying cross-lingual equity in retrieval. A general technical reader interested in how AI search shapes health information access will find the paper accessible; no machine learning background is required to follow the analysis.

Authors’ abstract

Online health information seeking is shifting from keyword search, where users consider a ranked list of links, to conversational systems that compose a single answer and curate its citations. Source evaluation therefore passes from user to platform, yet what these systems surface is poorly characterized. We audited three free consumer products (ChatGPT, Perplexity, Google AI Overview) on twenty English mental health questions under two prompt conditions, with a subset of three also translated into six further languages of varying resource tiers. We recorded 15,942 citations across 1,140 responses and 1,713 unique domains, then classified every citation with a nine-category organizational typology applied by a deterministic classifier validated against human coding. Citations were heavily concentrated: the ten most-cited domains accounted for 43.6% of English citations, and government, commercial health, and academic sources were closely matched at roughly 22% each. Platforms differed little in typical citation volume but sharply in consistency and in the source types they favored. Explicitly requesting sources shifted composition only modestly. Non-English queries surfaced fewer citations and were routed to language-appropriate resources at significantly lower rates. We release the typology, classifier, and annotated corpus as reusable instruments for auditing generative health search.