The Pulse
Stanford Study Finds 44% of Tested LLM Claims Expired
A Stanford study finds that 44% of 50 tested claims about large language models had expired by September 2026. Owen Queen and James Zou compared findings across 20 models released by eight developers.
AI.info Team · Written with AI · fact-checked ·
Forty-four percent of the large-language-model findings tested in a new Stanford study had expired in newer models, the researchers say. The report, by Owen Queen and James Zou, is dated November 2, 2026. It compares 50 published claims across 20 models, measuring how findings changed as models were released between July 2024 and September 2026.
Most tested claims weakened or disappeared
The team classifies 22 claims as expired and 17 as weakened. Eight persisted and three strengthened relative to the original findings. The researchers define expiration as an effect that has fallen to one-fifth or less of the paper’s reported effect; their fitted results are measured through September 2026.
Estimated expiration rose with time after publication: 4% of claims had expired by six months, 16% by one year and 39% by two years. The share still at full strength fell from 64% at six months to 47% at one year and 31% at two years. Among the 22 claims with estimated expiration dates, the average time to expiration was 15.7 months and the median was 14.7 months.
That average applies only to claims that had an estimated expiration date. The researchers say their overall estimates have wide confidence intervals because the study covers 50 claims, and they caution that publication comes after the experiments themselves.
Capability failures faded; some preferences held
Many expired findings concerned failures that newer models no longer showed as strongly. The report lists letter counting, arithmetic in other bases, retrieving facts from long contexts, fabricated citations and abandoning correct answers after a user challenges them. Other fading findings involved private details in meeting summaries and nonanswers in Arabic.
Claims about preferences and judgments were more likely to persist or strengthen. The three strengthened findings concerned judges favoring their own answers, praise that tracks a user’s stated preference, and nonuniform random-number generation. The authors summarize their view this way:
“A finding is an observation about particular models at a particular time, not a fixed property of language models.”
Owen Queen and James Zou, Stanford University
The pattern is not uniform across model releases. The study reports that the number of claims in the expiration zone ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol among models released from July through September 2026.
Expired across the panel does not mean gone everywhere
Fabricated citations provide the study’s clearest example of change—and of exceptions. Under the researchers’ literature-review prompts, Claude Sonnet 5.5 fabricated 1% of its references and GPT-6.1 Sol fabricated none. Qwen3.7-Flash, however, fabricated 69% in the same test, even though the claim was classified as expired across the panel.
The report says the outcome counted references to nonexistent works, not errors in the details of real publications. Its reproduction used 42 topics; GPT-3.5 Turbo fabricated 59% of 210 references, close to the original study’s reported 55%.
The researchers call for dated, reproducible claims
The authors argue that findings about model behavior should identify the model, version and date, rather than present a result as a fixed property of language models. They also recommend recording prompts and analysis code so later researchers can repeat tests on newer systems.
The concern reaches beyond research summaries: the report notes that ICLR 2026 reported desk-rejecting papers with hallucinated references, and ICML 2026 lists such references among its grounds for desk rejection. The authors caution that fabricated citations appear less frequently in most of the newest models they tested, while the Qwen3.7-Flash result shows how sharply that behavior can still vary by model.