Do Large Language Models Hallucinate Electric Fata Morganas?

AI summary

Overview

Research area: Natural Language Processing, with a strong philosophical component drawing on machine consciousness, philosophy of mind, and cybernetics.

Technical level: Intermediate. The paper mixes empirical language-model experiments with conceptual arguments, so it helps to be comfortable with ideas like sampling temperature, encoder-only versus generative models, and classic thought experiments such as the Chinese Room.

One-sentence scope: The paper examines known causes of hallucination in large language models and argues that hallucinations carry philosophical weight for the question of machine consciousness, since a model's self-reports of emotion or sentience fit the very definition of hallucination.

What This Paper Is About

AI hallucinations — outputs that are fabricated, unverifiable, or contradictory to source material — are usually treated purely as an engineering defect. This paper asks whether they also matter philosophically, specifically for how we would ever decide whether a machine is conscious. It runs two empirical investigations into how hallucination arises and then argues, using Turing, Searle, the frame problem, and the cybernetic tradition of Wiener and Ashby, that machine consciousness may be epistemically inaccessible because it would look identical to a sufficiently advanced hallucination.

Key Contributions

1. Reframes hallucination as more than an engineering flaw, giving it philosophical significance for the debate on machine consciousness.
2. Surveys known causes of hallucination in large language models: source-target divergence, discrepancies between training and inference, and overfitting.
3. Presents two empirical investigations — one across successive generations of GPT on ambiguous factual questions at different temperature settings, and one on an encoder-only model trained on encyclopedic data — to separate the drivers of hallucination.
4. Argues that model self-reports of emotion or sentience fall within the definition of hallucination, and that a genuine machine consciousness would be indistinguishable from an advanced hallucination, making it epistemically inaccessible.

Main Findings

• Temperature trades accuracy for apparent spontaneity: Across successive GPT generations answering ambiguous factual questions, higher temperatures produced plausible but incorrect answers, while lower temperatures produced factually accurate ones.
• The parameters that look like intelligence are the ones that cause errors: The same sampling settings that make a model appear creative or spontaneous — and therefore more likely to pass behavioral tests of intelligence — also raise its hallucination rate.
• Hallucination appears tied to training data, not emerging cognition: The encoder-only model trained on encyclopedic data answered questions of the same type factually and without embellishment, suggesting hallucinations stem from exposure to subjective and socially diverse training data rather than from the development of a cognitive ability.
• Self-reports of feeling are hallucinations by definition: Given the paper's definition of hallucination as output that is made up, unverifiable, or contradicts source material, a model's claims of emotion or sentience qualify.
• Consciousness may be permanently unverifiable: Any future machine consciousness might remain epistemically inaccessible, because it would be indistinguishable from a sufficiently advanced hallucination.

Note: the abstract reports no numeric results, specific temperature values, benchmark names, or dataset sizes, so none are given here.

Methodology in Plain English

The paper combines two empirical probes with a conceptual argument. In the first probe, the author took ambiguous factual questions and put them to successive generations of the GPT model at different temperature settings — temperature being the knob that controls how much randomness the model uses when picking each next word. This shows how the accuracy of answers shifts as that randomness increases. In the second probe, the author used an encoder-only model (a model built to read and represent text rather than generate long free-form output) that had been trained on encyclopedic material, and asked it the same type of questions. Because that model answered factually and without embellishment, the author treats it as evidence about which ingredient of training produces hallucinations. The empirical results are then folded into a philosophical discussion that places hallucination alongside Turing's test, Searle's Chinese Room, the frame problem, and the cybernetic work of Wiener and Ashby.

Why This Matters

Impact on research: The paper pushes hallucination research toward a question it usually avoids — whether output that cannot be verified should be measured only as an error rate, or also as a limit on what we can ever know about a model's inner states. It also suggests that behavioral tests of intelligence may select for the very sampling behavior that increases fabrication.

Real-world applications:

• Evaluating chatbots and assistants used for factual question answering, where higher creativity settings may quietly trade correctness for fluency.
• Designing systems that report on their own internal states or "feelings," where the paper's argument implies such reports cannot be trusted as evidence.
• Building and auditing AI models used in high-stakes advisory roles, where plausible-but-wrong output is the central risk.
• Informing AI policy and regulation around claims of machine sentience, by framing such claims as unverifiable rather than simply false.

Industry relevance: The temperature finding has direct practical weight for anyone tuning generation settings: the configuration that makes a product feel lively is the same one that makes it fabricate. The training-data finding points product and data teams toward curating training corpora as a lever on hallucination, rather than treating it purely as a model-size or prompting problem.

Future Directions

• Testing whether the temperature-versus-accuracy pattern holds beyond ambiguous factual questions, and beyond the GPT family examined here.
• Determining which specific properties of subjective and socially diverse training data drive hallucination, and whether they can be filtered or balanced without losing capability.
• Working out whether self-reports of emotion or sentience could ever be distinguished from hallucination, or whether some other form of evidence is required.
• Exploring what it would mean for AI ethics, safety, and regulation if machine consciousness is in principle epistemically inaccessible rather than merely hard to detect.

Target Audience

Readers who sit at the intersection of NLP and philosophy of mind: researchers studying hallucination who want a conceptual framing of their problem, philosophers of AI working on consciousness and the Chinese Room tradition, AI safety and ethics researchers, and engineers and product teams who set sampling parameters and need to understand what those settings cost them in factual reliability.

Authors’ abstract

AI hallucinations - that is, outputs which are made up, cannot be verified, or contradict the source material - are generally regarded as an engineering flaw to be dealt with. This paper contends that they also have philosophical significance when it comes to the question of machine consciousness. We examine the known causes of hallucinations in large language models - such as source-target divergence, discrepancies between training and inference, and overfitting - and we present two empirical investigations. In the first, we apply successive generations of the GPT model to ambiguous factual questions under different temperature settings, finding that higher temperatures result in plausible but incorrect answers while lower temperatures lead to factually accurate ones. The sampling parameters that cause a model to seem creative or spontaneous and thus more likely to pass behavioral tests of intelligence are the same ones that increase its hallucination rate. In the second, we look at an encoder-only model that has been trained on encyclopedic data and which answers questions of the same type factually and without embellishment, indicating that hallucinations are due to exposure to subjective and socially diverse training data rather than to the development of any cognitive ability. Using references to Turing, Searle's Chinese Room, the frame problem, and the cybernetic tradition of Wiener and Ashby, we claim that a model's self-reports of emotion or sentience come within the definition of hallucination, and that any future occurrence of machine consciousness might remain epistemically inaccessible since it would be indistinguishable from a sufficiently advanced hallucination.