Beyond World Models: Rethinking Understanding in AI Models

AI summary

Overview

Research area: Philosophy of artificial intelligence / epistemology of AI, specifically a conceptual critique of the "world models" research program.

Technical level: Beginner-Friendly. The paper is conceptual and argumentative, with no experiments, no benchmarks, and no technical machinery beyond basic familiarity with the world-model debate.

One-sentence scope: The paper argues that possessing world-model-like internal representations is an inadequate characterization of human-level understanding, using three case studies drawn from the philosophy of science.

What This Paper Is About

A growing body of AI research claims that models which internally track objects, their states, and the rules governing state transitions have "world models" — and that having such representations indicates the model understands the world in a human-like way. This paper asks whether that inference holds up, and answers that it does not: world models are an advance over surface-level correlation, but they still fail to capture what understanding amounts to across physical reasoning and problem-solving.

Key Contributions

1. A targeted critique, not a capability claim. The authors explicitly do not argue that AI models cannot understand. They attack only the specific claim that possessing world-model-like representations constitutes understanding, leaving open that understanding might arise through other mechanisms.

2. A deliberate refusal to define understanding. Rather than offering a universal or operational definition, the authors argue that any such definition would only shift the problem to its own terms, producing an infinite regress unless some terms are admitted as primitive (citing Popper 1945). They adopt a case-study method instead.

3. Three philosophy-grounded case studies where the world-model conception visibly falls short: Hofstadter's (2007) computer built from falling dominoes, Poincaré's (1914) distinction between verifying and understanding mathematical proofs, and Popper's (1979) account of understanding physical theories through their problem situations.

4. A pre-emptive rebuttal to the "just enrich the states" objection, arguing that allowing arbitrarily rich abstract states would make the world-model framework unfalsifiable and circular.

Main Findings

• State selection can miss the explanation entirely. In Hofstadter's domino chainium, tracking whether each domino is standing or fallen and simulating the physical propagation of falls answers which domino fell, not why the system behaves as it does. When the input stretch contains 641 dominoes and a particular domino never falls, the mechanistic trace merely says its neighbors never fell. The explanation is that 641 is prime — an abstract mathematical property no amount of state tracking reveals.

• Verification is not understanding. Poincaré's distinction, quoted in the paper, is that checking each syllogism of a proof for correctness is sufficient for some, but the majority "want to know not only whether all the syllogisms of a demonstration are correct, but why they are linked together in one order rather than in another." A world model applies to proofs because the Curry-Howard correspondence treats proofs as physical processes of symbolic manipulation, but tracing logical state transitions leaves the ordering appearing, in Poincaré's word, "engendered by caprice."

• Avigad's (2008) criteria expose specific, concrete gaps. Applied to Zagier's (1990) one-sentence proof that every prime p ≡ 1 (mod 4) is a sum of two squares: a world model can verify that the involution f has exactly one fixed point but cannot say why that step is the crucial insight; it cannot explain why constructing the set S = {(x, y, z) ∈ ℕ³ : x² + 4yz = p} was a natural choice; and it cannot supply the overarching strategy of counting the same set in two ways using involutions. The paper notes (citing Karai 2010) that many mathematicians can easily verify Zagier's proof but struggle to understand it.

• Picturing is not understanding. Bohr's (1913) theory modeled as electrons jumping between discrete orbits captures the visualization but not the problem situation: Bohr introduced discrete orbits and quantum jumps specifically to explain the sharp, discrete spectral lines that classical physics could not account for. Popper's point, as the authors relay it, is that someone presented with Bohr's theory without knowing it was invented to explain discrete spectral lines simply will not understand it as a solution to a problem situation.

• The enrichment counterargument is circular. If primality, proof strategies, historical context, and explanatory motivations can all be installed as states with matching abstract transitions, then any phenomenon can be retrofitted into the framework. The explanatory work is then done by the states rather than by the world model's dynamics, reducing the position to "world models can capture understanding if we put understanding into the states."

• The paper credits the research program. It states that world models represent a crucial advance beyond mere surface patterns and calls the world model research program "among current approaches, the most significant component, and the precursor to future theories of machine understanding."

• No quantitative results are reported. The paper contains no benchmarks, datasets, model evaluations, or performance metrics of any kind.

Methodology in Plain English

The authors do not run experiments. They proceed conceptually in three steps. First, they survey how world models are currently defined and studied: internal representations tracking objects, states, and transition rules, investigated in AI via probing techniques such as analyzing learned features in specific layers, studying activation patterns, and using linear decoding to recover world-state representations from internal states. Second, they take the prevailing conception of states used in that literature — discrete, recognizable objects like board squares, cars, or people — and apply it faithfully to three analyzed cases. Third, they show in each case that the resulting world-model account omits the element that the prior philosophical analysis identifies as central to understanding. The cases were chosen deliberately because they are ones where the gap is most pronounced; the authors acknowledge that other perspectives on understanding might align better with world models.

Why This Matters

Impact on research. World models are promoted as a key ingredient for general intelligence, and the connection between having a world model and understanding is asserted in widely cited work — the paper collects quotations from LeCun (2022), Ng (2023), Mitchell (2025a), and Ser et al. (2025) making exactly that link. This paper supplies a conceptual counterweight, arguing that algorithmic and experimental progress on world models needs philosophical scrutiny of the foundational concept it leans on.

Domains the debate bears on. The paper does not enumerate real-world applications, but its arguments land directly on:

• Model interpretability and auditing — probing results such as the Othello-GPT finding (Li et al. 2023) that board state can be linearly decoded, and chess analogues (Karvonen 2024), are read as evidence of world models; the paper asks what such evidence is actually evidence for. Counter-evidence that these representations are collections of learned heuristics rather than clean mental models (Karvonen and others 2024; Nikankin et al. 2024) is also cited.
• Video generation systems — Sora (OpenAI 2024), WorldGPT (Yang et al. 2024), and Veo (Google DeepMind 2025) are called world models because their output appears to follow physical laws and maintain temporal consistency; the paper distinguishes this from internal representations that track discrete states and transitions.
• Automated mathematical reasoning — Hu and Shu (2023) argue world models are needed for mathematical reasoning and propose explicit modeling of intermediate conclusions and simulation of future states; this paper argues verification-style simulation does not deliver proof-understanding.
• Scientific and educational explanations — the Bohr case is directly about what it means to explain a theory, which is the substance of science education and scientific communication.

Industry relevance. The framing has funding and roadmap consequences: if world models are treated as the path to understanding, investment flows toward architectures that simulate states and transitions. This paper argues that evaluation and claims-making should not equate simulation capability with understanding.

Future Directions

• Characterize understanding in AI without the world-model lens. Since the authors argue models might understand via mechanisms beyond or different from world models, an account of what those mechanisms would look like is left open.

• Test whether abstract concepts are represented separately from physical states. The paper's counterargument section raises the empirical question of whether a system actually maintains primality or proof-strategy representations distinct from domino or electron states, or whether such states are only added post hoc.

• Operationalize criteria like Avigad's. Identifying key or novel steps, motivating why steps are natural, and giving a high-level outline of a proof are named as concrete abilities; turning these into measurement targets is a natural next step.

• Address falsifiability of the world-model framework. The paper's central methodological worry — that any failure can be patched by adding richer abstract states — implies the research program needs criteria specifying what would count as evidence against it.

Target Audience

Researchers working on world models, LLM interpretability, and video generation who want a critical framing of what their probing results establish; philosophers of science and of mind interested in machine understanding; researchers on mathematical reasoning and automated theorem proving; and anyone making or evaluating claims about whether current AI systems "understand" the world. The paper is accessible to readers without a technical AI background, though familiarity with the world-model debate helps.

Authors’ abstract

World models have garnered substantial interest in the AI community. These are internal representations that simulate aspects of the external world, track entities and states, capture causal relationships, and enable prediction of consequences. This contrasts with representations based solely on statistical correlations. A key motivation behind this research direction is that humans possess such mental world models, and finding evidence of similar representations in AI models might indicate that these models "understand" the world in a human-like way. In this paper, we use case studies from the philosophy of science literature to critically examine whether the world model framework adequately characterizes human-level understanding. We focus on specific philosophical analyses where the distinction between world model capabilities and human understanding is most pronounced. While these represent particular views of understanding rather than universal definitions, they help us explore the limits of world models.