The Geometry of Reasoning: Flowing Logics in Representation Space

AI summary

Overview

Research area: Interpretability and theory of large language models, specifically the geometric/representation-space structure of chain-of-thought reasoning, drawing on differential geometry and concept-space theory.

Technical level: Intermediate. The core intuition (reasoning as a trajectory through a geometric space) is accessible, but the formalism draws on differential geometry (flows, velocity, Menger curvature), representation manifolds, and natural deduction.

Scope: The paper proposes and empirically validates a geometric framework in which LLM reasoning is a smooth flow through representation space whose velocity and curvature are controlled by logical structure rather than surface semantics.

What This Paper Is About

Large language models "reason" by generating a chain of thought, but it is unclear whether that chain reflects an internalized logical structure or merely fluent pattern matching over word statistics. This paper asks whether logic is encoded as a structural invariant inside the model's hidden representations, independent of what the sentences are about or which language they are written in. The authors test this by treating reasoning as a geometric trajectory and measuring how its shape (position, velocity, curvature) reacts when semantics are held fixed but logic changes, and vice versa.

Key Contributions

1. A formal geometric framework that models LLM reasoning as flows or trajectories in representation space, with definitions of concept space, logic space, representation space, flow velocity, and Menger curvature, plus alignment operators connecting them.

2. A purpose-built dataset of 2,430 reasoning sequences that decouples logical form from semantic carrier: 30 natural-deduction skeletons, each instantiated across 20 topical domains and 4 languages (English, Chinese, German, Japanese).

3. The empirical demonstration that raw embedding positions are dominated by surface semantics, whereas first-order (velocity) and second-order (curvature) geometric quantities are dominated by logical structure, validated across Qwen and LLaMA families.

4. A random-shuffle control showing that permuting reasoning step order collapses velocity and curvature similarity while preserving positional similarity, isolating order-dependent logical structure from static content.

Main Findings

• Position encodes semantics, not logic: Zero-order embedding similarity is high for sentences on the same topic or in the same language (up to ~0.89) and low for shared logic (~0.21–0.44), meaning surface content dominates raw representation.

• Velocity encodes logic: First-order differences (step-to-step changes) flip the pattern. Flows sharing a logical skeleton have far higher velocity similarity than flows sharing only a topic or language (e.g., ~0.17 logic vs. ~0.07 topic and ~0.08 language in Qwen3 0.6B).

• Curvature is the strongest logic invariant: Menger curvature similarity separates logic-driven flows most sharply (e.g., ~0.53 logic vs. ~0.11–0.13 for topic and language), suggesting logic constrains higher-order geometry most strongly.

• Order matters: Randomly shuffling the sequence of logical statements collapses velocity and curvature similarity to near zero (0.02–0.04) while leaving position similarity intact, confirming that logical structure is encoded in the order-sensitive dynamics, not in static embeddings.

• Stable across scale and architecture: The same pattern holds across Qwen1.5 0.5B, Qwen2 0.5B, Qwen3 0.6B/1.7B/4B, and LLaMA3 8B, suggesting a general, possibly universal representational law independent of training recipe or model family.

• Challenge to the "stochastic parrot" view: The authors argue that next-token prediction can lead a model to internalize logical invariants as higher-order geometry, which is difficult to explain under a purely surface-form account of language learning.

Methodology in Plain English

The researchers wanted to know whether a model's internal state changes in a way governed by logic or by what the sentences are about. To separate these, they built sentences that share exactly the same logical proof but use completely different words, topics, and languages. They then ran a model step by step on each reasoning chain, recording the hidden-state embedding after each step. This gives a sequence of points — a trajectory — through the model's representation space.

They treat the trajectory as if it were a smooth curve and measure three things: where the points sit (position), how they move from step to step (velocity, the first difference), and how sharply the path bends (curvature, computed using Menger curvature, the reciprocal radius of the circle through three consecutive points). They then compare these measurements across trajectories that share logic but not semantics, versus trajectories that share semantics but not logic. A shuffle control (randomly reordering the reasoning steps) checks that the observed structure really depends on the logical order rather than the presence of the content itself.

Why This Matters

Impact on research: The paper supplies a formal, testable framework for interpretability that goes beyond static probes of individual embeddings. It links chain-of-thought behavior to differentiable geometric quantities, providing a vocabulary (flows, velocities, curvature, controllers) that other researchers can reuse. If the "universal law" claim holds up, it also provides evidence for the Platonic Representation Hypothesis and gives a structural counterargument to the "stochastic parrot" critique.

Real-world applications:

• Steering and alignment: Intervening on a reasoning flow's velocity or curvature could offer finer control over model behavior than single-vector steering, with implications for safety and jailbreak resistance.

• Retrieval and search: Embeddings that respect reasoning-flow geometry rather than plain similarity could improve retrieval-augmented generation, reranking, and document matching where logical progression matters.

• Reasoning diagnostics: Curvature and velocity similarity offer a quantitative signal for detecting when a model is following a valid proof versus shallow pattern matching, useful in evaluation and monitoring.

• Architecture and efficiency: Understanding latent flow dynamics motivates architectures that parameterize reasoning as continuous latent trajectories, potentially enabling more efficient inference than long token-by-token chains.

Industry relevance: Any organization deploying LLMs for math, code, legal reasoning, or any multi-step task has an interest in knowing whether the model is genuinely following logical structure or overfitting to surface cues. This work supplies a measurable signal for that distinction, which matters for reliability engineering, evaluation suites, and trustworthiness reporting.

Future Directions

• Closing the theory gap: The authors deliberately restrict themselves to understanding rather than generation, so questions about why these geometric patterns emerge during training — learnability, training dynamics, and gradient-flow analysis — remain open.

• Extending the framework to other representation components: Beyond logic, learned representations encode tone, persona, discourse, and world knowledge. Disentangling these and modeling their interaction with logical flows is an open problem.

• Testing universality more broadly: The observed invariance across Qwen and LLaMA is suggestive but limited to a handful of models. Larger-scale, multimodal, and non-transformer tests would strengthen or qualify the claim of a universal representational law.

• From geometry to accuracy: The framework does not map its geometric quantities onto task output correctness. Establishing whether flow descriptors can predict or improve reasoning accuracy would turn a descriptive theory into a practical tool.

Target Audience

Interpretability researchers and mechanistic-analysis practitioners who want a rigorous alternative to probing-based accounts of reasoning; theorists in machine learning interested in concept-space and representation geometry; NLP researchers studying chain-of-thought, reasoning evaluation, and cross-lingual generalization; and graduate students or advanced practitioners comfortable with differential geometry who want an entry point into principled LLM analysis. The paper is less suited to readers seeking immediately deployable engineering recipes, since it is explicitly a post-hoc descriptive framework rather than a training or decoding method.

Authors’ abstract

We study how large language models (LLMs) ``think'' through their representation space. We propose a novel geometric framework that models an LLM's reasoning as flows -- embedding trajectories evolving where logic goes. We disentangle logical structure from semantics by employing the same natural deduction propositions with varied semantic carriers, allowing us to test whether LLMs internalize logic beyond surface form. This perspective connects reasoning with geometric quantities such as position, velocity, and curvature, enabling formal analysis in representation and concept spaces. Our theory establishes: (1) LLM reasoning corresponds to smooth flows in representation space, and (2) logical statements act as local controllers of these flows' velocities. Using learned representation proxies, we design controlled experiments to visualize and quantify reasoning flows, providing empirical validation of our theoretical framework. Our findings indicate that training solely via next-token prediction can lead LLMs to internalize logical invariants as higher-order geometry in representation space, challenging the ``stochastic parrot'' argument. Experiments across Qwen and LLaMA model families further suggest the presence of a general, possibly universal, representational law underlying machine understanding and human linguistic regularities, largely independent of specific training recipes or model architectures. Our work serves as both a conceptual foundation and practical tools for studying reasoning phenomena, offering a new lens for interpretability and formal analysis of LLMs' behavior.