The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Models

AI summary

Overview

Research area: Artificial intelligence / large language model reasoning evaluation, applying topological data analysis (TDA) to the geometry of reasoning traces.

Technical level: Advanced. The paper assumes familiarity with persistent homology, Vietoris–Rips filtrations, Betti curves, persistence landscapes, sequence alignment dynamic programming, and regression diagnostics such as VIF and adjusted R².

Scope: A four-stage empirical framework that embeds LLM reasoning steps, aligns them to expert solutions with a Smith–Waterman algorithm, extracts topological features, and compares those features against graph-theoretic baselines for predicting reasoning quality on AIME problems from 2020–2025.

What This Paper Is About

Evaluating whether an LLM's intermediate reasoning is actually good is currently slow, subjective, and usually reduced to checking only the final answer. Existing automated proxies convert reasoning into directed graphs and measure connectivity, but the authors argue these abstractions are too simplistic for an inherently complex process. The paper proposes instead to treat a reasoning trace as a point cloud in embedding space and to characterize its shape with topological data analysis, testing whether that geometry predicts alignment with expert solutions better than graph metrics do.

Key Contributions

1. A Smith–Waterman alignment pipeline for reasoning traces. The authors adapt the Smith–Waterman algorithm from biological sequence alignment (Smith et al., 1981) to align LLM-generated reasoning steps with expert-written steps in embedding space, producing a mean alignment score and a gold-step coverage metric. This addresses the scarcity of datasets with step-by-step solutions.
2. A topological data analysis framework for quantifying reasoning quality. Vietoris–Rips filtrations and persistence diagrams for homology dimensions k ∈ {0, 1} are converted into three feature families: VR summary statistics, Betti-curve descriptors, and persistence-landscape descriptors.
3. Empirical evidence that topology beats graphs as a quality signal. Across eight model settings, TDA features predict Smith–Waterman alignment score substantially better than graph features, and better in complexity-adjusted terms in 7 of 8 settings.
4. A stable, compact feature set. Because the 28 raw TDA features are strongly collinear, the authors cluster them into 18 clusters and identify four cluster-level effects that are consistently informative across models.

Main Findings

• Topology dominates graphs as a standalone predictor. Across eight model settings, mean R² for TDA-only was 0.236 versus 0.064 for Graph-only; mean adjusted R² was 0.112 versus 0.032. TDA had higher adjusted R² in 7/8 settings.
• Adding graph features to TDA helps raw fit but not complexity-adjusted fit. Mean ΔR² was +14.4%, while mean Δ adjusted R² was −3.4%. In two settings (Qwen3-235B and DeepSeek-r1-7B) adjusted fit decreased after adding graph features.
• Per-model R² values (Table 1, AIME 2020–2025 combined, 180 observations per model). Qwen3-8B: Graph 0.054, TDA 0.273, Graph+TDA 0.312. Qwen3-32B: 0.088, 0.181, 0.233. Qwen3-235B: 0.024, 0.163, 0.167. DeepSeek-r1-7B: 0.047, 0.210, 0.226. DeepSeek-r1-32B: 0.057, 0.190, 0.226. DeepSeek-r1-70B: 0.058, 0.249, 0.300. GPT-OSS-20B: 0.081, 0.296, 0.327. GPT-OSS-120B: 0.101, 0.327, 0.368.
• Four cluster-level topological effects are consistently informative. Cluster 2 (H0 betti_spread) is positively associated with alignment; Cluster 3 (H0 betti_width) is negatively associated; Cluster 12 (H1 betti_width) is positively associated; Cluster 16 (H1 max_birth/max_death) has a weak negative association.
• Interpretation of those effects. Higher-quality traces tend to maintain a coherent main line, include short and varied local checks, and avoid late, large-scale detours.
• TDA explains graph structure selectively. Regressing graph features on TDA gives R² of approximately .35–.38 for average clustering, average path length, diameter, and the small-world index, but only approximately .07 for loop incidence. Global cohesion and efficiency are strongly topological; loop multiplicity is more idiosyncratic.
• H0 governs cohesion, H1 governs cycle-richness. Clustering, path length, diameter, and small-worldness track H0 drivers (centroid, mean life, spread, entropy), while raw loop counts depend more on small connection patterns among a few nodes.
• Severe multicollinearity in raw features. Seven TDA features (H0 landscape max, H0 landscape mean, H0 max life, H1 landscape max, H1 max life) have infinite VIF, and H0 landscape area ranges from 551.4 to 1309.0, motivating the clustering step.

Methodology in Plain English

The researchers work in four stages.

1. Generate traces. AIME problems are parsed into (problem, reference solution) pairs, where the reference solution comes from the AoPS portal. Model reasoning traces are generated with an answer-blind system prompt via a local Ollama endpoint on a Quadro RTX 8000, and stored as JSONL. The prompt instructs the model to act as an expert competition mathematician, present a rigorous step-by-step solution, and end with "Final Answer: <number>".
2. Align model steps to the gold solution. Both the generated trace and the reference solution are split into steps by a rule-based segmenter. Each step is embedded with all-mpnet-base-v2. A local Smith–Waterman dynamic program over cosine similarity then finds the best-matching step pairs, with a gap penalty γ > 0. Two summary metrics come out: mean alignment score and gold-step coverage.
3. Extract topological features. Steps are treated as a point cloud under cosine distance, a Vietoris–Rips filtration is built, and persistence diagrams are computed for H0 (connected components) and H1 (1-cycles). From these the authors derive VR summary statistics (count, total life, max life, mean life, entropy, skewness, max birth, max death), Betti-curve descriptors (peak, location, width, centroid, spread), and persistence-landscape descriptors (mean, max, area).
4. Compute graph baselines. On the same embeddings, following Minegishi et al. (2025), per-trace graphs yield has_loop, loop_count, diameter, average path length, average clustering, and small-world index σ = (C/C_rand)/(L/L_rand).

Quality is then assessed by fitting OLS regressions that predict the Smith–Waterman alignment score from each feature set. Because the 28 raw TDA features are highly collinear, the authors cluster them by correlation structure into 18 clusters (selected via silhouette analysis over N = 1440 observations) and regress alignment on cluster representatives.

Why This Matters

Impact on research. The paper argues that reasoning quality is better captured by higher-dimensional geometric structure than by relational graphs, and offers a label-efficient, automated signal as an alternative to expert rubrics, manual annotation, and slow pairwise judgments. The authors frame the compact, stable feature set as a practical signal for future reinforcement learning algorithms, complementing process-reward methods such as GraphPRM (Peng et al., 2025).

Real-world applications (as suggested by the framing):

• Automated screening of chain-of-thought outputs before a model's answer is trusted in high-stakes settings.
• Process-level reward signals for reinforcement learning that optimizes reasoning steps rather than outcomes.
• Model comparison and diagnostic tooling that reveals whether a smaller model's reasoning differs structurally from a larger model's.
• Detection of plausible-but-unfaithful explanations, a concern raised by Agarwal et al. (2024) and Nguyen et al. (2024).

Industry relevance. Teams deploying reasoning models need cheap, scalable quality checks that do not require human graders per trace. A geometric score computed from embeddings is attractive because it avoids explicit trace graphs (which the authors note are rarely available) and can run on locally hosted models, as demonstrated with Ollama on a single Quadro RTX 8000.

Future Directions

1. Broaden the dataset. The current study rests entirely on AIME, which the authors describe as the only publicly available corpus with step-by-step solution traces for non-trivial problems. They call for curated or annotated datasets in commonsense reasoning, science, programming, and real-world problem solving, noting that even small human-annotated corpora would test whether the patterns generalize.
2. Ground topological events in interpretable operations. The authors want to connect persistence events to human-readable reasoning moves such as opening a branch, running a short check, and rejoining — while remaining graph-free.
3. Stress-test embedding dependence. Because alignment and topology both operate on sentence embeddings with cosine distance, changing the embedder, the segmentation, or the metric can create or remove cycles and shift lifetimes without altering the textual logic. The paper explicitly warns against reading persistence diagrams as faithful maps of the latent reasoning program.
4. Turn the signal into training feedback. The stated motivation is that a compact, stable topological feature set could serve as a practical signal for future reinforcement learning algorithms.

Target Audience

Researchers and engineers working on LLM reasoning evaluation, reasoning faithfulness, and process-level reward modeling; applied mathematicians and data scientists with a topology or TDA background looking for a new application domain; and ML practitioners who need automated, label-efficient quality checks for chain-of-thought outputs. The paper is written for readers comfortable with persistent homology and regression analysis — the main text is accessible, but the appendices contain the full feature dictionary, VIF table, clustering membership, and pseudocode for segmentation, graph building, alignment, and graph analysis.

Authors’ abstract

Evaluating the quality of reasoning traces from large language models remains understudied, labor-intensive, and unreliable: current practice relies on expert rubrics, manual annotation, and slow pairwise judgments. Automated efforts are dominated by graph-based proxies that quantify structural connectivity but do not clarify what constitutes high-quality reasoning; such abstractions can be overly simplistic for inherently complex processes. We introduce a topological data analysis (TDA)-based evaluation framework that captures the geometry of reasoning traces and enables label-efficient, automated assessment. In our empirical study, topological features yield substantially higher predictive power for assessing reasoning quality than standard graph metrics, suggesting that effective reasoning is better captured by higher-dimensional geometric structures rather than purely relational graphs. We further show that a compact, stable set of topological features reliably indicates trace quality, offering a practical signal for future reinforcement learning algorithms.