Correlation Dimension of Auto-Regressive Large Language Models

AI summary

Authors: Xin Du, Kumiko Tanaka-Ishii (Waseda University) — arXiv:2510.21258v1 [cs.CL], 24 Oct 2025

Overview

Research area: Natural Language Processing — specifically LLM evaluation and generation quality, drawing on methods from fractal geometry and dynamical systems theory.

Technical level: Intermediate. The measure itself (a power-law exponent fit to a recurrence curve) is conceptually simple, but the paper assumes familiarity with correlation integrals, embeddings, and LLM inference mechanics.

Scope: The paper proposes correlation dimension — computed from next-token log-probability vectors — as a single metric that captures the long-range structural complexity of text as perceived by a language model, and demonstrates its behavior across pretraining, context limits, hallucination, and text degeneration.

What This Paper Is About

Large language models can produce repetitive, incoherent, or bland text even when their perplexity is low, because perplexity measures only local next-token accuracy and misses long-range structure. The authors adapt correlation dimension, a fractal-geometric measure of self-similarity originally used for chaotic dynamical systems, to quantify how structurally complex a text appears to a given autoregressive model. Their goal is a lightweight, interpretable metric that runs alongside standard inference and reveals model behaviors that perplexity cannot.

Key Contributions

1. A new LLM-based definition of correlation dimension. The authors compute correlation dimension over the sequence of next-token log-probability vectors produced by a model (Eq. 3), using Euclidean distance to define recurrences. This differs from prior work that measured fractal structure over static word vectors or over cumulative log-perplexity series.

2. A demonstration that the measure reveals hidden training and context dynamics. Correlation dimension uncovers a three-stage evolution during pretraining and a two-stage pattern as context length grows — neither visible in perplexity alone.

3. Evidence linking correlation dimension to hallucination. On knowledge-intensive texts, models that recall facts show substantially higher correlation dimension than models that hallucinate, and every model with a dimension below 5.0 in the case study hallucinated.

4. A unified degeneration detector. Correlation dimension is shown to detect repetition, incoherence, and blandness, outperforming perplexity, Zipf/Heap coefficients, Rep-N, Distinct-N, Self-BLEU, BERTScore, and MAUVE in coverage of these degeneration types (Table 2).

Main Findings

• Typical human-language texts converge near 6.5. Across the Stanford Encyclopedia of Philosophy (SEP) dataset, correlation dimensions of well-trained models stabilize near a consistent value around 6.5 as perplexity decreases (Figure 2).

• A reference spectrum exists. Randomly shuffled texts exhibit high correlation dimensions, typically above ten; self-reinforcing processes such as the Polya urn model display values below two; n-gram processes with small n also yield lower dimensions; and programming languages (Python, Java, and C) show a consistent correlation dimension around 5.

• Early models show a dimension around 7. Correlation integral curves for GPT2, Pythia, Falcon3, OpenLLaMA, Yi1.5, and Mamba yield a consistent correlation dimension around 7 on the "Newton's Philosophy" SEP article, while Gaussian random vectors in equivalent dimensional spaces (ℝ^50000) show distinctly different scaling.

• Complexity tracks grammar complexity. Using the Lin-Tegmark probabilistic context-free grammar over {0,1} with Bernoulli parameter q, correlation dimension measured by OpenLLaMA-13B rises from near zero to values above ten as q approaches 0.5, i.e., as predictability drops.

• Context length produces a two-stage pattern. For Pythia-1B, Qwen2.5-1.5B, and Llama3.2-1B, correlation dimension rises sharply from approximately 3 to about 8 as context extends to 32 tokens, then gradually falls to around 6.5 at longer contexts (a context length of c = 1 reduces the model to a bigram approximation).

• Pretraining follows three stages. Across Pythia (12B, 2.8B, 1B, 160M, 14M), OpenLLaMA (3B, 13B), and Amber-7B, correlation dimension shows (1) an initial rapid drop from learning short-range bigram structures, (2) a subsequent increase from capturing longer-range dependencies, and (3) a final gradual decline reflecting improved generalization — all while perplexity falls monotonically.

• Small models fail differently. Pythia-14M and -160M instead show a sudden rise in correlation dimension to around 8 late in training, coinciding with a drop in accuracy on a simple in-context symbol-repetition task.

• Hallucination correlates with low dimension. On the SEP article "process-theism," Falcon3-7B (6.68) and Falcon3-10B (8.49) recalled the obscure names correctly, while Qwen2.5-0.5B/7B/32B (3.32/3.56/4.42) and Falcon3-1B/3B (3.28/3.14) hallucinated. Model size alone was not decisive: Qwen2.5-32B had more parameters than Falcon3-7B but a much lower dimension.

• Repetition detection is semantic, not surface-level. Explicitly repetitive texts score below 2.0 versus around 6.5 for normal SEP texts (Table 3). For 10 Japanese novels, correlation dimension stayed highly consistent between kanji+kana and kana-only scripts (6.44 vs. 6.57; 5.7% mean relative difference), whereas Rep-2 differed by 29.8%.

• Degeneration lowers dimension consistently. On a controlled set of responses to twenty generic questions generated by GPT-4o and scored with Falcon3-10B: normal (5.04), repetitive (3.80, p = 9.5E-7), incoherent (3.96, p = 2.9E-6), and bland (4.51, p = 1.1E-3), all significant under a Wilcoxon signed-rank test. Perplexity's direction of change was inconsistent across these categories.

• Stress tests agree with benchmarks. On random-name completions of roughly 1024 tokens, mean correlation dimension at input length n = 512 correlated strongly with HelloEval text-completion scores (Spearman's ρ = 0.952), from Qwen2-7B-Instruct (HelloEval 5.12, CorrDim 3.54) up to Yi1.5-34B (HelloEval 44.73, CorrDim 8.89).

• The method is cheap and robust. Computation costs O(N²) memory and O(N²Ω) time with N ~ 10^4 and Ω ~ 10^5, but GPU kernel fusion plus vocabulary reduction achieve more than a 10× speedup with zero additional memory overhead beyond standard inference. Under GPTQ and AWQ quantization, the average change across the SEP dataset was below 3% when distances are computed in FP32.

Methodology in Plain English

The researchers treat an LLM's output as a trajectory through a high-dimensional space. At each token position, they record the model's vector of log-probabilities over the entire vocabulary (Eq. 3). Two positions count as a "recurrence" if their vectors fall within a distance threshold ε, and as ε grows more recurrences appear. For self-similar systems, the fraction of recurring pairs S(ε) follows a power law S(ε) ∝ ε^d, and the exponent d is the correlation dimension.

Intuitively, a text whose future stays predictable after a skip has low dimension; a text full of unpredictable turns has high dimension. The authors interpret recurrences as "textual skips" — segments that could be omitted without changing subsequent generation — with small ε capturing local word-level skips and large ε capturing sentence-level ones.

To test whether next-token vectors alone are enough, they compare simple vectors (k = 1) against time-delayed embeddings (k > 1) and find negligible differences, suggesting single-step distributions already encode substantial long-term structure. They then run the measure across many open models, datasets, grammar-generated synthetic text, pretraining checkpoints, and controlled degenerate texts, comparing against perplexity and established metrics.

Why This Matters

Impact on research. The paper offers a metric that bridges local, token-level evaluation and global, structural evaluation — a gap the authors explicitly identify in the introduction. It also reframes degeneration as a collapse from a higher-dimensional trajectory into a lower-dimensional attractor, analogous to a boundary crisis in dynamical systems, which is a genuinely different theoretical lens for LLM behavior.

Real-world applications:

• Hallucination monitoring — flagging when a model is likely fabricating rather than recalling knowledge, since hallucinating models showed correlation dimensions below 5.0.
• Generation quality control — detecting repetition, incoherence, and blandness at inference time, including subtler forms that perplexity and lexical metrics miss.
• Training diagnostics — using the three-stage curve as a signal for whether pretraining is progressing toward generalization or degrading, as in the Pythia-14M and -160M case.
• Model comparison and robustness testing — the random-text stress test and its 0.952 Spearman correlation with HelloEval offer a cheap proxy for long-context generation ability.

Industry relevance. The method requires no extra memory beyond standard inference, runs at inference time, can be integrated into serving infrastructure such as vllm, and remains stable under 4-bit quantization — all conditions that matter for deployment at scale. The authors note that although closed models like GPT-4 do not expose full logits, the measure relies only on log-probabilities already produced during inference and could be exposed through commercial APIs.

Future Directions

1. Formal analysis of estimation properties. The authors list this first, since the measure is currently defined for infinite sequences but estimated from finite ones.

2. Extension to conditional or multi-modal settings. Applying the measure beyond plain text generation.

3. Deployment as an online signal. Using correlation dimension during training for diagnostics and during generation for control.

4. Closing the theory gap on sufficiency. The observation that k = 1 log-probability vectors match higher-order embeddings is empirically supported but not formally explained; the paper notes it mirrors findings in knowledge distillation and points to Appendix E for details.

Target Audience

Researchers and engineers working on LLM evaluation, generation quality, and training dynamics; practitioners who need a cheap, inference-time signal for hallucination and degeneration; and anyone interested in applying dynamical-systems or fractal-geometry tools to language. Readers seeking a purely theoretical treatment should look elsewhere — the paper states there are no theoretical results, and its contributions are entirely empirical.

Authors’ abstract

Large language models (LLMs) have achieved remarkable progress in natural language generation, yet they continue to display puzzling behaviors -- such as repetition and incoherence -- even when exhibiting low perplexity. This highlights a key limitation of conventional evaluation metrics, which emphasize local prediction accuracy while overlooking long-range structural complexity. We introduce correlation dimension, a fractal-geometric measure of self-similarity, to quantify the epistemological complexity of text as perceived by a language model. This measure captures the hierarchical recurrence structure of language, bridging local and global properties in a unified framework. Through extensive experiments, we show that correlation dimension (1) reveals three distinct phases during pretraining, (2) reflects context-dependent complexity, (3) indicates a model's tendency toward hallucination, and (4) reliably detects multiple forms of degeneration in generated text. The method is computationally efficient, robust to model quantization (down to 4-bit precision), broadly applicable across autoregressive architectures (e.g., Transformer and Mamba), and provides fresh insight into the generative dynamics of LLMs.