Can LLMs Reliably Evaluate Themselves? A Probabilistic VC Framework

AI summary

Overview

• Research area: Self-evaluation and confidence calibration in large language models, grounded in statistical learning theory (Vapnik–Chervonenkis dimension and its scale-sensitive extensions).
• Technical level: Intermediate to Advanced. The experimental protocol (generate two solutions, pick one, report confidence) is easy to follow, but the core contribution — extending VC theory to probabilistic predictors — assumes familiarity with generalization bounds and fat-shattering.
• Scope: The paper introduces two new complexity measures, Probabilistic VC (PVC) and Calibration-aware PVC (C-PVC), proves generalization and sample-complexity bounds for them, and applies them to eleven 7–8B parameter models on one new math benchmark and two existing benchmarks.

What This Paper Is About

Existing evaluations of LLM reasoning mostly score final-answer accuracy, which says nothing about whether a model can tell its own correct reasoning from its own incorrect reasoning. The authors ask whether models can reliably choose the better of two solutions they generated themselves, and whether the confidence they attach to that choice matches how often they are actually right. To answer this in a principled rather than ad hoc way, they adapt classical VC-dimension theory to models that output probabilities, producing metrics that combine expressive self-assessment with calibration quality.

Key Contributions

1. Two new complexity measures. The Probabilistic VC (PVC) dimension generalizes shattering to predictors that output a distribution over labels: a set of inputs is shattered at confidence threshold γ if, for every labeling, some predictor assigns the intended label to each input with probability at least γ. The Calibration-aware PVC (C-PVC) dimension adds a tolerance τ, requiring the model's stated confidence to be within τ of its actual success probability on each shattered item.

2. Theory linking these measures to generalization. Proposition 1 establishes that classical VC equals PVC at γ = 1, that PVC is monotone in γ (VC(F) = PVC₁(F) ≤ PVC_γ₁(F) ≤ PVC_γ₂(F) for γ₁ > γ₂), and that C-PVC is monotone in τ and bounded above by PVC (C-PVC_γ^τ₂ ≤ C-PVC_γ^τ₁ ≤ PVC_γ). Theorem 1 gives a sample complexity of order (C/ε²)(d_{γ,τ} + log(1/δ)) for the C-PVC dimension d_{γ,τ}, with simultaneous guarantees on prediction error (≤ ε) and calibration error (≤ ε + τ) holding uniformly over the function class with probability at least 1 − δ.

3. A practical measurement protocol. Because the dimensions are parameterized by γ and τ, the authors integrate over both to obtain a single Volume Under Surface (VUS) score for each metric (PVC-VUS, C-PVC-VUS, SC-VUS), and they operationalize shattering at the level of dataset categories using two measurable quantities: Self-Evaluation Accuracy (SEA) and Calibration Error.

4. An empirical study across eleven models and three domains. The paper reports a systematic trade-off between self-evaluation expressiveness and calibration fidelity, which the authors argue persists across architectures, training paradigms, and reasoning domains.

Main Findings

• A consistent expressiveness–calibration trade-off. Models with high PVC-VUS tend to have larger gaps to their C-PVC-VUS. On the average column of Table 1, s1.1-7B reaches a PVC-VUS of 5.83 with a gap of 1.56, and Qwen2.5-7B-Instruct reaches 5.61 with a gap of 1.52, while JiuZhang3.0-7B has a lower PVC-VUS of 4.79 but the smallest gap at 0.95.

• Calibration leader is not the expressiveness leader. JiuZhang3.0-7B records the lowest average ECE in the table (0.209; per-dataset 0.169 on Math-360, 0.309 on TruthfulQA, 0.149 on CSQA) despite only moderate expressiveness. The authors describe the relationship in Figure 3's right panel as non-monotonic rather than strictly linear.

• Highest expressiveness and best sample complexity go to s1.1-7B. It achieves the top average PVC-VUS (5.83), the top C-PVC-VUS (4.27), the top SC-VUS (727.3), and the lowest average AE in the table (0.378; 0.407 / 0.379 / 0.348 across Math-360 / TruthfulQA / CSQA).

• Instruction tuning appears to help. Instruction-tuned models outperform their base counterparts on both PVC-VUS and C-PVC-VUS; the clearest comparison is Qwen2.5-7B (Pretrain) at 5.39 PVC-VUS and 3.76 C-PVC-VUS versus Qwen2.5-7B-Instruct (SFT+RL) at 5.61 and 4.09.

• Domain-specific strengths. Qwen2.5-7B-Instruct is strongest on factual questions (TruthfulQA PVC-VUS 6.13), Llama-3.1-8B-Instruct is stronger on mathematics (Math-360 PVC-VUS 4.84), and general-purpose models such as s1.1-7B are more even across domains (4.76 / 6.22 / 6.52).

• Calibration may transfer across domains better than expressiveness. JiuZhang3.0-7B keeps relatively small PVC-VUS gaps on all three datasets (0.75 / 1.33 / 0.77), which the authors read as evidence that calibration quality is more domain-portable than raw discriminative power.

• Weakest calibration in the reported averages. Ministral-8B-Instruct-2410 shows the highest average ECE in the table (0.402) and a gap of 1.88, alongside a PVC-VUS of 4.88.

• The trade-off is not a domain artifact. It appears on Math-360, TruthfulQA, and CommonsenseQA alike, which the authors present as consistency with Theorem 1's claim that the framework generalizes.

Methodology in Plain English

Building the test. The authors wrote 360 original math problems spread across 8 domains and stratified by difficulty, specifically to avoid contamination from pretraining data. To check whether their measures work outside mathematics, they added TruthfulQA (factual accuracy under adversarial phrasing) and CommonsenseQA (everyday reasoning), grouping each into 10 broad categories and sampling 240 questions per benchmark.

The self-evaluation task. For each problem, a model generates two distinct candidate solutions. It then reads both and must (a) pick which one it believes is better and (b) report a confidence level. This is a direct test of introspection: the model has no access to an external answer key, only its own output.

Grading. The objectively superior solution is determined by an ensemble of three larger LLMs — Claude Sonnet 3.7, Amazon Nova Premier, and DeepSeek-R1 — to reduce any single judge's bias.

Turning behavior into theory. For each category, the authors compute Self-Evaluation Accuracy (how often the model's pick matches the judge's) and Calibration Error (the absolute difference between the model's average stated confidence and its actual accuracy). A category counts as γ-shattered if SEA ≥ γ; it counts toward C-PVC only if additionally Calibration Error ≤ τ. The estimated dimension is the largest number of categories satisfying this — a direct, measurable analogue of shattering.

Collapsing the parameter grid. Because each dimension depends on γ and τ, single-number comparisons are awkward. The authors integrate each metric over γ ∈ (0,1] and τ ∈ [0,1) to get Volume Under Surface scores. For sample complexity they fix C = 1, ε = 0.1, and δ = 0.05 so all models are compared under identical theoretical guarantees.

Theory side. They treat a single LLM as a function class in three complementary ways: parameter-level randomization (Bayesian), prompt variation with fixed weights, and randomness from stochastic decoding. The first two plus decoding randomness define the practical hypothesis class used in experiments; the proofs relate PVC to fat-shattering via a margin α = γ − 1/2, which lets them import uniform-convergence results for real-valued function classes.

Why This Matters

Impact on research. The paper reframes "can a model judge itself?" as a capacity question with a measurable complexity, rather than a collection of prompt-level heuristics. It also provides a theoretical bridge between the calibration literature (ECE and related metrics) and classical generalization theory, arguing that ECE and actual error alone give an incomplete picture because they do not measure how expressive a model's self-assessment can be.

Real-world applications:

• Autonomous agent pipelines that must decide when to stop, retry, or escalate to a human — a reliable internal correctness signal is the gate for all three.
• Self-improvement loops (rejection sampling, self-training, preference data generation) that depend on the model's ability to rank its own outputs without supervision.
• Safety-critical deployment in medicine, law, or finance, where communicating "I might be wrong" matters more than being right on average.
• Model auditing and selection — PVC-VUS and C-PVC-VUS give procurement and QA teams a way to compare candidates on introspection, not just accuracy.

Industry relevance. For organizations deploying many models internally, the practical lesson is that picking the highest-accuracy or most expressive model may select the worst-calibrated one. JiuZhang3.0-7B's low ECE with moderate expressive power is a useful counterexample for teams who assume capability and self-knowledge rise together. The finding that instruction tuning lifts both metrics, while not explaining the trade-off, is directly actionable for anyone allocating fine-tuning budget.

Future Directions

• Individual-question rather than category-level analysis. The authors list category-level aggregation as a limitation that may obscure fine-grained self-assessment patterns; per-item estimation would sharpen the picture.
• Scale and architecture. All results are on 7–8B models. Whether the trade-off holds for substantially larger or smaller models, and whether the theory's assumptions survive inside real Transformer implementations, is untested.
• Causal work on training methods. The current training-methodology analysis is explicitly observational, so which specific techniques improve self-evaluation — and whether the trade-off can be broken rather than merely measured — remains open.
• Breaking the trade-off, not just documenting it. The paper concludes that the tension "transcends architectural variations, training paradigms, and cognitive domains," which invites the question of whether any objective or decoding scheme can raise calibration without sacrificing expressiveness.

Target Audience

Machine learning researchers working on LLM evaluation, calibration, uncertainty quantification, and self-improvement; statistical learning theorists interested in extensions of VC and fat-shattering theory to probabilistic predictors; and applied practitioners — ML engineers, model-audit and risk teams — who need to choose between models on grounds other than benchmark accuracy. Readers wanting the empirical takeaway can focus on Section 5 and Table 1; readers wanting the guarantees should read Propositions 1 and Theorem 1 with the appendices.

Authors’ abstract

As large language models (LLMs) find increasing use in critical applications, evaluating their ability to assess their own outputs has become crucial. We present a theoretical and empirical framework that examines whether LLMs can differentiate between correct and incorrect solutions while maintaining properly calibrated confidence. Building on classical Vapnik-Chervonenkis (VC) theory, we adapt it to probabilistic predictors through two complexity metrics: Probabilistic VC (PVC), which measures a model's ability to confidently classify across problem types, and Calibration-aware PVC (C-PVC), which additionally demands alignment between confidence scores and actual success rates. Unlike Expected Calibration Error (ECE) and Actual Error (AE), these measures give an integrated assessment of self-evaluation expressiveness and calibration, with sample complexity bounds and generalization guarantees comparable to classical VC theory. We test eleven 7-8B models on 360 original mathematical reasoning problems, TruthfulQA, and CommonsenseQA. Each model chooses between two of its own generated solutions and reports its confidence, with ground truth determined by an ensemble of larger models. The results show a systematic inverse relationship: models with greater self-evaluation expressiveness have weaker calibration. s1.1-7B and Qwen2.5-7B-Instruct achieve the highest PVC-VUS scores, while JiuZhang3.0-7B is best calibrated, with the lowest ECE and the smallest PVC-VUS gap. The trade-off persists across architectures, training paradigms, and domains, pointing to a fundamental challenge in developing self-reflective LLMs, and our framework offers practical tools for diagnosing it.