A Coherence-Based Measure of AGI

AI summary

Overview

• Research area: Evaluation and measurement of artificial general intelligence (AGI), drawing on human psychometrics (the Cattell–Horn–Carroll model), multi-criteria decision analysis, and systems theory.
• Technical level: Intermediate. The mathematics is limited to generalized (power) means and a numerical integral, but the argument assumes familiarity with AGI benchmarking terminology and composite scoring.
• Scope: The paper proposes replacing the arithmetic mean of domain proficiencies with a family of generalized means indexed by a compensability exponent, plus an area-under-the-curve summary, and illustrates it on CHC-style AGI domain scores and on a suite of 17 heterogeneous benchmarks.

What This Paper Is About

Most current AGI evaluations, including the psychometrically grounded framework of Hendrycks et al. (2025), reduce a system's many abilities to a single number by taking the arithmetic mean of its scores across cognitive domains. That averaging rule implicitly assumes strengths in one domain can cancel out severe weaknesses in another, which the author argues is inconsistent with psychometric evidence and with the way bottlenecks work in complex systems.

The paper's goal is to replace this compensatory aggregation with a "coherence-based" one: a generalized mean whose compensability is controlled by a continuous exponent p, together with an integrated area-under-the-curve score (AGI_AUC) that summarizes how well a system holds up as the assumption that strengths can offset weaknesses is progressively tightened.

Key Contributions

1. Conceptual: Identifies compensability as a fundamental limitation of existing AGI measurement practice, and situates the problem within cognitive coherence, systems interdependence, and limiting-factor dynamics.
2. Methodological: Proposes evaluating AGI through the full AGI_p curve across compensability regimes, and through its integral, an AUC-based indicator of the coherence and robustness of a system's capabilities.
3. Illustrative (CHC domains): Applies the framework to published CHC-style domain scores for GPT-4 and GPT-5, showing that coherence-based aggregation exposes imbalances obscured by arithmetic averaging.
4. Illustrative (heterogeneous benchmarks): Applies the same methodology to a suite of 17 benchmarks drawn from the Gemini 3 Pro Model Evaluation Report, demonstrating that coherence-based evaluation reveals unevenness even in narrower, non-CHC task collections.

Main Findings

• Arithmetic mean overstates progress. Under the compensatory p = 1 regime, GPT-4 scores AGI_1 = 27% and GPT-5 AGI_1 = 58%, reproducing the values reported by Hendrycks et al. (2025) and suggesting GPT-5 has passed the halfway mark toward human-level proficiency.
• That advantage shrinks sharply under stricter aggregation. The geometric mean (AGI_0) collapses to 16% for GPT-5 and 0% for GPT-4, and both the harmonic mean (AGI_-1) and the inverse-quasi mean (AGI_-0.5) fall to 0% for both models.
• The AUC measure is in between, and roughly triples across generations. GPT-5 reaches AGI_AUC = 24% versus AGI_AUC = 7% for GPT-4, which the paper reads as greater but still limited robustness.
• Steep curves indicate bottlenecks. AGI_p curves decline steeply for negative p, which the paper attributes to weaknesses such as long-term memory and adaptive reasoning acting as system-wide limiting factors. A system approaching genuine generality would instead show a flatter, higher curve.
• Fixing the weakest faculty helps disproportionately (speculative). In a simulated "GPT-6?" case, raising GPT-5's long-term memory storage (MS) score from 0% to 30% while holding all other domain scores constant raises both AGI_p and AGI_AUC substantially.
• AUC aligns better with independent reasoning benchmarks. GPT-5's AGI_1 = 58% suggests near-human generality, but its AGI_AUC = 24% sits much closer to ARC-AGI-2 results, where GPT-5 (Pro) reaches approximately 18% and GPT-5 (High) about 10%. Similarly, GPT-4's AGI_AUC = 7% is close to its roughly 6% on BIG-Bench Extra Hard, while its arithmetic mean of 27% is not.
• Even the 24% figure is treated as an upper bound. The appendix recalculates the ten constituent Hendrycks et al. (2025) cognitive scores using geometric and unweighted means, finding several domains collapse: auditory processing drops from roughly 60% to near 0%, visual processing from 40% to near 0%, processing speed from 30% to near 0%, and on-the-spot reasoning from 70% (arithmetic) to 19% (weighted geometric) or 5% (unweighted).
• The same pattern appears in the 17-benchmark suite. Aggregation with p = 1 again yields optimistic assessments by masking imbalance, while negative-p aggregation exposes persistent bottlenecks; models with similar averages diverge sharply in coherence, and AGI_AUC gives a clearer ordering of holistic competence.
• Several domains are flatly deficient across models. In the appendix tables, long-term memory storage is 0% in all subdomains for both GPT-4 and GPT-5, and processing speed aggregates identically for both models (AM and WAM of 30.0, GM and WGM of 0.01). Long-term memory retrieval aggregates are also identical across the two models (AM 33.5, WAM 40.2, GM 0.1, WGM 0.3).

Methodology in Plain English

Start with a model's normalized scores (0–100) across n capability dimensions. The conventional AGI score is just their arithmetic mean, written AGI_1.

The paper generalizes this to the generalized mean (also called the power mean), parameterized by an exponent p:

• p = 1 gives the arithmetic mean (strongly compensatory — high scores can cancel out low ones).
• p = 0 gives the geometric mean (moderately non-compensatory).
• p = -1 gives the harmonic mean (strongly non-compensatory).
• As p approaches negative infinity, the measure approaches the minimum score — a strict bottleneck.

A small stability constant, epsilon = 10^-6, is applied to each score before aggregation so that a domain score of exactly zero does not cause the geometric mean to collapse numerically.

Because the extreme regimes are impractical (fully non-compensatory scoring changes only when the weakest domain improves; the harmonic mean can collapse to near zero on uneven profiles), the paper restricts the analysis to p in [-1, 1]. Plotting AGI_p over that interval gives a curve — a diagnostic of coherence — and integrating it with the composite trapezoidal rule over a uniform grid produces the single scalar AGI_AUC, normalized by the width of the interval and expressed as a percentage.

The framework makes no assumption about the underlying taxonomy: it needs only normalized scores that reflect abilities contributing to general-purpose competence. That is what allows the same machinery to be applied both to the ten CHC-inspired domains and to a heterogeneous benchmark suite. To test external alignment, the author compares AGI_AUC values against performance on ARC-AGI-2 (Chollet, 2019; Chollet et al., 2025) and BIG-Bench Extra Hard (Kazemi et al., 2025).

Why This Matters

The paper argues that the arithmetic mean is a fragile indicator of generality because it rewards specialization and conceals structural weakness, and that strict aggregations are excessively punitive. Integrating across compensability regimes is presented as a way to capture both breadth and balance — neither collapsing for incomplete systems nor inflating scores for narrow specialists. This reframes AGI evaluation around coherent sufficiency rather than average proficiency, and links AI measurement to established ideas in psychometrics (interdependence of abilities, the positive manifold), systems theory (limiting-factor dynamics), and multi-criteria decision analysis (non-compensatory aggregation).

Real-world applications (as implied by the paper's framing):

• Model cards and evaluation reports: The paper explicitly recommends that future benchmarks and model cards report both AGI_1 (average proficiency) and a coherence-oriented measure such as AGI_AUC, so that breadth and balance can be disentangled.
• Benchmark and leaderboard design: The AUC framing could be applied to any task collection — symbolic reasoning suites, multimodal assessments, or mixed portfolios such as Humanity's Last Exam and ARC-AGI-2 — to produce coherence-aware summaries.
• Model selection for demanding deployments: Because the measure exposes bottlenecks such as long-term memory storage, it gives a more conservative signal for choosing systems that must sustain competence across many domains rather than excel in a few.
• Capability-gap auditing: The AGI_p curve acts as a visual diagnostic of where performance degrades as compensability assumptions tighten, which can guide where further capability work would most improve holistic competence.

Industry relevance: Frontier model developers, independent evaluation organizations, and enterprise adopters all rely on composite scores to communicate and compare progress. If those composites systematically overstate generality, as this paper argues, then procurement decisions, deployment risk assessments, and public claims about "general" capability inherit that bias. The paper's proposal is deliberately conservative — it lowers headline numbers rather than raising them.

Future Directions

• Broaden benchmark coverage. The paper states that to meaningfully quantify progress toward AGI, benchmarks must span the full spectrum of essential faculties: continual learning, long-horizon reasoning, memory formation, planning, and other fundamental abilities.
• Fix inflation at the domain-score level. The appendix shows that the underlying ten CHC scores themselves inflate when aggregated arithmetically; recalibrating or re-deriving those scores is left open, and the paper treats the resulting AGI_AUC of 24% as conservative yet still optimistic.
• Refine domain weights. The appendix notes that subdomain weights follow fixed allocations from the Hendrycks et al. (2025) framework and that "these weights may be refined in future extensions of the methodology."
• Embrace metric pluralism rather than a single leaderboard number. The paper argues that no single scalar fully captures general intelligence: the arithmetic mean highlights breadth, negative-p regimes reveal bottlenecks, and the full AGI_p curve provides the diagnostic — a multidimensional toolkit rather than one score.

Target Audience

Researchers and practitioners working on AI evaluation and benchmarking, especially those measuring progress toward AGI or designing composite scores; AI safety and capability-assessment teams interested in identifying bottlenecks rather than averages; psychometricians and cognitive scientists interested in how CHC theory is being imported into machine evaluation; and technically literate policy or standards audiences who need to interpret headline AGI percentages with appropriate skepticism. Readers looking for new model architectures or training methods will not find them here — the contribution is purely to measurement.

Authors’ abstract

Recent approaches to evaluating Artificial General Intelligence (AGI) typically summarize a system's capability using the arithmetic mean of its proficiencies across multiple cognitive domains. While simple, this implicitly assumes compensability: exceptional performance in some areas can offset severe deficiencies in others. Genuine general intelligence, however, requires coherent sufficiency: balanced competence across all essential faculties. We introduce a coherence-based measure of AGI that integrates the generalized mean over a continuum of compensability exponents. This yields an area-under-the-curve (AUC) metric spanning arithmetic, geometric, and harmonic regimes, quantifying how robust an evaluated capability remains as compensability assumptions become stricter. Unlike the arithmetic mean, which rewards specialization, the AUC penalizes imbalance and exposes bottlenecks that constrain performance. To illustrate the framework, we apply it to cognitive profiles derived from the Cattell-Horn-Carroll (CHC) model, showing how coherence-based aggregation highlights imbalances that are obscured by arithmetic averaging. As a second, independent example, we apply the same methodology to a set of 17 heterogeneous benchmarks, demonstrating how coherence-based evaluation can reveal unevenness even in narrower task collections. These examples show that the proposed approach offers a principled, interpretable, and stricter foundation for measuring progress toward AGI.