RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

AI summary

Overview

• Research area: Large language model evaluation and benchmarking (Natural Language Processing).
• Technical level: Intermediate. The method is conceptually simple but rests on Markov chain stationarity, the Perron-Frobenius theorem, damped score propagation, and comparisons against Item Response Theory (IRT) baselines. (The underlying proofs are placed in the paper's appendices.)
• One-sentence scope: The paper introduces Empirical Interaction Propagation (EIP), a non-parametric graph-propagation framework that jointly scores question difficulty and model competency in order to produce difficulty-aware LLM rankings instead of flat accuracy rankings.

A note on naming: the metadata for this entry lists the title as "RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty," while the paper content itself is titled "EIP: Weighted Ranking of LLMs by Quantifying Question Difficulty," and the framework is named Empirical Interaction Propagation (EIP) throughout. This summary uses the names as they appear in the paper content.

What This Paper Is About

Popular benchmarks such as MMLU-Pro and MATH reduce evaluation to accuracy inside topic categories, which implicitly treats every question as equally informative. That masks differences that depend on difficulty and can flip model rankings when the mix of easy and hard items shifts. The paper's goal is to model difficulty explicitly at the item level and use it, together with model competency, to produce finer-grained and more stable comparisons of LLMs.

Key Contributions

1. A new evaluation framework. The authors introduce Empirical Interaction Propagation (EIP), a difficulty-aware, non-parametric approach that jointly estimates question difficulty and model competency, moving evaluation beyond flat accuracy toward difficulty-sensitive comparisons.
2. Comprehensive empirical validation. Experiments on six benchmarks covering 35,550 questions and 30 models show strong alignment with human difficulty judgments (90% agreement), scalability to large evaluation settings, and robustness to changes in the model pool and dataset composition.
3. New insights into models and datasets. Applying EIP at scale reveals distinctive dataset difficulty profiles, consistent family-level patterns across model scales, and the reliability of open-weight and mixed-scale model pools for producing stable difficulty estimates, offering practical guidance for benchmark design and model selection.

Main Findings

• Human alignment of 90%. In a controlled blind trial with 20 human evaluators (V1–V20) judging 70 randomly ordered question pairs, EIP reached 90% agreement with the human consensus. On the "Consensus" column of Table 2, EIP reports 37.1, 70.0, 62.9, 71.4, 67.1, 61.4, 74.3, 64.3, 57.1, and 62.9 for evaluators V1–V9, and 90.0 for V20, versus 62.9 for Simple Rank at V20. EIP agreed with human judgments for 18 out of 20 evaluators compared to Simple Rank, with an average margin of +9.3% in individual alignment scores.
• Model diversity improves difficulty estimates. The mixed "Whole" group achieved 90.0% consensus alignment with human judgments and a Cohen coefficient κ of 0.80, while homogeneous size groups ranged from 38.6% to 64.3% consensus with κ between 0.13 and 0.29.
• Efficiency and scalability. EIP converged on 30 models and 35,550 questions in 0.00597 seconds on consumer-grade hardware (Intel i7), compared with 1782.75 s for 1PL IRT, 3787.03 s for 2PL IRT, and 18.76 s for Multi-IRT (3D). Time complexity is O(tQM); across synthetic response matrices with Q up to 1,000,000 and M up to 2,000, EIP converged in a constant 9 iterations, with per-iteration time growing linearly with Q × M (for example, 2.00 × 10^9 interactions took 5.28 s total, 0.5867 s per iteration).
• Robustness to model-pool changes. Randomly removing k models from the 30-model pool over 50 trials each: removing 1 model gave mean question correlation ρ = 0.9978 (SD 0.0021) and model correlation ρ = 0.9997 (SD 0.0002); removing 15 models (half the pool) still gave question ρ = 0.9382 (SD 0.0187) and model ρ = 0.9934 (SD 0.0069).
• Ranking differs from accuracy in specific cases. EIP scores correlate positively with accuracy (Kendall's τ = 0.8492), but Qwen2-0.5B (20.2% accuracy) is ranked higher than DeepSeek-Chat-Lite (30.49%) because Qwen2-0.5B correctly answered 5.5% of hard questions versus 2.4% for DeepSeek-Chat-Lite. Among top models, Gemini-1.5-Pro receives a higher EIP score than GPT-4o despite GPT-4o's higher overall accuracy, because Gemini-1.5-Pro answered 5% more medium- and high-difficulty questions.
• Dataset-specific difficulty profiles. GPQA has a relatively uniform difficulty distribution. GSM8k, HellaSwag, and BBH skew toward easier questions. MATH emphasizes harder questions, and MMLU-Pro displays a bimodal pattern blending challenging items.
• Families preserve difficulty patterns across scale. Within the Llama-3 family (Llama-3.1-8B, Llama-3.2-3B-Instruct, Llama-3.2-1B-Instruct), overall accuracy shifts from 51.2% to 21.0% to 10.2% and EIP scores from 44.3% to 23.9% to 15.4%, yet the distribution of correct responses across difficulty levels stays similar. The 8B model shows a slight disadvantage in hard-question accuracy (11.1% versus 15.7% and 15.6% for the smaller variants) while maintaining nearly identical easy-question accuracy (62.9% versus 60.4% and 61.5%). The paper reports the same trend for Qwen and Yi families.
• Open-weight pools track full-pool difficulty. Difficulty distributions from open-weight models of all sizes correlate with the full model set at Spearman 0.96, Pearson 0.94, and Kendall 0.85. All-proprietary pools correlate at 0.78, 0.84, and 0.64, a noticeable drop particularly in Spearman correlation.
• Homogeneous pools produce extreme difficulty estimates. The Large group classified 58% of questions as "overly easy," while the Small and Medium groups contained over 30% "impossible" questions. The mixed-scale group reduced extremes to below 3% and reduced dead nodes by 83% compared to homogeneous groups.
• Controlled simulation favors EIP. With 5 hypothetical models (M1–M5) and 100 questions (70 easy, 21 medium, 9 hard) and ground-truth ranking M1 > M2 > M4 > M5 > M3, only EIP matched the ground truth on both test conditions (M1 > M2 correct, and M4 > M5 > M3 correct). Standard accuracy, dataset-difficulty-weighted accuracy, 1PL-IRT, 2PL-IRT, and Multi-IRT each failed at least one condition. EIP's simulated difficulty scores were 18.24 (σ 0.45) for easy items, 73.44 (σ 1.56) for medium items, and 98.59 (σ 1.34) for hard items.
• Rare degenerate items. Of the 35,550 questions and 30 models, only 2% of questions were universally solved or failed, and no models showed 100% or 0% accuracy.

Methodology in Plain English

EIP treats evaluation as a two-way relationship between models and questions, drawing on the intuition behind PageRank.

1. Build a graph. Models and questions become nodes in a directed bipartite graph. A question points to every model that solved it correctly (a "competency edge"), and a model points to every question it failed (a "difficulty edge"). These edges are stored as two binary matrices that are transposes of each other's complements.
2. Filter trivial items. Questions solved by all models or failed by all models are excluded, guaranteeing graph connectivity and avoiding trivial solutions. Such cases are assigned conceptual lowest or highest difficulty.
3. Propagate scores in both directions. A model gains competency weight from the questions it solves, divided by how many models solved each question, so solving a rare hard question counts more. A question gains difficulty weight from the models that fail it, divided by how many questions each model failed, so being failed by a strong model counts more.
4. Add damping. A pure random walk on a bipartite graph is 2-periodic, so a damping factor α acts as a "teleportation" probability and makes the chain ergodic, guaranteeing convergence to unique stationary scores. The paper states the process is guaranteed to converge by the Perron-Frobenius theorem, with a formal proof in Appendix C.
5. Iterate efficiently. The procedure is non-parametric, requires no per-item logistic curve fitting (unlike IRT), and runs in linear time in the number of model-question interactions, with a single damping hyperparameter.
6. Support partial credit. For benchmarks with free-form answers and partial credit, the response matrix is allowed to take values in the unit interval ([0, 1]), and all quantities and updates carry over unchanged.

Why This Matters

Impact on research. Accuracy-only leaderboards can hide which model is actually better at hard reasoning, and rank order can shift with the easy/hard mix of items. EIP provides a difficulty-aware, human-aligned alternative that supplements accuracy rather than replacing it (the reported Kendall's τ of 0.8492 with accuracy supports this framing). It also offers a far cheaper alternative to IRT, which the paper measures in the thousands of seconds for 1PL and 2PL fits versus 0.00597 seconds for EIP.

Real-world applications.

• Building and maintaining LLM leaderboards that remain stable as models are added or retired, which the paper explicitly supports through its reported robustness to removing up to 15 of 30 models.
• Benchmark design and curation, using difficulty profiles to choose test sets that cover a desired difficulty range (uniform like GPQA, or advanced-reasoning-oriented like MATH).
• Selecting models for reasoning-heavy deployments, where the ordering that matters is about hard-item performance rather than aggregate accuracy.
• Low-cost continuous evaluation pipelines, given the reported linear scaling to Q of 1,000,000 and M of 2,000 and the cited AWS EC2 cost of roughly $0.05 per vCPU-hour.

Industry relevance. The paper describes a Hugging Face leaderboard platform supporting continuous updates of new models and benchmarks, capable of handling hundreds of daily updates while maintaining cost efficiency. For organizations that must rank many models across large question sets on routine hardware, the reported constant iteration count and linear per-iteration cost make the approach practical.

Future Directions

• Ground truth for difficulty at scale remains unavailable. The paper states that identifying model capability on difficult questions is hard because large-scale evaluations lack ground truth, which is why it relies on a controlled simulation. Extending validation beyond the 20-evaluator, 70-pair human trial is an open direction.
• Wider model and benchmark coverage. The reported evaluation covers 30 models and 35,550 questions across six benchmarks (BBH, GPQA, GSM8k, HellaSwag, MATH, MMLU-Pro); the paper does not report extensions to agentic, multimodal, or safety-oriented benchmarks, though its related-work section discusses such benchmarks.
• Pool composition policy. The paper shows mixed-scale pools reduce extreme misestimates and improve human alignment, but does not specify an optimal pool size or composition; identifying such a policy is a natural next step.
• Governance of difficulty-aware rankings. The ethics statement notes that difficulty-aware rankings could be misused to dismiss models that prioritize safety or fairness over raw competence, and calls for adoption that complements broader alignment assessments.

Target Audience

This paper is most useful to LLM evaluation researchers and benchmark maintainers, leaderboard operators and platform engineers who need efficient large-scale ranking, and model-selection practitioners in industry who must compare closely matched models on reasoning-heavy workloads. Readers with a background in statistics or probabilistic modeling will find the IRT comparisons and convergence discussion accessible; readers primarily interested in model architecture or training will find the contribution is entirely on the evaluation side.

Authors’ abstract

Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field. However, existing benchmarks fail to differentiate question difficulty, limiting their ability to effectively distinguish models' capabilities. To address this limitation, we propose RankLLM, a novel framework designed to quantify both question difficulty and model competency. RankLLM introduces difficulty as the primary criterion for differentiation, enabling a more fine-grained evaluation of LLM capabilities. RankLLM's core mechanism facilitates bidirectional score propagation between models and questions. The core intuition of RankLLM is that a model earns a competency score when it correctly answers a question, while a question's difficulty score increases when it challenges a model. Using this framework, we evaluate 30 models on 35,550 questions across multiple domains. RankLLM achieves 90% agreement with human judgments and consistently outperforms strong baselines such as IRT. It also exhibits strong stability, fast convergence, and high computational efficiency, making it a practical solution for large-scale, difficulty-aware LLM evaluation.