Uncovering Competency Gaps in Large Language Models and Their Benchmarks

AI summary

Overview

Research area: Natural Language Processing / LLM evaluation and mechanistic interpretability. Technical level: Intermediate (some familiarity with LLMs and interpretability helps, but the core ideas are accessible). Scope: This paper introduces "Competency Gaps" (CG), an automated method that uses sparse autoencoder (SAE) concept activations to pinpoint two things standard benchmarks hide — where models are weak, and where benchmarks themselves fail to test what they claim to test.

What This Paper Is About

Standard LLM evaluation collapses performance into a single number, like accuracy or pass@k. That aggregation hides two problems: specific sub-areas where a model is weak (model gaps), and concepts that benchmarks ought to test but barely cover (benchmark gaps). The authors build a method that opens up the black box of benchmark scores by decomposing them into thousands of interpretable, model-internal concepts.

Key Contributions

1. The Competency Gaps (CG) method. A simple, automated technique that uses SAE concept activations from a model's internal representations to produce per-concept coverage scores for benchmarks and per-concept performance scores for models. It requires only one extra linear projection per token, making it cheap relative to running the benchmarks themselves.

2. Two gap-detection metrics. A normalized per-concept coverage metric (χ_bench) that is scale-invariant across benchmarks, and a per-concept performance metric (χ_model) that weights benchmark scores by how strongly a concept activates on each data point. Both aggregate into cross-benchmark scores.

3. Demonstrations across five models and over a dozen benchmarks. Applied to Llama3.1-8B-Instruct, Gemma2-2B-Instruct, Mistral-7B-Instruct-v0.1, Qwen3-4B, and DeepSeek-R1-Distill-Llama-8B across factuality, math, reasoning, ethics, arena-style, and general-ability benchmarks. The method recovers known weaknesses (e.g., sycophancy) and surfaces new ones.

4. An open-source interactive web tool. A searchable, filterable interface that lets users browse concepts, inspect per-benchmark details, view example generations, and explore benchmark coverage and cross-benchmark correlations.

Main Findings

• Benchmark coverage is heavily skewed. Across the ten static benchmarks, most concepts have low coverage while a small number dominate. Mean-based summary statistics are therefore controlled by a handful of "top concepts" — mainly around starting new conversations and sports news (especially English Premier League football).

• Benchmarks systematically under-test AI meta-cognition. The lowest-coverage concepts concern the assistant's own boundaries, roleplay limits, discussion of user inputs, and explanations of its limitations. 314 concepts (about 1%) were entirely missing from the suite, including legal and regulatory concepts.

• Individual benchmarks miss relevant concepts. Every benchmark except Vectara misses at least 30% of concepts overall, but more importantly, some miss concepts that seem squarely in scope. AGI Eval misses "the need for thorough and objective assessment of evidence"; Social IQA misses "instructions about how someone should behave or what qualities to embody."

• Model strengths cluster around coding and helpfulness. Llama 3.1 8B scored highest on concepts like programming sequence iteration, providing illustrative examples, and expressing commitment to help.

• Model weaknesses mirror benchmark weaknesses. The worst-performing concepts include polite rejection of inappropriate requests and maintaining professional boundaries — the semantic opposites of the sycophantic behaviors that score highest. This suggests benchmark gaps can causally produce model gaps.

• Additional weak areas validated and discovered. The method confirmed anecdotally known weaknesses (time representations, image manipulation metadata, palindrome checks, mathematical operations) and surfaced a previously unstudied category: "appeals to intuition in reasoning or decision making."

• The method is robust. Using a different model's SAE (Gemma 2 2B instead of Llama's own) produced qualitatively similar themes. Re-running with 20% of data randomly dropped 100 times gave standard deviations of 0.014 (model scores) and 0.025 (coverage scores). Removing under 1% of data points aligned with top or bottom concepts shifted median performance predictably (down 0.6% and up 1.3%, respectively).

• Existing popular benchmarks show structural bias. MMLU privileges factual recall over calibrated refusal; SWE-Bench encodes a narrow Python-patch slice of software engineering, and CG flags compilation and computer security as among its most underrepresented concepts.

Methodology in Plain English

The method leans on sparse autoencoders, a tool from mechanistic interpretability. An SAE is trained to reconstruct a model's internal activations at a chosen layer using a large over-complete dictionary of features. Each feature is supposed to correspond to a single interpretable "concept," and an automated pipeline assigns each one a text label, like "iteration or traversal through sequences in programming."

The workflow is straightforward:

1. Encode each benchmark item. Run the benchmark's inputs through the model and record how strongly each SAE concept activates on each token. Average over tokens per data point so longer items do not dominate.

2. Measure coverage. For each concept and benchmark, divide that concept's total activation by the average activation of all concepts in that benchmark. This normalization makes values comparable across benchmarks — a score of 1 means the concept is covered at the benchmark's own average. Concepts below a very small threshold are labeled "missing"; the bottom and top deciles are "underrepresented" and "overrepresented."

3. Measure performance. For each concept, compute a weighted average of the benchmark's scores, where the weights are the concept's activation strength on each data point. A concept with a very low score is a model gap.

4. Aggregate. Average across benchmarks to produce cross-benchmark scores. Because concepts are fixed per model, results can be composed across benchmarks and compared cleanly.

The authors also ran an LLM-as-judge step to label each concept as semantically relevant or irrelevant to each benchmark, and used an LLM to cluster and summarize large concept lists. They released the relevance labels and an interactive web app so users can sift through results themselves.

Why This Matters

Impact on research. Aggregated benchmark scores are the primary currency of LLM progress, but this paper shows they can mask substantial per-concept variance and structural bias. CG offers a concept-level decomposition that complements existing benchmarks rather than replacing them, and it provides a feedback loop for benchmark developers to find and fix coverage holes.

Real-world applications:

• Benchmark design and iteration. Developers can identify underrepresented or missing concepts relevant to their benchmark's scope and generate targeted new data — including via SAE-based steering during generation.
• Model evaluation and selection. Teams can use concept-level profiles to pick models that are strong in the specific competencies their application requires, rather than relying on a generic leaderboard rank.
• Safety and alignment testing. The finding that polite refusal and boundary-setting are consistently weak is directly actionable for anyone deploying chat systems that need to say no gracefully.
• Benchmark discovery. The method can be integrated into benchmark databases to help users assemble a suite with desired concept coverage.

Industry relevance. Frontier labs and evaluation organizations are already moving toward disaggregated evaluation. CG shows how to do this automatically and cheaply using existing interpretability infrastructure. The entanglement of benchmark and model gaps is also a governance-relevant insight: if benchmarks under-test refusals and meta-cognition, model developers have little incentive to fix weaknesses there, and the resulting models look strong on paper while remaining fragile in deployment.

Future Directions

• Calibrating for item difficulty. A low concept score might just mean the concept activates on the hardest questions. Disentangling genuine model gaps from a "hard-question confound" is called out as an important open problem.

• Incorporating sensitivity analysis and finer granularity. The authors suggest automated sensitivity analysis over the choice of SAE layer, and per-token rather than per-item activations for more precise attribution.

• Building targeted benchmark data. Using the list of underrepresented concepts, new benchmark items could be generated by prompting LLMs with autointerpretability labels or by directly steering with SAE directions during generation.

• Extending beyond text and SAE coverage. CG can only detect gaps for concepts that exist in the SAE space; concepts absent from a model are invisible by construction. More representative SAEs — and extensions to multimodal settings — would broaden what the method can see.

Target Audience

This paper is most valuable for benchmark developers and maintainers who want to audit and improve coverage of their datasets; LLM evaluators and safety researchers who need to move beyond aggregate scores; interpretability researchers interested in applying SAEs to concrete evaluation problems; and model developers at labs who want a concept-level diagnostic of strengths and weaknesses across model families. Readers without prior exposure to sparse autoencoders will benefit from the SAE primer in Appendix A, but should expect to encounter terms like "concept dictionary," "activation scores," and "autointerpretability labels."

Authors’ abstract

The evaluation of large language models relies heavily on standardized benchmarks. These benchmarks provide useful aggregated metrics, but can obscure (i) particular sub-areas where the models are weak ("model gaps") and (ii) imbalanced coverage in the benchmarks themselves ("benchmark gaps"). To automatically uncover both types of gaps, we propose a simple new method using concept activations from sparse autoencoders, to identify fine-grained gaps on a per-concept basis. The method also benefits from grounding evaluation in the model's internal representations, as well as easy comparison across benchmarks. We applied the method to five popular open-source models and more than a dozen benchmarks, as illustrative examples. As validation of the approach, we found that our automatic, unsupervised method was able to recover model gaps that have been previously documented in the literature (e.g. relating to sycophancy), in addition to identifying novel model gaps. We were also able to automatically uncover benchmark gaps: core concepts that should fall within the scope of a given benchmark. Our "competency gaps" method can be used to complement existing benchmarks, by providing a concept-level decomposition of model behavior, and by helping benchmark developers iterate upon benchmark design. Code is available at https://competency-gaps.github.io.