ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models

AI summary

Overview

Research area: Evaluation of factual knowledge retrieval in large language models (benchmark design, LLM-as-judge metrics, factual hallucination measurement).

Technical level: Intermediate. The conceptual setup is easy to follow, but the evaluation framework involves bipartite matching, linear sum assignment, and multi-metric scoring that assume some familiarity with LLM benchmarking.

Scope (one sentence): The paper introduces ListQA, a 9,045-question benchmark that tests whether LLMs can retrieve multiple facts and compose them into correctly structured lists of 3–10 elements, and reports results for 34 models across 42 configurations.

What This Paper Is About

Existing factual knowledge benchmarks almost always ask for a single fact ("Who directed this film?"), so they cannot measure whether a model can recall the complete set of entities that satisfy a set of constraints, avoid adding fabricated extras, and organize the result into the requested structure. The authors build ListQA to fill that gap: a human-curated dataset of questions whose answers are lists, ranging from flat entity lists to hierarchical lists with sub-attributes such as recipient and result. They also design an evaluation pipeline that separates factual retrieval from format compliance, and use it to benchmark a wide range of models.

Key Contributions

1. A new benchmark, ListQA: 9,045 human-curated, cross-validated question–answer pairs requiring retrieval of multiple facts composed into structured lists of 3–10 elements. It has two subsets — SimpleListQA (single-faceted flat lists, 59.5% of samples) and ComplexListQA (multi-faceted hierarchical lists, 40.5%) — grounded in 3,765 unique Wikipedia tables across eight topical categories and 60+ countries, with an evergreen design (explicit temporal constraints or historically immutable facts) so ground truth stays valid without periodic updates. Reported audited error rate is 1.7%.

2. An LLM-Judge-F1 metric: element matching is framed as a linear sum assignment problem — a bipartite graph between reference and predicted elements weighted by pairwise ROUGE overlap, solved with the Jonker-Volgenant algorithm as implemented in SciPy — followed by LLM-based semantic grading of each matched pair (using GPT-OSS-120B in all reported experiments). This reduces LLM judge calls from O(n²) to O(n). Four complementary metrics are also reported: Relaxed-Recall, Relaxed-Precision, Format-Correctness, and ROUGE-F1, plus an ordering-specific LLM-Judge-F1-Ordered.

3. A large-scale model evaluation: 34 models from 8 families across 42 configurations, spanning closed-source and open-source models and parameter counts from 3B to 675B, evaluated under standard and thinking-enabled inference, with base vs. instruction-tuned checkpoints and token efficiency covered in the appendix.

4. An oracle RAG experiment: supplying the source Wikipedia page used during annotation as context, keeping list-formatting instructions and metrics unchanged, to separate missing knowledge from failure to compose known facts into structure.

Main Findings

• Parametric knowledge is the bottleneck: the best standard-inference model scores below 37% LLM-Judge-F1. The top tier (GPT-5.4, Claude Opus variants, Gemini 2.5 Flash, Nova Premier) clusters in a narrow band suggesting a shared ceiling, and the majority of models fall below 20%. GPT-5.4 is the top closed-source model at 36.5 overall (44.5 SimpleListQA, 24.9 ComplexListQA); the lowest reported overall scores are 3.3 (Ministral 3 3B) and 4.8 (Ministral 3 8B).

• Multi-faceted questions are harder for every model: every model scores higher on SimpleListQA than ComplexListQA, with the gap widening for stronger models and the weakest collapsing to near-zero on Complex. Gemini 3.1 Pro achieves the strongest overall result and is the only model to cross 50% LLM-Judge-F1 on ComplexListQA, and only barely.

• Thinking helps unevenly: thinking-enabled inference yields larger gains on SimpleListQA than ComplexListQA, indicating it helps models organize knowledge they already possess but gives diminishing returns when parametric recall is the limit. Thinking universally improves factual accuracy, but its effect on Format-Correctness is architecture- and family-dependent — closed-source models slightly lose format compliance while open-weight models gain it dramatically.

• Oracle RAG produces large gains for every model: providing the source page raises LLM-Judge-F1 by +34.3 to +65.0 points (for example, Gemma 4 31B improves from 16.0 to 80.5, +64.5; Claude Sonnet 4.6 improves from 28.0 to 83.0, +55.0), far exceeding gains from thinking, and confirming that inference-time reasoning cannot substitute for missing knowledge. Closed-book rankings reshuffle once context is supplied — Gemma 4 31B trails GPT-5.4 mini and Gemini 2.5 Flash Lite closed-book but overtakes both with context.

• Knowledge and instruction-following are partially decoupled: several models achieve high Format-Correctness while scoring low on LLM-Judge-F1, following structural instructions without the knowledge to populate them, while others with moderate factual accuracy format poorly. Format scores show that many models produce the wrong list structure, defaulting to sub-bullet formatting and thus failing the flat-list requirement in SimpleListQA while inflating ComplexListQA format scores.

• Timeless questions are easier than time-constrained ones: every model performs better on timeless than on time-constrained questions. The paper cites prior work (Jiang et al.) showing that 24–64% of time-sensitive samples in popular factual datasets carry outdated gold answers.

• Other patterns: category-level difficulty is model-invariant and attributed to training corpora distribution; there is a consistent geographic hierarchy (strongest on South American/North American content, weakest on African/Asian content); list length itself has only a modest effect; and generation verbosity does not correlate with factual accuracy.

• Metric validation: LLM-Judge-F1 was validated against 3,313 human-graded list elements from 571 sample–model pairs (roughly 200 stratified questions, 3 models), correlating strongly with human F1 (Pearson r = 0.965; Spearman ρ = 0.956) and being slightly conservative relative to human judgment. ROUGE-based matching aligns elements correctly 99.7% of the time, and replacing it with embedding-based matching changes aggregate scores by less than 0.002. Rescoring all 9,045 samples from one evaluated model with Qwen 3.5 27B, 9B, and 4B judges gave pairwise Spearman correlations above ρ = 0.96 and mean absolute differences ≤ 0.024; the 4B judge gave ρ = 0.972 and MAD = 0.022.

• Dataset composition: 9,045 total samples; elements per answer listed as 3–8 in the statistics table with a mean of 4.67 and median of 4 (the task definition states 3 ≤ n ≤ 10); 70.1% year-tagged, 29.9% timeless; 14.6% ordered, 85.4% unordered; mean sub-elements for multi-faceted items is 8 (range 1–10); years covered are 2000–2024 with deliberate emphasis on 2017–2024; answers validated as correct as of December 31, 2024.

• Data quality process: 44 annotators authored questions, each independently validated by a second annotator with no involvement in its creation (100% coverage), with significant-rework items discarded (10% discard rate) and all corrections verified by at least one additional reviewer (two reviewers covering 20% of corrections); every sample was assessed by at least three independent people. Of submitted annotations, 39% were approved with no flags, 51% required minor corrections, and 10% were discarded. Among flagged errors, factual accuracy errors were only 10% of flags. An audit of 481 questions stratified across all 8 categories and both question types found 8 errors (1.7%; 95% Wilson CI [0.8%, 3.2%]).

Methodology in Plain English

The authors started from Wikipedia tables because tables naturally contain many rows that can satisfy a query condition and because Wikipedia is widely represented in model training data. Human annotators picked tables across eight categories (Entertainment including Music, Sports, Politics, Movies, Lifestyle, Food, Travel, and Places of Interest), wrote closed-ended objective questions whose answers require reading one or more columns, and composed reference answers as lists of 3 to 10 elements. Each answer was independently cross-validated by a second annotator, and questions were framed with explicit time constraints (for example, "opened from 2000 to 2024") so that the answers would not go stale.

For evaluation, they parse list elements out of both the reference and the model's answer with a regex. One metric (Relaxed-Recall/Relaxed-Precision) just string-matches reference elements anywhere in the raw model output, ignoring format. Another (Format-Correctness) only checks whether the model produced the right kind of list structure, requiring at least 3 elements. The headline metric, LLM-Judge-F1, first aligns predicted elements to reference elements by building a cost matrix from pairwise ROUGE-1 scores and solving for the best one-to-one assignment, which cuts expensive LLM judge calls from quadratic to linear. Each matched pair is then graded CORRECT or INCORRECT by an LLM judge, and unmatched predicted elements are automatically graded INCORRECT; precision, recall, and F1 are computed over judged-correct elements. A separate ordered variant pairs elements positionally instead of by assignment.

Models are run with a system instruction specifying the expected output format, in zero-shot mode (with few-shot mode available using 5 in-context examples), on the full dataset or the simple/complex splits. The authors then compare results across model families, with and without thinking, with and without the source Wikipedia page supplied as context.

Why This Matters

Impact on research. ListQA reframes factual evaluation from single-answer lookup to multi-element retrieval plus structured output, a combination the paper argues no prior benchmark covers simultaneously (multi-answer datasets such as QAMPARI and RoMQA evaluate only set overlap, not structured output or format compliance). It also makes temporal validity an explicit design constraint at the list level, where a single outdated entry can corrupt the ground truth. The metric design — optimal element alignment paired with semantic grading, validated against human judgment — offers a reusable template for evaluating any set-based or list-based output.

Real-world applications:

• Clinical decision-making: listing which drugs were approved for a condition in a given window along with their mechanisms of action, where each element carries independent downstream consequences.
• Legal research: identifying which countries ratified a treaty before a given year.
• Travel and logistics: enumerating which direct routes an airline launched since a particular year.
• Structured report generation and summarization: producing organized, auditable lists of facts where individual hallucinations can be detected and removed entry by entry.

Industry relevance. The results show a large practical gap: no current model, including the strongest evaluated, masters both factual retrieval and format compliance. Because the paper demonstrates that supplying retrieved source pages lifts scores by 34–65 points, it gives direct quantitative support for retrieval-augmented and tool-use pipelines over relying on parameters alone, and it provides a way to measure whether the residual errors after retrieval come from composition rather than knowledge.

Future Directions

• Multilingual extension: the benchmark is English-only, and the authors identify multilingual coverage as a natural next step.
• Explicit abstention measurement: the current metrics penalize hallucinated elements as an indirect overconfidence signal, but the benchmark does not reward models for recognizing the limits of their knowledge and declining to answer.
• Reducing evaluation cost and judge noise: the primary metric relies on a 120B reasoning model as judge, adding non-trivial inference cost per run; the paper offers agreement figures for smaller judges (down to 4B) as a lighter-weight alternative, and notes the inherent limitation of reference-based judging when a model's prediction contains information the judge cannot verify or refute.
• Region and language coverage: 78% of samples concern US entities, and all questions are posed in English, so observed regional gaps conflate regional knowledge with English-language coverage and cannot be attributed to either alone. The authors present this as a measured boundary rather than one they resolve.
• Ground-truth completeness: reference answers are built from a single Wikipedia table and were not independently audited for completeness against external sources, so a valid element absent from the source table would be penalized; each sample ships with its source URL so such cases can be audited and corrected in future releases.

Target Audience

Researchers and engineers working on LLM evaluation, factual knowledge benchmarking, hallucination measurement, and retrieval-augmented generation will benefit most, along with benchmark designers interested in LLM-as-judge methodology validated against human grading. The plain-language framing of the knowledge-versus-format gap and the oracle RAG findings also make the paper useful to product and applied teams deciding whether to invest in retrieval pipelines rather than larger models. Some familiarity with standard QA benchmarks and LLM evaluation metrics helps, but the core argument is accessible without deep technical background.

Authors’ abstract

Many real-world questions demand not a single fact but an organized collection of them, yet existing factual knowledge benchmarks almost exclusively target single answer retrieval. We introduce ListQA, a benchmark of 9,045 human-curated, cross-validated questions (audited error rate of 1.7%) that require LLMs to recall multiple facts and compose them into structured lists of 3-10 elements. Questions range from flat lists ("Which NCAA teams went undefeated between 2000 and 2024?") to hierarchical lists with sub-attributes ("...and what was their record and result?"), spanning eight categories across 60+ countries. An evergreen design (explicit temporal constraints or historically immutable facts) keeps ground truth valid without periodic updates. For evaluation, we frame element matching as a linear sum assignment problem: optimal bipartite matching paired with LLM-based semantic grading jointly captures factual accuracy, hallucination, and format compliance. We benchmark 34 models (8 families, 42 configurations across standard and thinking enabled inference) and find: the best standard-inference model scores below 37% LLM-Judge-F1, multi-faceted questions are harder across the board, and providing the source Wikipedia page lifts scores dramatically, confirming that models can organize facts but struggle to retrieve them from parameters alone. Extended analyses cover base vs. instruction-tuned checkpoints and token efficiency. We release the dataset and evaluation code.