Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

AI summary

Overview

Research area: AI evaluation infrastructure — benchmark discovery, retrieval, and metadata auditing for large language models and other AI systems.

Technical level: Intermediate. The system-design and retrieval portions are accessible to newcomers, while the measurement-eligibility audit and census tables require some familiarity with benchmark reporting practice.

Scope: The paper introduces Benchmark Radar, a living database, search engine, and evidence-auditing tool that aggregates AI benchmark records, daily discovery observations, model-report mentions, and score histories into one searchable catalog, and reports a full-catalog census of that catalog.

What This Paper Is About

Researchers who want to evaluate an AI system must find relevant benchmarks, locate their datasets and code, and understand the exact settings behind reported scores — but this evidence is scattered across paper servers, code hosts, dataset hubs, vendor releases, and separate leaderboards. Benchmark Radar addresses that fragmentation by combining a searchable benchmark catalog with daily discovery of new benchmark papers, repositories, datasets, and releases, while keeping source identities and citations attached so readers can inspect the evidence behind any benchmark or score. The paper describes how the system collects and retrieves this material, then audits the complete catalog to characterize coverage, documentation, and the limits of score comparison.

Key Contributions

1. A living search system for benchmarks. Benchmark Radar combines daily discovery, a searchable benchmark catalog, mentions in model cards and technical reports, and score histories, with shared benchmark IDs connecting a web dashboard, downloadable catalog, and offline query clients.

2. A full-catalog census. The authors audit every record in the v0.11.0 catalog — 1,283 source records drawn from 4 catalogs, with 12,916 numeric score observations on 790 records — counting benchmark records, scored models, cited documents, and numeric observations as separate units.

3. A measurement-eligibility framework. The paper defines what evidence is required before scores can be compared across records, separating score values from the units, directions, test versions, prompts, tools, attempts, and evaluators that would make comparisons valid.

4. Analysis of benchmark saturation, adoption trends, and documentation gaps. The paper examines capability taxonomy across the catalog, release-year composition, discovery source concentration, and how much documentation metadata remains unresolved.

Main Findings

• Catalog composition and score coverage: The v0.11.0 catalog contains 1,283 source records from 4 sources — LLM Stats (687), OpenCompass Hub (461), Artificial Analysis (25), and model reports (110). Of these, 790 records have numeric scores and 493 do not. Numeric score observations total 12,916: 5,544 from LLM Stats, 7,050 from Artificial Analysis, 322 from model reports, and 0 from OpenCompass Hub.

• Unscored records still carry artifact links: Of the 493 unscored records, 464 link to at least one paper, repository, or dataset. Across the full catalog, 475 records link to papers, 506 to repositories, and 293 to datasets.

• Benchmark counts are not counts of distinct tests: The paper states that source records are not a count of distinct underlying tests, and that two sources can describe a related benchmark while both source records are retained.

• Documentation is widespread but organization metadata is thin: The shared document registry contains 1,208 distinct cited documents attached to 1,278 of 1,283 records; 5 records have no cited documents. Of the 1,208 documents, 1,171 lack a named organization.

• One citation can document hundreds of models: The Artificial Analysis record for GPQA Diamond contains scores for 586 models but cites 1 registry page; Humanity's Last Exam (577 models, 1 document) and SciCode (577 models, 1 document) follow the same pattern, while the model-reports GPQA Diamond record cites 27 documents for 21 numeric observations on 19 models. This is why model, document, and score counts cannot substitute for one another.

• Most scored records cannot support a shared percentage comparison: Only 82 of the 790 scored records declare a percentage unit with a known direction and values within 0–100. The remaining 708 scored records use other or unverified scales, and 493 records have no numeric score at all.

• Date coverage is partial and date bases differ: Benchmark release dates are recorded for 615 records, leaving 668 without one. Across score observations, 12,594 entries carry model-announcement dates and 322 carry document-publication dates — neither of which directly establishes an evaluation date.

• Capability labels come from publishers: 1,279 of 1,283 records are classified into 11 top-level domains and 63 sub-domains using the publishers' own fields. The remaining 4 are LLM Stats community rows whose crawl supplied no description, category, or modality.

• Interaction paradigm is a facet, not a class: Of 345 agentic records, 128 take Agentic & Tool Use as their Level 1 class while 117 sit under Coding & Software Engineering, with the rest spread across 6 further classes. A single-axis scheme would hide those 117 records from any count of agentic evaluation.

• Agentic share growth is robust to catalog composition: Over the years meeting the paper's reporting criteria, reweighting each year to a common source composition moves the agentic share by at most 1.1 percentage points.

• Discovery volume and source concentration: Daily discovery contains 11,068 observations and 6,546 artifacts linked by exact identifiers across 46 snapshots, of which 4 are simulated historical backfills. Five discovery source labels account for 9,743 of 11,068 observations (88.0%), and 59 of the 6,546 artifacts have observations from multiple sources.

• Scored-model coverage varies sharply by source: 22 benchmark records contain numeric scores for at least 100 distinct source model IDs. Median scored models per record are 3 for LLM Stats (maximum 239), 245 for Artificial Analysis (maximum 586), and 2 for model reports (maximum 19); OpenCompass Hub is unknown because it has no scored records.

• A worked prior-art search: A contributor surveyed August 2026 work on credit assignment in agentic training using small Qwen-series models, using a coding agent with the CLI and Skill to assemble a five-row comparison table covering SRPO, ContextPilot, SkillGate, CIPO, and MoRSE.

• A snapshot of one collection run: The 2026-09-07 snapshot records 1,003 fetched rows and 954 candidates after duplicate removal, of which 366 qualified for publication and 138 met the recommendation threshold.

Methodology in Plain English

The authors build a pipeline that watches public sources and normalizes what it finds into two separate layers. The first layer is the benchmark catalog: one record per benchmark entry per contributing source, pulled from LLM Stats, OpenCompass Hub, Artificial Analysis, and model reports. Records are kept even when a score, date, or citation is missing, and reviewed identity links connect related records without merging their separate measurements.

The second layer is daily discovery. Each collection run searches a 48-hour window across 37 sources — 13 direct connectors and 24 first-party research and engineering feeds — and records mentions, releases, and updates as observations, with artifacts defined by exact identifiers such as DOIs, arXiv IDs, and repository URLs. These discovery units deliberately do not add to the benchmark catalog total. Collection records counts and errors by source, removes future-dated rows, and requires core sources (arXiv, Hugging Face Hub, and GitHub Search at the cutoff) to be healthy before publication.

For retrieval, the system uses BM25F, a field-weighted word-matching score, with bounded boosts for name and phrase matches. Source membership does not affect ranking, and each result exposes which query words matched, which fields they occurred in, and the score components — so a reader can inspect why a candidate surfaced rather than trusting a single number. The same benchmark IDs and response format serve the web dashboard, dataset exports, and an offline CLI that downloads archives verified by SHA-256 checksum, records its schema version and provenance, and answers queries without network access.

The audit then starts from every record in the rebuilt index with no date, score, or interface filter. It counts numeric observations once by observation ID, counts scored models by source model ID within each record, and counts documents by document ID — never summing model counts into a global total. Eligibility for a percentage-scale summary requires a declared percentage unit, a known score direction, and numeric values within 0–100; the authors explicitly state that rescaling a displayed value or reading an aggregator's declared maximum does not establish that unit.

Why This Matters

Impact on research. Benchmark claims are only as trustworthy as the evidence behind them, and that evidence is currently distributed across disconnected resources. Benchmark Radar makes the provenance searchable: a reader can find a candidate benchmark, follow its paper, repository, or dataset link, see which models were scored, and inspect the reported settings. The paper also quantifies how thin that evidence often is — only 82 of 790 scored records meet the percentage-scale rule, and 1,171 of 1,208 cited documents lack a named organization — which gives a concrete baseline for how much of the evaluation literature can currently support direct score comparison.

Real-world applications:

• Designing new evaluations: A researcher or agent assembles prior art before committing to a benchmark design, as in the paper's worked example on credit assignment in agentic training.
• Model selection and procurement: Teams comparing models can locate the specific benchmark records, scored-model counts, and cited sources behind a vendor's reported numbers.
• Reproducibility and audit workflows: Offline CLI queries against a checksum-verified dataset version let an analyst repeat a search against a recorded snapshot rather than a moving website.
• Agent-assisted literature review: The shipped agent skill tells an agent when a request calls for a benchmark query, which queries to run, and how to initialize data, letting a reader and an agent work from the same records and score components.

Industry relevance. Catalog search and the leaderboard's Pareto frontier view of score against measured use directly serve decisions about which evaluations are worth running and which reported results are comparable. The paper's insistence that matching scales do not establish matching test versions, prompts, tools, attempts, or evaluators is a practical caution for anyone building model cards, technical reports, or procurement criteria.

Future Directions

• Evaluate retrieval quality. The authors state that retrieval precision, task suitability, and time saved remain to be evaluated, and that the worked example has no controlled baseline. Accomplishing this will require reviewed relevance judgments across queries and candidate records, including less familiar benchmark families.

• Address lexical matching limits. Lexical search can miss paraphrases and renamed tasks, and the authors identify evaluating semantic retrieval as a necessary next step.

• Improve discovery backfill. The arXiv discovery route collects new submissions but does not backfill papers first posted before the collection window when later versions appear; the authors also note that their own Section 2 searches missed two relevant earlier studies that a coauthor identified and read.

• Validate task-capability classification. The deterministic null extractor used in CI assigns no capability levels to its 5,863 discovery-derived tracks, and completing and evaluating those labels is described as a separate task from maintaining the benchmark catalog.

• Establish comparable measurement conditions. Measuring benchmark saturation or score stagnation requires comparable test versions and settings with dates tied to score reporting or evaluation, which the current catalog does not uniformly provide.

Target Audience

This paper is most useful to evaluation researchers and benchmark designers deciding what to build or reuse; to LLM and agent developers who need to locate benchmarks and inspect the settings behind reported scores; to analysts and engineers building model-comparison or procurement workflows that depend on comparable measurements; and to tooling developers interested in how a discovery-and-catalog system can be structured so that provenance survives aggregation. Readers looking for new modeling techniques will not find them here — the contribution is infrastructure and measurement auditing.

Authors’ abstract

Benchmark researchers and developers of large language models (LLMs) and other AI systems need to find relevant evaluations, locate their benchmark datasets and code, and understand the settings behind reported scores. We present Benchmark Radar, a living database and search engine for retrieval and discovery of AI benchmarks, covering LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations. The system combines daily discovery of benchmark papers, repositories, datasets, and releases with a searchable benchmark catalog, mentions in model cards and technical reports, and score histories. It retains source identities and citations so readers can inspect candidate benchmarks and their evaluation evidence. Daily discovery draws on 37 sources: 13 direct connectors and 24 first-party research and engineering feeds. The catalog contains 1,283 source records drawn from 4 benchmark catalogs and 12,916 numeric observations on 790 records. We describe collection and retrieval, audit the full catalog, and examine benchmark saturation, adoption trends, and the limits of score comparisons. A worked example walks through a complete prior-art search, showing how to query the catalog and inspect benchmark evidence when designing a new evaluation. We release the web dashboard with a benchmark leaderboard, a Pareto frontier view of score against measured use, saturation and trend views, daily feeds, downloadable evidence, a command-line interface (CLI) for offline queries, and reproducible analysis.