# GitHub Opens ReviewBench to Test AI Code-Review Agents

Published 2026-10-18 · AI.info Team · AI.info

> GitHub launched ReviewBench on October 5, 2026, as an open benchmark for AI code-review agents. Its 219 pull requests, labeled findings and shared scoring process let teams compare reviewers and test their own systems.

> “We built ReviewBench, a new code review offline benchmark, to address that gap, and it is available for you to use today.”
>
> Michelle Zhou and Alejandro Carderera de Diego, [GitHub Blog](https://github.blog/ai-and-ml/github-copilot/reviewbench-an-open-benchmark-for-ai-code-review/), 5 October 2026

## GitHub targets the review-quality gap

GitHub announced ReviewBench on October 5, 2026, offering developers a shared way to assess AI systems that review pull requests. The open benchmark pairs real-world code changes with a reference set of findings, then measures whether a reviewer catches useful issues without adding false alarms. The benchmark was built to give teams a more consistent offline signal before they run production experiments, GitHub says.

The release arrives with the dataset, evaluation methodology and tools for teams to test their own agents. Reviewers can run locally for private tuning or register through ReviewBench’s website to submit an official result. The project also helps GitHub’s own Copilot code review team assess changes before testing them with users.

## 219 pull requests, with larger changes represented

ReviewBench draws on an analysis of 103.9 million GitHub pull requests. Its full corpus contains 219 public pull requests from 187 repositories across 19 programming languages. The language and repository-size mix follows GitHub’s broader platform, the company says, while the benchmark deliberately weights pull-request size toward the reviewable middle and larger changes rather than letting tiny edits dominate.

Each pull request has a multi-source set of candidate findings, gathered from human reviews, author follow-up commits, deterministic tools and frontier language models. The benchmark deduplicates overlapping findings and applies a shared rubric, counting an issue as valid only when it is true, relevant and non-trivial. Findings also receive severity and category labels, letting teams examine performance on areas such as security, correctness, reliability and testing.

## Two metric families address missed findings

ReviewBench reports grounded precision and recall against the published reference findings. It also uses augmented precision and recall: an evaluator judges a reviewer’s unmatched findings, so an agent can receive credit for identifying a real issue missing from the reference set. GitHub uses grounded recall as the main cross-system comparison because augmented recall can expand as an agent discovers additional issues.

Senior engineers who had not helped build the dataset independently relabeled the benchmark’s ground-truth findings, agreeing with its true-or-false-positive judgments 96.6% of the time, GitHub reports. That agreement check addresses label quality, but it does not make the benchmark a complete measure of how useful a reviewer will be inside a particular development team.

## Submissions share a judge, not a model

Teams can first run an agent on a 25-pull-request test set, then submit it for three official rounds across all 219 pull requests. Participants provide their own agent and model access; GitHub supplies the judge for the final evaluation. The repository publishes the agent contract, corpus, findings and scoring materials, and carries an MIT license.

Results stay private until a maintainer reviews and approves them. GitHub’s blog also describes internal experiments in which offline benchmark changes tracked the direction of later production tests, while emphasizing that online experiments remain the ultimate measure of user impact. ReviewBench therefore offers a common test and a transparent set of labels, not a promise that the highest-scoring agent will fit every team’s review preferences.

## Sources

1. Original source [The GitHub Blog](https://github.blog/ai-and-ml/github-copilot/reviewbench-an-open-benchmark-for-ai-code-review/) github.blog

Other pages read

2. [GitHub](https://github.com/review-bench/ReviewBench) github.com

## Companies in this story

- [GitHub](https://ai.info/companies/github)

## Related stories

- [A Mixed AI Review Caught More Planted Errors in One Test](https://ai.info/news/mixed-ai-review-planted-errors-cross-model-study)
- [Microsoft and Hugging Face Test Agents Across 507 Workflows](https://ai.info/news/microsoft-hugging-face-thinkingbox-agent-reliability-benchmark)
- [GitHub Octoverse: Agents Started More Than 40% of September Pull Requests](https://ai.info/news/github-agents-started-40-percent-pull-requests)
- [GitHub Says New Git Storage Architecture Supported 5,000 Concurrent Agents in Internal Testing](https://ai.info/news/github-rebuilds-git-storage-5000-concurrent-agents)
- [Vercel opens paid Security Review to unlinked GitHub repositories](https://ai.info/news/vercel-security-review-unlinked-github-repositories)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/github-reviewbench-code-review-agents
