The Pulse
GitHub Opens ReviewBench to Test AI Code-Review Agents
GitHub launched ReviewBench on October 5, 2026, as an open benchmark for AI code-review agents. Its 219 pull requests, labeled findings and shared scoring process let teams compare reviewers and test their own systems.
AI.info Team · Written with AI ·
“We built ReviewBench, a new code review offline benchmark, to address that gap, and it is available for you to use today.”
Michelle Zhou and Alejandro Carderera de Diego, GitHub Blog, 5 October 2026
GitHub targets the review-quality gap
GitHub announced ReviewBench on October 5, 2026, offering developers a shared way to assess AI systems that review pull requests. The open benchmark pairs real-world code changes with a reference set of findings, then measures whether a reviewer catches useful issues without adding false alarms. GitHub says it built the benchmark to give teams a more consistent offline signal before they run production experiments.
The release arrives with the dataset, evaluation methodology and tools for teams to test their own agents. Reviewers can run locally for private tuning or register through ReviewBench’s website to submit an official result. GitHub says the project also helps its own Copilot code review team assess changes before testing them with users.
219 pull requests, with larger changes represented
ReviewBench draws on an analysis of 103.9 million GitHub pull requests. Its full corpus contains 219 public pull requests from 187 repositories across 19 programming languages. GitHub says the language and repository-size mix follows its broader platform, while the benchmark deliberately weights pull-request size toward the reviewable middle and larger changes rather than letting tiny edits dominate.
Each pull request has a multi-source set of candidate findings, gathered from human reviews, author follow-up commits, deterministic tools and frontier language models. The benchmark deduplicates overlapping findings and applies a shared rubric, counting an issue as valid only when it is true, relevant and non-trivial. Findings also receive severity and category labels, letting teams examine performance on areas such as security, correctness, reliability and testing.
Two metric families address missed findings
ReviewBench reports grounded precision and recall against the published reference findings. It also uses augmented precision and recall: an evaluator judges a reviewer’s unmatched findings, so an agent can receive credit for identifying a real issue missing from the reference set. GitHub uses grounded recall as the main cross-system comparison because augmented recall can expand as an agent discovers additional issues.
GitHub reports that senior engineers who had not helped build the dataset independently relabeled the benchmark’s ground-truth findings, agreeing with its true-or-false-positive judgments 96.6% of the time. That agreement check addresses label quality, but it does not make the benchmark a complete measure of how useful a reviewer will be inside a particular development team.
Submissions share a judge, not a model
Teams can first run an agent on a 25-pull-request test set, then submit it for three official rounds across all 219 pull requests. Participants provide their own agent and model access; GitHub supplies the judge for the final evaluation. The repository publishes the agent contract, corpus, findings and scoring materials, and carries an MIT license.
GitHub says results stay private until a maintainer reviews and approves them. Its blog also describes internal experiments in which offline benchmark changes tracked the direction of later production tests, while emphasizing that online experiments remain the ultimate measure of user impact. ReviewBench therefore offers a common test and a transparent set of labels, not a promise that the highest-scoring agent will fit every team’s review preferences.