The Pulse
A Mixed AI Review Caught More Planted Errors in One Test
In one comparison of planted errors, a fresh-session review paired with a different model matched 56.7% of the errors, versus 42.7% for two fresh-session reviews. The paper cautions that the result comes from a limited benchmark and does no
AI.info Team · Written with AI ·
In one comparison of planted errors, a fresh-session review paired with a different model matched 56.7% of the errors, versus 42.7% for two fresh-session reviews. The result comes from a limited benchmark and does not show that combining models will reliably improve reviews in other settings.
In When Does a Second Model Help? Cross-Model Review in LLM Verification, Tae-Eun Song tested whether reviews by different models would identify defects missed by a same-model review in a fresh session. The paper reports that the two approaches found some different errors, while warning that reviewer capability and model identity were not separated in the experiment.
One pairing beat two fresh-session reviews
The experiment used 30 artifacts: ten code modules, ten technical tutorials and ten presentation scripts. Each artifact contained five planted defects, for 150 errors in total. The study examined ten review conditions and 900 review sessions using three reviewer models from two developers.
For the headline comparison, the paper combined one cross-context review (CCR), in which the same model reviewed the artifact in a fresh session, with one cross-model review (XMR) by a top-tier reviewer. That second reviewer received the artifact without the original requirements. Together, the two reviews matched 85 of 150 planted errors; two CCR reviews matched 64. Song writes:
“At two review calls, one CCR plus one cross-model review matches more planted errors than two CCR reviews (56.7% vs. 42.7%; Holm-adjusted p=.006), but not significantly more than two reviews by the top-tier cross-model reviewer, so model difference and reviewer capability are not separated.”
Tae-Eun Song, author of the study
The comparison uses set-level results from one of three runs, not average performance across all review sessions. Across the three runs, the mixed pair’s coverage ranged from 44.7% to 56.7%, while two CCR reviews ranged from 30.0% to 42.7%; these reused runs are not independent replications. The gain over two reviews by the top-tier cross-model reviewer was not statistically significant.
Different reviewers found different defects
The findings of CCR and the cross-model condition had a Jaccard similarity of 41.2%, meaning they overlapped but also identified different errors. Performance varied by artifact type: CCR led on code, while cross-model review scored higher on documents. The paper describes those category-level results as descriptive, not statistically tested.
Reviewer capability also mattered in the reported results. The top-tier cross-model condition had an F1 score of 32.3%, while the lightweight cross-model condition scored 24.0%; CCR scored 28.6%. The top-tier result was not significantly different from CCR. The paper identifies the reviewer models by requested API parameters and notes that the endpoints did not confirm which model checkpoints were served. Because each capability tier was represented by one model, the experiment cannot distinguish the effect of model difference from the reviewers’ other characteristics.
A narrow test with measurement limits
The artifacts and planted errors were in Korean and were generated by a single model, Claude Opus 4.6, according to the study’s run logs. Review output language was not fixed. The automated matcher compared some review language against Korean ground-truth text, so the author says it may have undercounted English-language reviews. The planted errors were not independently annotated, and the matcher was not audited by hand.
The study also notes that the same-model baseline records lacked raw model outputs for independent checking. It excluded one baseline run of uncertain provenance and 14 failed cross-model calls from its main analysis, while reporting results with those sessions included as well. The author describes the findings as a snapshot of the experiment and says the study is intended to inform, not replace, human review.