Critique, Debate, and Result Aggregation

Critique agents can identify missing evidence, contradictions, unsafe actions, or rubric violations. Debate can surface alternatives when the participants know different things, or work in different ways. Where that has been measured, the gain is large.

Two expert LLM debaters argue opposite answers to a QuALITY reading-comprehension question, and a non-expert judge decides between them. Khan and colleagues ran that experiment in 2024. The abstract gives the result in one line: “We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%).” Note what moved those numbers. Not the debaters converging. A judge weighing two opposed cases that each had to be argued from the text.

Agreement is not verification. Aggregation should rest on tests, on evidence, on calibrated rubrics, or on an accountable person deciding — not on whoever argued most fluently. The 76% and the 88% are what a structured disagreement bought. They are not what mutual assent buys, and the two are routinely confused.

Independent re-running is what separates them. Two critics agreeing tells you they made the same call. It does not tell you either of them checked. SWE-bench Verified ships 500 human-validated tasks, by the benchmark's own documentation. Epoch AI runs that set on its own infrastructure and reports fewer: “Epoch evaluations of this benchmark use 484 samples that are validated on our infrastructure”. Sixteen tasks disagree. The gap only became visible because a second party rebuilt the setup instead of inheriting it. Two runs of the same harness are one opinion counted twice.

Multi-agent systems fail in patterns, and in 2025 Cemri and colleagues counted the patterns. Their MAST-Data release holds 1600+ annotated traces collected across 7 multi-agent frameworks. Rigorous analysis of 150 of those traces produced the MAST taxonomy: 14 failure modes in 3 categories, validated at inter-annotator agreement κ = 0.88.

In the labelled failure distribution of Figure 4 — 210 traces, 30 per system — task verification accounts for 21.3% of all failures. Premature termination (FM-3.1) is 6.20%, no or incomplete verification (FM-3.2) 8.20%, incorrect verification (FM-3.3) 9.10%. These are systems that had a reviewer. The reviewer ran, and the artifact shipped broken anyway.

What the reviewers were doing is the useful part: “During our GT analysis of MAS traces, we find that many existing verifiers perform only superficial checks, despite being prompted to perform thorough verification, such as checking if the code compiles or if there are leftover TODO comments.” A check that the code compiles runs on every artifact. It passes on almost all of them, including the wrong ones.

The same paper priced the repair. Adding one high-level task-objective verification step to ChatDev yielded a +15.6% improvement in task success on ProgramDev. The reviewer was already there. What it looked at was the problem.

Self-critique, an Independent critic, and Panel aggregation buy progressively more independence at progressively higher cost. Self-critique costs one more model call and inherits every blind spot of the first pass. An independent critic starts from its own context. A panel pays for several.

The price of the cheapest option has been measured. Intrinsic self-critique means self-correction with no external feedback: the model rereads its own answer and tries again. Huang and colleagues tested it in 2024. On GSM8K, GPT-4 fell from 95.5% to 91.5% after one self-correction round, and to 89.0% after two. GPT-3.5 fell from 75.9% to 75.1% to 74.7%. Over the same rounds the call count rose from 1 to 3 to 5. That is two extra calls per round to go from 95.5% to 89.0%. Their summary: “We observe that, after self-correction, the accuracies of all models drop across all benchmarks.”

The control condition in that paper is the whole argument for paying more. Given oracle labels — an external signal telling the loop when it was wrong — the same procedure instead rose to 97.5% and 84.3%. The lift was never in the reflection. It was in the outside information, and it vanishes when the outside information does.

Whichever of the three you buy, the same test applies at the end. The critiques have to name concrete failures that somebody else can reproduce. And they have to survive the pull toward rewarding whichever voice sounds most sure of itself.

Criticism becomes useful when a Candidate artifact meets a Critique assignment, produces Counterevidence, and forces a Revision. Revision and Aggregation should answer to different owners, and to different tests.

The third step is where a critique earns its keep, and it has a number on it. Ten math questions, judged in both orders, make 20 judgements. Zheng and colleagues counted how often GPT-4 accepted an incorrect answer as correct: 14 times out of 20 with the default prompt, 6 out of 20 with a chain-of-thought judge, and 3 out of 20 with a reference-guided judge that first generated its own answer before grading. The scoring rule was blunt: “A failure means when GPT-4 says an incorrect answer is correct.” The chain-of-thought judge thought harder about the same material and still missed 6. The reference-guided judge produced independent evidence first, and missed 3. Independence of evidence, not more discussion, is what moved the number.

The Aggregation step can also be a rule rather than a conversation. Self-consistency drops the single greedy answer and takes a deterministic majority vote over independently sampled reasoning paths. Wang and colleagues published it in 2023, and the abstract lists what the vote bought: “Our extensive empirical evaluation shows that self-consistency boosts the performance of chain-of-thought prompting with a striking margin on a range of popular arithmetic and commonsense reasoning benchmarks, including GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%) and ARC-challenge (+3.9%).” Two properties make that vote work. The paths are sampled independently, and the tally is mechanical. Against +17.9 on GSM8K, “whoever argued most fluently” has no number at all. A critique that names no evidence is a comment, and a comment cannot force a revision.

Without a protocol, critique drifts into commentary. So write one, for a workflow that already reviews its own output. The protocol says who criticises what, against which test, and what has to change before the artifact is accepted.

The 2024 debate study is a published example of exactly that, and it is worth copying. The 76% and the 88% were produced under a protocol rather than by asking for good faith. Debaters had a quotation tool. It marked a quote <v_quote> when it matched the source text and <u_quote> when it did not, mechanically, and judges were told to trust only verified quotes. That is the separation-of-evidence step made enforceable. An assertion about the passage either carries a machine-checked tag or it is visibly unsupported.

The protocol also capped length, deliberately and on the record: “To control for the quantity of information presented to the judge across protocols and mitigate the LLM judge verbosity bias, we restrict transcripts to 900 words in total, limiting consultants to 300 words per argument and debaters to 150 words”. Three rounds, 150 words each for a debater. A long, assured objection with no verified quote behind it cannot be written at all, let alone win. Given the 91.3% preference for padding measured on those judges, a word limit is not a tidiness rule. It is a control.

What the protocol collects should be a list of named defects, each with the input that triggers it. A critique that cannot be turned into a failing test is not yet a finding.

Use critique to generate error hypotheses and repairs. Use evidence to decide whether the repaired artifact is acceptable.

The thing to watch in a panel is that debate can optimize confidence and verbosity instead of correctness. A preference model picked the convincingly-written sycophantic answer over the plainly truthful one 95% of the time. Judges took padded lists over the originals 91.3% of the time. Neither failure announces itself in a transcript. Both read as a confident panel reaching a conclusion.

So accept the panel on its output rather than on its discussion: defects that were named, triggered, and then repaired. What that buys is measurable in the same studies. Three failures out of 20 for a judge that produced its own reference answer, against 14 out of 20 for one that argued from the candidate alone. And +15.6% on ProgramDev from adding a single verification step that checked the task objective instead of whether the code compiled. Reproduced means somebody else ran it and saw the same thing.