The Pulse
Opera Tracks Whether Coding Agents Follow Through
Researchers introduced Opera, a critic framework that keeps track of its corrections while coding agents work. Across three benchmarks, the paper reports higher task-resolution rates and tests whether Opera-guided runs can train a smaller m
AI.info Team · Written with AI ·
A paper revision posted October 8 describes Opera, a framework that keeps a coding agent’s correction open until evidence shows the underlying problem is fixed. Rather than offer a one-off critique and move on, Opera checks the agent’s later actions against the issue it originally identified. The researchers report improved task-resolution rates across three coding benchmarks, and show that runs guided by Opera can also provide training data for a smaller model.
Opera is the name of the research framework in the paper, not a feature announcement from the Opera browser company. The work targets a common weakness in long coding tasks: an agent may repeat a failing command, edit the wrong code, or claim completion before verifying its changes. Advice can also cause trouble when it mistakes an unfinished but reasonable step for an error.
One correction stays open until the evidence changes
“We present Opera, a verbal critic framework that treats each correction as a persistent note, followed until the diagnosed problem is resolved.” Kai Mei et al., “Opera: A Verbal Critic Framework for Long-horizon Coding Agents”
The framework combines scheduled reviews with immediate checks for events such as repeated actions, failed tool calls, claimed completion, or a final submission. A review does not guarantee feedback: Opera can stay silent when it finds no justified intervention. When it does flag a problem, it uses one of nine typed operators to identify the kind of failure and specify evidence, corrective guidance, and a criterion for resolution.
Opera keeps at most one correction open at a time. An admission audit checks whether the evidence supports the diagnosis and whether the proposed correction is necessary; a separate release audit decides whether the fixed resolution criterion has been met. The system tracks whether the agent tried the advice separately from whether the problem was actually solved, so an attempted change alone does not close the note.
Results vary across three coding benchmarks
The team tested Opera on Terminal-Bench 2.1, a 100-task subset of SWE-Bench Pro, and DeepSWE v1.1, using four policy models. The paper reports maximum improvements over agents without a critic of 12.4 percentage points on Terminal-Bench, 15.0 points on SWE-Bench Pro, and 8.9 points on DeepSWE. Those are best gains across the tested models, not a single agent’s result across every benchmark.
In a comparison using Qwen3.8-27B as the coding policy, Opera raises resolution from 65.9% to 73.8% on Terminal-Bench 2.1, from 75.3% to 79.3% on the SWE-Bench Pro subset, and from 32.4% to 41.3% on DeepSWE v1.1. It records the highest mean rate among the tested critic methods on all three, but its lead over the strongest baseline on SWE-Bench Pro is only 0.6 percentage points. The authors also report that gains tend to shrink when an agent already solves more tasks without help.
Guided runs can train a smaller model
The researchers also fine-tuned Qwen3.5-9B on successful runs in which the model itself received Opera’s corrections. On held-out SWE-Bench Pro repositories, the model’s resolution rate rose from 28.7% to 38.9% without a critic at inference time. That result was close to the 39.5% achieved by fine-tuning on successful runs from the larger Qwen3.8-27B model.
The difference appeared on Terminal-Bench 2.1, a benchmark with a different task format and harness. Training on the larger model’s runs reduced Qwen3.5-9B’s rate there from 23.6% to 7.9%, while Opera-guided training produced 25.8%. The paper’s explanation is that Opera’s training examples preserve the smaller agent’s own actions and add targeted corrections, rather than replacing its experience with another model’s behavior.
A critic cannot supply missing capability
The results have clear limits. Qwen3.5-9B solved no DeepSWE tasks in the reported runs, with or without Opera, and the paper says self-critique adds inference computation that its comparison does not match. More capable critics generally helped more when the coding model struggled, while a model critiquing itself worked better when it already had enough capability to act on the advice.
Opera therefore improves how a coding agent receives and checks corrections in these tests; it does not guarantee that every diagnosis is right or that feedback can rescue an agent unable to solve the task. On DeepSWE, Qwen3.5-9B’s resolution rate remained at zero with and without the critic.