Opera: A Verbal Critic Framework for Long-horizon Coding Agents

AI summary

Overview

• Research area: Natural Language Processing / LLM-based coding agents, test-time intervention and critique, and training-data generation via critic-guided rollouts.
• Technical level: Advanced. The paper assumes familiarity with agent harnesses, test-time scaling, process reward models, and supervised fine-tuning.
• Scope in one sentence: The paper proposes Opera, a critic that treats every correction as a persistent "note" — scheduled, audited before delivery, and tracked until the diagnosed problem is actually resolved — and shows gains as a test-time critic and as a source of approximately on-policy training data.

What This Paper Is About

Coding agents that work on repository-level tasks over many turns often repeat failing actions, explore redundantly, or claim completion before the work is verified. A critic that intervenes during execution is the natural fix, but judging unfinished work is hard: incomplete evidence makes a reasonable intermediate step look like a mistake. Prior work builds critics that evaluate trajectories and emit feedback, but rarely checks what happens after the feedback is delivered. Opera's goal is to supervise the whole lifecycle of a correction — deciding when to intervene, ensuring the intervention is justified by visible evidence, and verifying that the diagnosed problem is actually gone rather than merely acknowledged.

Key Contributions

1. Framework. The paper formulates verbal criticism as the management of persistent corrective notes and introduces Opera, which decides when to review, audits feedback before delivery, and follows each correction until resolution during long-horizon coding-agent runs.
2. Test-time critic. Opera improves task resolve rate by up to 12.4, 15.0, and 8.9 percentage points on Terminal-Bench 2.1, SWE-Bench Pro, and DeepSWE v1.1 across four policy models, and outperforms competitive critic baselines. It also improves most policy models when the policy model critiques itself.
3. Training recipe. Opera-guided student rollouts provide approximately on-policy training data. Fine-tuning Qwen3.5-9B on them matches distillation from a stronger model on held-out SWE-Bench Pro repositories (+10.2 pp) while preserving its Terminal-Bench 2.1 performance, whereas teacher-rollout fine-tuning does not.
4. Explicit design mechanisms for the three challenges of criticism: hybrid review scheduling (when), operator-typed diagnosis plus an admission audit (whether the feedback is justified), and separate adherence/resolution tracking with a release audit (whether the intervention worked).

Main Findings

• Best mean resolve rate on all three benchmarks. With Qwen3.8-27B as the policy, Opera scores 73.8 ± 2.3 on Terminal-Bench 2.1 (89 tasks), 79.3 ± 3.5 on a 100-task SWE-Bench Pro subset, and 41.3 ± 3.1 on DeepSWE v1.1 (113 tasks), versus 65.9 ± 1.7, 75.3 ± 1.2, and 32.4 ± 1.0 for the non-critic agent (+7.9, +4.0, +8.9 pp). Its margin over the strongest baseline is 1.9 pp on Terminal-Bench 2.1 and 1.8 pp on DeepSWE, but only 0.6 pp on SWE-Bench Pro, where the non-critic agent is already strong.
• Baselines compared. SWE-PRM (68.9 / 77.7 / 36.6), SWE-Search (70.0 / 77.3 / 38.6), LLM-as-verifier (71.2 / 78.7 / 38.9), and Agentic Rubrics (71.9 / 78.3 / 39.5) all trail Opera on the mean of every benchmark.
• Weaker policies benefit more. Opera improves every policy model on every benchmark with one exception: Qwen3.5-9B resolves no DeepSWE task, with or without the critic. Qwen3.5-9B gains 12.4 and 15.0 pp on Terminal-Bench 2.1 and SWE-Bench Pro, and gains tend to shrink as the non-critic resolve rate rises. Muse Glimmer-30B on DeepSWE, starting from a low base, has its resolve rate nearly tripled by feedback, though the absolute gain remains limited.
• Edit-stage operators dominate. Across all four policy models, Edit-stage operators account for most delivered notes, about 80% for Qwen3.8-27B; Search and Submit operators are rare. Qwen3.5-9B mostly receives requirement-contract notes (misreading requirements), while Qwen3.8-27B and DeepSeek-V4-Flash shift toward state-transition notes (faulty execution logic); Muse Glimmer-30B shows a flatter distribution with more focused-verifier notes.
• Operator ablation. The full operator set achieves the highest mean on all three benchmarks (73.8 / 79.3 / 41.3), but Edit-only (71.9 / 79.0 / 40.1) and Non-edit (70.2 / 78.3 / 39.5) each retain most of the gain; the full set's margin over Edit-only is small (0.3 pp on SWE-Bench Pro) and within run-to-run variance. Non-edit operators are only about one fifth of delivered notes but recover over half of the full gain on Terminal-Bench 2.1, three quarters on SWE-Bench Pro, and about four fifths on DeepSWE.
• Rescue–regression trade-off. On trials receiving notes, Opera improves over expected no-critic success for all models, but stronger models convert feedback into more rescues of previously unsolved tasks and also regress more often on tasks they would otherwise solve, while weaker models rarely regress but are rescued less often.
• Audits matter most where evidence is easy to misread. Removing both admission and release audits drops the Terminal-Bench 2.1 gain from 7.9 to 2.6 pp (about two thirds lost) and the DeepSWE gain from 8.9 to 4.5 pp (about half); SWE-Bench Pro falls from 4.0 to 3.0 pp.
• Robust to critic choice, including self-critique. With the policy itself (Self), Claude, or GPT as critic, all three improve over the no-critic agent in all but one of the remaining 33 policy–benchmark–critic combinations (Qwen3.5-9B on DeepSWE is excluded because it resolves no task under any critic). On DeepSWE, self-critique improves Qwen3.8-27B and DeepSeek-V4-Flash by 6.2 pp each, matching or exceeding gains from Claude. With GPT as critic, Qwen3.5-9B improves at least 12.4 pp whereas self-critique yields at most 2.7 pp; Muse Glimmer-30B gains 11.0 pp on SWE-Bench Pro with GPT and nearly triples its DeepSWE resolve rate.
• Training result: on-policy data generalizes, teacher data does not. Base Qwen3.5-9B scores 28.7 ± 1.9 on held-out SWE-Bench Pro and 23.6 ± 1.9 on Terminal-Bench 2.1. Training on Qwen3.8-27B teacher rollouts gives 39.5 ± 1.4 on SWE-Pro but collapses Terminal-Bench 2.1 to 7.9 ± 1.9. Training on Opera-guided rollouts gives 38.9 ± 2.4 on SWE-Pro while preserving Terminal-Bench 2.1 at 25.8 ± 2.2.
• Adherence is not resolution. Stronger models act on feedback almost immediately and close most notes. Qwen3.5-9B edits after feedback as often as Muse-glimmer-30B but resolves only about half of its requirement-contract and state-transition notes, while Muse-glimmer-30B rarely edits right away yet eventually resolves most issues. Across models, agents seldom test unprompted, with the focused-verifier operator the main trigger for testing, and failure-triage notes have the lowest edit and closure rates; failure triage helps only Qwen3.5-9B and yields slightly negative gains for the other models.
• The critic stays silent most of the time. In nearly all settings, fewer than a third of reviews produce a candidate. Qwen3.5-9B draws the most interventions on Terminal-Bench 2.1 and SWE-Bench Pro; GPT-5.6 generally delivers the most notes. With GPT-5.6 most candidates are delivered, whereas self-critique delivers only a third, with nearly half filtered before the audit — the dominant filter being the open-note constraint, since the self-critic keeps proposing new issues instead of following up on its open note.

Methodology in Plain English

The three questions Opera is built around. The authors frame a critic's feedback as an intervention whose value is only revealed afterwards, and organize the system around (1) when to intervene, (2) whether the feedback is justified, and (3) whether the intervention worked.

When to review (hybrid scheduling). A pure periodic review reacts too slowly — an agent can repeat a failing command several times before the next check — while purely event-driven review misses silent failures such as confidently editing the wrong file. Opera does both: a periodic review every k turns plus immediate reviews on five events — Idle (no visible progress), Repeat (repeated actions), Error (failed actions or tool calls), Claim (reported completion), and Submission (final patch). Simultaneous signals produce a single review, and an event-triggered review resets the periodic timer. A review does not force an intervention; the critic may let the agent continue. In the implementation, most requests pass through unreviewed, with checks every 5 agent turns (10 on the longer DeepSWE tasks), starting from turn 3 and at least 2 turns apart.

How to intervene (operator-typed diagnosis). Free-form critique tends to be generic or to mix concerns. Opera requires each diagnosis to use one of nine typed operators spanning five repair stages:

• Search: repo_localization_search, implementation_target_shift
• View: implementation_readiness_review, failure_signature_triage
• Edit: requirement_contract_review, diff_scope_review, state_transition_review
• Test: minimal_repro_or_focused_verifier
• Submit: submission_readiness_review

Each operator is specified by a contract, in the spirit of Design by Contract, stating when it applies, the evidence it requires, the correction it may request, when the issue counts as resolved, and which problems it excludes — the exclusion clause keeps operators from overlapping, and the resolution clause supplies the criterion for later checking. At each review the critic either returns continue or selects one supported issue and the smallest correction for it, naming the operator, citing evidence, and specifying location, current behavior, required change, and how to verify.

Whether the intervention worked (audited note management). Each correction is a persistent note, open or closed, with at most one open note at a time. An admission audit gates creation and updates, accepting or rejecting proposals without rewriting them — rejecting, for example, optional cleanups unrelated to the task or reminders that repeat guidance without new evidence. Updates may change the operator and guidance but preserve the note's identity and fixed criterion. After delivery, adherence (checked by rules: did the agent attempt the correction?) is tracked separately from resolution (judged against the fixed criterion: is the original problem gone?). Closing a note requires a release audit with supporting evidence, such as execution results for behavioral requirements or visible edits for static ones; accepted closure archives the note and removes its guidance from context. Closure is processed before admission, so one review can close an issue and open the next.

Deployment. Opera runs as a proxy between the agent and its model endpoint, so the harness sends requests unchanged. The critic never executes commands, accesses files, or runs tests, and it influences the agent through a single channel: inserting at most one message into a forwarded request. If it times out or returns an unparseable reply, the request is forwarded unchanged, so the critic can never block the agent. The critic reads the task and rollout bounded to its context window — the 30 newest messages verbatim, long messages shortened to their beginning and end, older messages elided — plus the open note, its criterion, and the history of earlier notes.

Evaluation setup. Three benchmarks: Terminal-Bench 2.1 (89 tasks) with Terminus-2, SWE-Bench Pro (100 tasks) with OpenHands, and DeepSWE v1.1 (113 tasks) with mini-swe-agent. Harbor manages tasks and Pier provides network isolation for anti-cheating. Four policy models: Qwen3.5-9B (thinking), Muse-glimmer-30B (high), Qwen3.8-27B (xhigh), and Deepseek-V4-Flash-0731 (max). GPT-5.6-Sol (high) is the default critic. Baselines are adapted to share Opera's review schedule for fairness: SWE-PRM sees the task and eight most recent steps against a twelve-inefficiency taxonomy; SWE-Search uses its value function with a −100 to 100 reward scale; LLM-as-verifier scores on a 1–20 scale and accepts at least 14; Agentic Rubrics writes up to eight repository-grounded rubric items.

Training recipe. SWE-Bench Pro is split by repository into 270 training tasks and 108 held-out tasks. Two trajectory sources are compared: Opera-guided student rollouts (Qwen3.5-9B policy, Qwen3.8-27B critic) and teacher rollouts (Qwen3.8-27B policy). Sixty-seven training tasks are selected where the base Qwen3.5-9B resolves the task at most once in three runs without a critic, at least one critic-guided run resolves it, and Qwen3.8-27B resolves it without a critic. Each agent turn becomes one sample; critic notes are removed from contexts, and on the 4.9% of turns where an audited note was delivered, its diagnosis is rewritten as the model's own reasoning and prepended to that turn's thought. Training uses full-parameter bf16 SFT on Qwen3.5-9B with ms-swift and DeepSpeed ZeRO-3, 128k max sequence length, learning rate 2×10⁻⁶ with cosine schedule and 3% warm-up, 3 epochs, seed 42.

Why This Matters

The paper's distinctive move is to treat a critique not as a one-shot message but as a tracked commitment with a fixed resolution criterion, and to separate "the agent did what I said" from "the problem is gone." It also shows that the same mechanism that helps at inference time produces training data that generalizes better than distillation from a stronger model — a practical route to self-improvement for open-weight coding agents, because the trajectories contain the student's own actions rather than a teacher's.

Real-world applications:

• Autonomous software maintenance. Agents that fix repository-level bugs over long horizons benefit from timely, audited correction rather than repeated failing commands.
• Self-hosted / open-weight coding assistants. The training recipe improves a 9B model without requiring a critic at inference time, which matters when deploying under compute or cost constraints.
• Training pipelines for engineering agents. Critic-guided student rollouts supply approximately on-policy supervision at the states the student actually visits, using a stronger model's expertise through critique rather than through imitation.
• Agent-harness and platform integration. Because Opera is a proxy that interacts only through execution traces and natural-language feedback, it plugs into existing harnesses without updating agent weights.

Industry relevance: The evaluation spans three benchmarks, four policy models, and two harnesses (OpenHands and Terminus-2). The finding that teacher-rollout fine-tuning collapses Terminal-Bench 2.1 resolve rate from 23.6% to 7.9% while Opera-guided training holds it at 25.8% speaks directly to teams that fine-tune coding agents on distilled trajectories and then find that performance does not survive a change of harness.

Future Directions

• Scale the training recipe. The authors note the training results use supervised fine-tuning on a single model with one out-of-domain benchmark; scaling to larger datasets is left open.
• Iterate the recipe. Repeating the process with the trained model as the new policy is suggested as future work.
• Combine critic feedback with reinforcement learning to internalize self-correction rather than relying on supervised imitation of corrected turns.
• Handle low-capability policies. A critic cannot substitute for missing capability: Qwen3.5-9B resolves no DeepSWE task with or without the tested critics, and self-critique is less effective when the policy shares the critic's blind spots. Deciding when no critic can help remains an open question.

Target Audience

Researchers and engineers working on LLM-based coding agents, test-time compute allocation, process reward models, and agent self-improvement. It is most useful to readers already comfortable with agent harnesses, trajectory-level

Authors’ abstract

Long-horizon coding agents need timely corrections, yet feedback can be ineffective or even harmful when it misjudges ongoing work or fails to address the underlying problem. Existing critics focus on evaluating trajectories and generating feedback, but rarely track what happens after feedback is delivered. We present Opera, a verbal critic framework that treats each correction as a persistent note, followed until the diagnosed problem is resolved. Opera decides when to review through periodic and event-driven triggers, diagnoses issues with typed operators, audits feedback against visible evidence before delivery, and tracks the agent's subsequent actions to distinguish mere compliance from actual resolution. As a test-time critic, Opera improves the resolve rate of non-critic agents by up to 12.4, 15.0, and 8.9 percentage points on Terminal-Bench 2.1, a SWE-Bench Pro subset, and DeepSWE v1.1, respectively, across four policy models, and achieves the highest mean resolve rate among competitive critic baselines on all three benchmarks, and also improves policy models when the policy critiques itself. Beyond inference, Opera-guided rollouts provide approximately on-policy training data: fine-tuning Qwen3.5-9B on them improves its resolve rate on held-out SWE-Bench Pro repositories by 10.2 percentage points without a critic at inference time, matching fine-tuning on rollouts from a stronger model, while preserving its performance when switching harness, i.e., from Openhands to Terminus-2, which the latter substantially degrades. Our code is available at: https://github.com/dongyuanjushi/Opera.