CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents

AI summary

Overview

• Research area: Natural Language Processing / LLM agents — specifically reliability training for long-horizon, stateful, tool-calling agents in dynamic environments.
• Technical level: Intermediate. The framework combines supervised fine-tuning, multi-agent data synthesis, and actor–critic style verification, but the core ideas are readable without a deep RL background.
• Scope: This paper introduces CAST, a three-stage training framework that turns sparse trajectory-level task outcomes into structured, action-level critique supervision, and evaluates it on four dynamic tool-calling domains (Retail, Airline, Telecom, Telehealth) across two Qwen3 model sizes.

What This Paper Is About

Long-horizon tool-calling agents can fail catastrophically from a single bad intermediate action — for example, refunding the wrong purchase — and these failures may not show up on every run, so reliability across repeated trials matters more than one-shot success. Existing approaches either rely on prompt-based critique agents at inference time (expensive, and with no supervision for why an action is wrong) or on optimization methods like rejection fine-tuning, which use only a binary trajectory-level reward and cannot say which step caused the failure. CAST addresses this by learning an explicit, action-level critique model that judges whether each proposed action is valid given only the information available at that step, and then using that critique signal to train a more reliable policy.

Key Contributions

1. CAST, a critique-aware training framework that converts trajectory-level outcomes into structured, action-level verification signals for training tool-calling agents.
2. An agentic data-generation pipeline that produces high-quality step-level critique rationales, used both to train a critique model (CAST-Critic) and to construct critique-enriched policy optimization data for CAST-Policy.
3. A verifier formulation grounded in partial observability: the verifier conditions only on the step-local context (history, observation, candidate action, tool set and domain constraints), never on future observations, while a teacher verification procedure may use privileged information available only at annotation time.
4. Extensive evaluation across two model sizes and four domains, showing improved reliability and reduced inference cost relative to more resource-intensive agentic baselines.

Main Findings

• In-domain Retail reliability improves, with the largest gains on pass^4. CAST-Policy-4B improves over the base instruct model by 18.9% on pass^1 and 10.4% on pass^4 (Retail). The paper's introduction separately summarizes that fine-tuned Qwen3 models improve over base instruct models by 22.6% on pass^1 and 12.94% on pass^4 in the in-domain environment.
• Critique supervision beats training on successful trajectories alone. CAST-Policy-4B reaches comparable pass^1 to RFT-4B but improves pass^4 by 4.3%. CAST-Policy-8B outperforms RFT-8B and Base-8B by 2.4% and 15.9% on pass^1, and by 2.6% and 9.6% on pass^4.
• Smaller CAST policies beat a much larger model on repeated-run reliability. CAST-Policy-4B and CAST-Policy-8B have lower pass^1 than Qwen3-32B but outperform the best Qwen3-32B setup by 3.4% and 1.7% on pass^4, respectively.
• Out-of-domain generalization. Across out-of-domain settings, CAST achieves average gains of 3.7% on pass^1 and 5.1% on pass^4. On Airline and Telecom, CAST-Policy-8B paired with CAST-Critic-8B outperforms both the base instruct model and RFT-8B on pass^1 and pass^4.
• Agentic CAST transfers better than the standalone policy. The critique-guided (agentic) configuration shows stronger generalization to unseen domains than the standalone critique-aware policy.
• Frontier models make poor prompted critics. Fixing Qwen3-32B as the actor, adding no critique agent yields 31.7% average pass^1 and 11.5% average pass^4, while critique agents built from GPT-4.1 (22.8% / 6.0%), Qwen3-235B (24.1% / 3.4%), and Qwen2.5-72B (17.9% / 6.3%) all reduce performance. CAST-Critic-4B (31.8% / 15.9%) and CAST-Critic-8B (30.4% / 19.5%) instead improve it.
• CAST critics are far better calibrated. GPT-4.1-Critic flags 46.8% of all judged actions even when they are correct; CAST-Critic-4B and CAST-Critic-8B reduce this to 13.6% and 11.4%. The CAST critics correctly leave 62.2% and 64.7% of valid actions unflagged, versus 24.0% for GPT-4.1-Critic.
• Competitive against test-time agentic frameworks at lower helper-model cost. CAST outperforms IRMA on Airline by 3.0% on pass^1 and on Retail by 4.5% on pass^4, using 4B or 8B critique agents instead of the Qwen2.5-72B helper agents used by IRMA and FAMA. CAST introduces more overhead than single-pass inference, but token usage and latency remain within the range of existing agentic frameworks; specific latency and token figures are not reported numerically in the text.
• Large margins over failure-recovery and evolutionary baselines. Against PALADIN, CAST-Critic-8B improves pass^1, pass^3, and pass^4 by 20.9, 15.3, and 14.9 percentage points. Against EvoTool, it improves those metrics by 2.6, 4.5, and 4.3 percentage points.
• Verification budget is not monotonic. Varying refinement rounds from 2 to 5, Qwen3-32B achieves its best pass^4 with 3 rounds, whereas CAST-Policy-4B reaches its best pass^4 with 4 rounds. CAST-Policy-4B uses fewer tokens than Qwen3-32B in almost all settings.
• Critics do not over-trigger revisions. The critique agents flag only 33.2% of agent actions as unsuitable per task.
• Trained policies accept critique more readily. CAST-Policy agents show roughly 2 times less resistance to verification feedback than non-fine-tuned agents.
• Critique feedback is actionable. Feedback from CAST Critics leads to successful correction in 82–88% of cases across hallucination, domain violation, and wrong tool call. CAST Critics substantially outperform GPT-4.1 at detecting hallucinations, are comparable or slightly better on domain violations, and remain competitive on wrong tool calls.

Methodology in Plain English

CAST runs in three stages, all using Qwen3-family models as backbones.

Stage 1 — Collect trajectories. The team used the official τ-Bench Retail training split of 500 tasks and ran a teacher policy (Qwen3-32B in ReAct mode) five times per task at temperature 0.7, with Qwen2.5-72B as the simulated user, producing 2,500 trajectories spanning both successes and failures.

Stage 2 — Annotate every action. An agentic verification procedure labels each action in each trajectory. Specialized LLM sub-agents extract the relevant domain rules, identify candidate tools, and separately check three failure types: hallucination (unsupported assumptions or fabricated arguments), domain violation (conflicts with system rules or task constraints), and wrong tool usage. An orchestrator aggregates their outputs into a binary label plus a written rationale. The verifier can use privileged information available only at annotation time, but the resulting labels are tied to the step-local context that the deployed critic will actually see. Qwen3.5-27B and Qwen3.5-122B-A10B were tried as verification agents, with Qwen3.5-122B-A10B producing higher-quality annotations.

Stage 3 — Train the critic and then the policy. CAST-Critic is optimized with a combined objective balancing rationale generation and verification classification, keeping critique learning separate from policy optimization. The trained critic is then redeployed during a second round of data collection over the same 500 tasks, this time verifying the teacher's actions mid-interaction; only trajectories with reward 1 are kept, yielding critique-enriched successful trajectories. CAST-Policy is then fine-tuned on these with supervised learning, so the verification signals are internalized into the actor's context. At inference, CAST-Policy can run together with CAST-Critic for explicit critique-guided interaction, or alone as a standalone tool-calling agent.

Fine-tuning used a maximum sequence length of 32,768, learning rate 5 × 10^-6, 3 epochs, 4 devices, total batch size 16, AdamW, and a cosine learning rate scheduler.

Why This Matters

Reliability — not peak one-shot accuracy — is the blocker for deploying agents in settings where actions are irreversible. By showing that a 4B or 8B critic can outperform prompted frontier critics and that small policies can beat much larger models on pass^4, the paper reframes critique supervision as a trainable, cheap component rather than an inference-time luxury. It also connects to reliability-oriented benchmarking by reporting pass^k throughout, which measures consistency across repeated runs rather than a single lucky trajectory.

Real-world applications:

• Customer-service agents that handle retail order cancellations, returns, exchanges, and address changes, where a mistaken write action can be irreversible.
• Airline reservation workflows such as booking, modification, cancellation, and refunds, where domain-policy violations have financial and compliance consequences.
• Telecom account-support agents operating under domain-specific rules and tool sets.
• Telehealth interaction agents, where acting on fabricated or unsupported information carries elevated risk.

Industry relevance: The result that 4B and 8B critique and policy models can be competitive with or better than much larger models and resource-intensive helper-agent frameworks, at practical token and latency cost, makes critique-aware training attractive for cost-sensitive production deployments. The out-of-domain gains on Airline, Telecom, and Telehealth after training only on Retail suggest the approach can transfer to new domains without large per-domain annotation efforts.

Future Directions

• Replace supervised fine-tuning with on-policy reinforcement learning. The authors note that CAST's SFT training provides stable optimization but does not train the policy through on-policy interaction with critique feedback; the learned critic could instead serve as a feedback model for RL, letting the policy learn more adaptive correction strategies from its own exploration.
• Model downstream consequences, not just local validity. CAST verifies the current action rather than modeling how it affects later states; forward-looking rationales describing downstream effects could provide stronger supervision for long-horizon decisions.
• Pin down the verification budget. Since the best pass^4 came at 3 rounds for Qwen3-32B and 4 rounds for CAST-Policy-4B, how to select the optimal verification budget per actor remains an open question.
• Attribute the reported in-domain gains precisely. The paper reports both a 22.6% pass^1 / 12.94% pass^4 improvement for fine-tuned Qwen3 models in its introduction and an 18.9% pass^1 / 10.4% pass^4 improvement for CAST-Policy-4B in its results section, so clarifying which model-size aggregation each figure describes would help future comparisons.

Target Audience

Researchers and engineers working on LLM agents, tool use, and multi-turn reliability will get the most from this paper, particularly those interested in process-level supervision, verifier or critic training, and actor–critic formulations with textual feedback. It is also relevant to practitioners deploying customer-service or transactional agents who need repeated-run consistency and want to know whether small, fine-tuned models can replace large prompted critics. Readers focused on reinforcement learning will find the critique-data-generation pipeline useful, though the paper's own limitations section notes that it stops short of on-policy RL.

Authors’ abstract

Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments. In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution. Such failures may not appear in every single run, but can emerge across repeated trials, making reliability across steps and trials critical. However, ensuring agentic reliability is challenging: even frontier LLMs struggle to explain why an action may be wrong, especially in long, intertwined trajectories governed by domain-specific policies. Much recent work relies on prompt-based critique agents, while optimization-based methods lack a systematic way to produce rich verification rationales for training. We address this gap with CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization. CAST analyzes agent trajectories to synthesize structured rationales explaining action validity under partial observability. The resulting critique model is used to construct critique-aware training data for optimizing the policy model. Fine-tuning Qwen3-family models on dynamic tool-calling benchmarks, CAST improves reliability across domains, outperforming GPT-OSS-120B by over 10% pass^4 on Retail tasks and yielding an additional 9% improvement on Telehealth in an out-of-domain setting. These results demonstrate that critique-aware training improves the robustness of LLM agents in realistic dynamic environments.