The Pulse
Microsoft: 67.24% of failed runs looked complete in a 12-model test
Microsoft’s ThinkingBox evaluation found that many failed agent runs still showed signs of completion in the agent’s final response and tool activity. The post also describes repeated-success comparisons and a reinforcement-learning experim
AI.info Team · Written with AI · fact-checked ·
Microsoft’s ThinkingBox evaluation found that 67.24% of failed agent runs still showed signs of completion in the agent’s final response and tool activity. The company checked the database records left behind, rather than relying on an agent’s account of what it had done. Microsoft published the post on Nov. 6, 2026.
When tool activity masks a failed task
Of 121,680 valid trials across 12 models, 79,853 failed the executable check. In 67.24% of those failures, an agent gave a final response signaling completion without asking a question, called at least one tool that changes state, and received no explicit error in its final tool response. The final database differed from the expected one in 98.95% of failures; 77.61% had a wrong field value, 43.30% included a change nobody requested, and 25.36% missed a required change.
Microsoft labels one dominant failure signature for each failed run. Averaged across the evaluated models, 79.9% of failures were classified as tool-usage failures, including tool errors, failed preconditions or empty lookups that the agent did not recover from. The authors caution that these labels describe what the trace shows, not a proven root cause.
Passing once is not the same as passing every time
Microsoft’s expanded comparison reports results for 507 tasks, with each task attempted 20 times. Kimi-K3 passed 476 tasks at least once, but passed only 68 on all 20 attempts; Claude Opus 5 passed 401 at least once and 241 on every attempt. The results show how a ranking based on a model’s best attempt can differ from one based on consistent success.
GPT-6 Astra scored 58.31% on pass@1, below GPT-5.4’s 65.36%, but passed 231 tasks on all 20 attempts compared with GPT-5.4’s 128. The post says the all-20 measure is more relevant when an agent changes production records without review, while noting that 20 successful attempts do not guarantee success on a later attempt.
Inference costs change with the success measure
Microsoft estimates inference costs for a run of 507 tasks with 20 attempts each, counting failed attempts in the totals. The estimates use OpenRouter pricing as of 2026-09-20 and exclude the simulated user, judge, infrastructure, negotiated rates and downstream failure costs. They are estimates, not cloud bills.
GPT-5.6 Sol costs an estimated $0.127 per successful attempt and $9.76 per task passed on all 20 attempts. Qwen3.8-27B’s successful attempts cost $0.177 each, while each task it passed on all 20 attempts cost $24.36. The post’s comparison ranks models differently depending on whether it measures cost per successful attempt or cost per consistently passed task.
Microsoft turns the check into a reward
The executable check produces a binary verdict per attempt. For most tasks, it is based on deterministic checks of the final state; a minority also use a narrow judge-model response check for requirements that a database field cannot capture. Microsoft used the verdict as a reinforcement-learning reward for Qwen3.8-27B. The company trained the model with GRPO, using groups of eight attempts; groups in which all attempts pass or all fail provide no reward signal under that setup. Microsoft selected a fixed pool of 187 training tasks by probing candidate tasks with the base model and keeping those in the three middle difficulty bands.
The training run used full-parameter training with token-mean GRPO on 24 H100 GPUs, and the reported results come from checkpoint 50. Microsoft evaluated that checkpoint on all 507 benchmark tasks, with 20 attempts per task, and says no benchmark task was used in training. Training and evaluation tasks still shared the ThinkingBox framework, simulated-user design, grading approach and task style.
The trained model became more repeatable
Qwen3.8-27B’s pass@1 rose from 51.70% to 60.78% after training. The number of tasks it passed on all 20 attempts increased from 38 to 107, while the number it passed at least once fell from 453 to 446. Its estimated cost per all-20 task fell from $24.36 to $8.00; the post says those after-training costs apply the base model’s listed rates to measured token use and exclude the training run.
In the hotel example shown in Figure 1, the trained model left the booking and payments untouched in all 20 attempts. All expected records matched in 19 attempts; one attempt missed an expected tag. Microsoft also reports gains from smaller LoRA runs: Qwen3.6-27B rose from 32.94% to 53.17% pass@1, and Qwen3.5-9B from 5.65% to 13.33%.
The benchmark and training tasks are synthetic reconstructions of enterprise workflows with simulated users, not production traffic. Microsoft says it has not measured transfer to unrelated agent settings, and notes that a reward based on executable checks rewards only what those checks specify.