The Pulse
IBM Targets the 24-Point Consistency Gap in AI Agents
IBM researchers describe a method for finding and reducing inconsistent decisions in AI agents. The source's block quote is unattributed, so it was omitted.
AI.info Team ·
An AI agent can solve a task correctly and still be unreliable. IBM researchers say a ReAct agent running GPT-4.1 on the AppWorld benchmark succeeded on 77.4% of runs, but completed all five repetitions of a task only 53.0% of the time.
That 24.4-point difference is the problem targeted by a new addition to IBM’s open-source ALTK-Evolve toolkit. The system, described in a September 15 article on Hugging Face, analyzes an agent’s recorded decisions, identifies steps likely to change on another run, and turns those findings into guidance for future executions.
IBM’s researchers call the gap between average success and success on every repeated run a “consistency gap.” Their work argues that the standard accuracy number used in many agent evaluations can conceal failures that users experience when they ask an agent to perform the same operation again.
Average accuracy misses repeat-run failures
Most agent benchmarks report a mean pass rate across several attempts. IBM’s article calls that metric Mean@k: the average percentage of successful runs when a task is executed k times. A separate measure, Passk, counts only tasks that succeed on every one of those runs.
The distinction matters for workflows such as financial reconciliation, contract review, or software maintenance. A system that succeeds four times out of five may look strong in aggregate, yet still produce an inconsistent result whenever a user repeats the same request.
IBM reports that the consistency gap grows on harder AppWorld tasks, reaching 30 percentage points in the evaluation described in the article. The researchers say the issue is separate from general model capability: a model can perform well on average and still make unstable choices during a multi-step process.
ALTK-Evolve searches for unstable decisions
The new Consistency Analyzer works from a single recorded trajectory rather than rerunning the entire task against the environment. It replays each decision point in the existing trace and requests five completions by default, measuring how much the model’s output varies at that step.
The process requires one additional model call per decision point and does not need access to model logits or internal instrumentation. It also does not require a grader or a second end-to-end rollout, according to the researchers. The result is a scorecard that highlights decisions where small changes could send the agent down a different path.
Those decisions might involve selecting an API, forming a tool argument, choosing a search result, or deciding whether to retry. IBM’s explanation attributes the variation to near-tied token probabilities: when several possible next tokens have similar probabilities, small changes in serving conditions can alter the selected action, even when the agent runs at temperature zero.
Guidelines lift five-run consistency
ALTK-Evolve converts flagged decisions into “consistency guidelines,” which are stored alongside the toolkit’s existing memory and retrieved when a related task begins. The guidelines focus on patterns that can recur across tasks rather than preserving a single successful sequence.
One example comes from an AppWorld task that asks an agent to count completed activities in a SimpleNote entry. The generated guidance tells the agent to use a line-anchored regular expression instead of a plain substring count, because a note title may repeat the same marker symbol. Another guideline tells the agent to verify multiple search matches before selecting the correct note.
IBM evaluated the approach on 168 AppWorld test_normal tasks using a ReAct agent backed by GPT-4.1. After generating guidelines from one baseline trajectory per task, the researchers tested each task on five fresh runs.
Pass5 rose from 53.0% to 69.0%. Mean@5 increased from 77.4% to 81.0%, shrinking the consistency gap from 24.4 points to 12.0 points. IBM says the system improved average accuracy rather than trading it away for more repeatability.
The method transfers beyond one trajectory
The guidelines also improved performance on related task variants. On similar tasks from the same AppWorld scenarios, Pass5 increased by 13.0 percentage points, compared with a 16.0-point gain on the original tasks.
The researchers tested a weaker gpt-oss-120b model as well. Same-task Pass5 increased by 6.0 points, from 10.1% to 16.1%, while similar-task performance rose by 8.7 points. IBM presents the result as evidence that the system is capturing reusable failure patterns instead of memorizing one trajectory.
The toolkit is available in the ALTK-Evolve GitHub repository, which includes the Consistency Analyzer and the guideline-generation components used in the experiments. The broader framework stores agent trajectories, extracts operational guidance, and injects relevant instructions into later runs.
A different standard for agent reliability
IBM’s proposal does not replace ordinary accuracy measurements. It adds a stricter question: can an agent complete the same task repeatedly without changing its answer or breaking its workflow?
That distinction gives developers a way to separate a capable agent from a dependable one. The company’s results do not show that every agent will become consistent, but they demonstrate a measurable reduction in repeat-run failures without requiring a larger model or full task replay.
For teams deploying multi-step agents, IBM recommends reporting Passk beside Mean@k. The two figures expose whether a system’s success rate reflects dependable behavior or a sequence of occasional wins.