The Pulse
Retrieved Tool Output Cuts GPT-5.1 Accuracy by 11.3 Points
A study of AI agents using the PHREEQC geochemistry simulator found that retrieving tool output instead of receiving it inline reduced GPT-5.1’s pooled accuracy by 11.3 percentage points. The researchers traced many errors to agents relying
AI.info Team · Written with AI · fact-checked ·
“LLM agents that compute the right answer can still answer wrongly.”
Ke Zhang, Sahchit Chundur, Jerry Peprah Owusu, Mohammad Javad Abdolhosseini Qomi and Maziar Raissi, the paper’s authors, write.
A new study finds that the way an AI agent receives a tool’s results can change whether it answers correctly, even when the agent’s simulation produces the right value. In experiments with the geochemistry simulator PHREEQC, GPT-5.1’s pooled accuracy fell by 11.3 percentage points when its report appeared as a searchable section index instead of being placed directly in the model’s context.
The paper, submitted to arXiv on October 19, examines a practical choice for agent builders: whether to put lengthy tool output directly in front of a model or save it for the model to retrieve as needed. The authors trace errors not simply to failed calculations, but to agents choosing the wrong evidence after a calculation succeeds.
158 PHREEQC questions expose a retrieval trade-off
The researchers tested six models, including GPT-5.1, GPT-5.2, GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6 and Gemini 3 Flash Preview, on multiple-choice questions requiring PHREEQC, an aqueous-geochemistry simulator. They retained 158 questions after re-running an original pool of 200 and checking that each answer matched a stable, uniquely correct result. Some simulator reports reached 846,800 characters, making them long enough to create a real context-management problem.
In the inline condition, agents received the report in their context, truncated at 100,000 characters when necessary. In the retrieval condition, they received an index of report sections and could read selected line ranges. The study also tested a keyword-search condition and, for some models, a hybrid approach that placed reports inline when they fit.
Results varied by model. In the researchers’ pooled comparison across runs, retrieval reduced GPT-5.1 accuracy by 11.3 percentage points. That pooled estimate differs from the single-run figures: accuracy was 67.7% inline and 57.0% under retrieval, a 10.7-point gap. The pooled difference was statistically significant after the study’s correction for multiple comparisons. GPT-5.2’s estimated decline was 7.9 points, but that pooled result did not meet the corrected significance threshold. Claude Opus 4.6, Claude Sonnet 4.6 and GPT-5.4 showed no detectable net accuracy change in that comparison. The researchers excluded Gemini 3 Flash Preview from the analyses below because it often did not provide a parseable answer.
Agents often trust their own files over the simulator
To locate the failures, the authors re-ran 11,007 logged simulator calls and examined whether an agent produced the correct value, saw it, and then answered correctly. GPT-5.1 and GPT-5.2 often skipped the official report under retrieval and read their own selected-output files instead. Those files could print more digits than the report, leaving the agent to round a value to match the answer choices.
That extra precision became a trap. For example, a file could show a pH of 2.75065 while the simulator report printed 2.751, the correct multiple-choice value. GPT-5.1 and GPT-5.2 frequently truncated the longer number to 2.750, a neighboring wrong option. When GPT-5.2’s already-logged simulation was held fixed and only the report’s delivery changed, retrieval still lowered accuracy by 9.4 points.
GPT-5.4 illustrates why a final score can conceal such a failure. The model also sometimes relied on its own output and lost accuracy at the evidence-use stage, but its simulations produced the keyed value more often under retrieval. Those opposing effects left its overall accuracy nearly unchanged.
Naming the authoritative report helped; rounding advice did not
The researchers then told agents to read the relevant section of the simulator’s report and treat its printed value as authoritative. That instruction raised accuracy by 6.3 to 7.9 points over plain retrieval for the three GPT models tested in this intervention, while sharply reducing report-skipping and rounding errors. A separate instruction to round carefully, without identifying which file to trust, did not fix the problem.
A hybrid setup also helped: reports appeared inline when they fit within the 100,000-character limit, with retrieval used for longer outputs. The authors say harness designers should test how each model handles its chosen output-delivery method, and examine where errors occur rather than relying only on the final score.
The result comes with boundaries. The experiments cover one scientific simulator and a set of verified geochemistry questions; the models were closed systems accessed through their APIs, and the authors did not establish which model traits explain the difference between GPT and Claude. For these PHREEQC tasks, telling the agent which report to trust worked better than merely telling it to round correctly.