The Pulse
IBM Puts GPT-4.1 First on τ²-Bench Retail With Learned Memory
IBM Research says its ALTK-Evolve system lifted a GPT-4.1 customer-service agent to a 67.5% Pass⁴ score on τ²-bench retail, topping the leaderboard snapshot it compared against. The system adds guidelines learned from past agent runs to the
AI.info Team · Written with AI · fact-checked ·
“The only change we make is a block of learned guidelines appended to the system prompt — everything else (policy, tools, scaffold) is stock τ²-bench.”
Ashwath Vaithinathan Aravindan, Vatche Isahagian, Vinod Muthusamy, Jayaram Radhakrishnan, Evelyn Duesterwald, Gaodan Fang, Punleuk Oum, G Thomas, Michał Ulewicz, Merve Unuvar and Ayhan Sebin, IBM Research
IBM Research says its ALTK-Evolve memory system helped an agent using GPT-4.1 reach a 67.5% Pass⁴ score on τ²-bench retail, ahead of every retail entry in the leaderboard snapshot the team used. The result, published October 20, comes from feeding the agent short guidelines distilled from its previous task attempts—not from updating the model’s weights.
The score needs context: Pass⁴ counts a task only when the agent succeeds in all four attempts. IBM reports 67.5% for the full benchmark base split, which combines training and test tasks, and compares it with a leaderboard snapshot dated October 12. On a separate set of 40 held-out retail tasks, the same GPT-4.1 setup rose from 57.5% to 67.5% Pass⁴ when IBM added its full set of learned guidelines.
Rules learned from earlier customer-service runs
ALTK-Evolve analyzes the agent’s attempts on training tasks and distills recurring successes and failures into behavioral rules. The examples include authenticating customers before order actions, getting explicit approval before changing a database, and explaining return refunds with an itemized breakdown. IBM used GPT-4.1 both to produce the guidelines and to run the retail agent, then appended the rules to the standard system prompt.
The benchmark tests agents on customer-service conversations in retail and airline domains. A simulated customer has a goal, the agent can use tools such as order lookup or refunds, and a written policy constrains what it may do. A task passes only if the final database state matches the expected result, so a fluent exchange alone does not count as success.
IBM says its test used τ²-bench’s standard ReAct agent scaffold, with GPT-5.5 simulating the customer and four attempts per task. The team generated guidelines from training tasks and applied them unchanged to held-out tasks. It describes the method as requiring no human annotation and no test-set leakage.
Retail reliability rose, but the full prompt cost more
On the 40 held-out retail tasks, the full “lossless” configuration retained 292 guidelines and raised mean accuracy from 78.1% to 82.5%. Pass⁴ increased by 10 percentage points, from 57.5% to 67.5%. A smaller configuration kept only 31 guidelines with support from at least three distinct training tasks; it reached 60% Pass⁴, a 2.5-point gain over the baseline.
The gains came with a higher reported language-model bill. IBM’s full-base-split cost table puts retail cost per conversation at $0.084 for the baseline and $0.139 with the 292-rule configuration, a 65% increase. In matched test runs, however, the team says the extra guidance did not lengthen conversations: retail conversations solved under both settings averaged 10.59 assistant turns at baseline and 10.16 with the full guideline set.
The leaderboard claim spans two models and two domains
IBM’s retail comparison uses GPT-4.1, which the authors describe as an older, non-reasoning model. They report its 67.5% Pass⁴ result as higher than RAFT-30B-A3B’s 61.4%; IBM identifies RAFT as fine-tuned on synthetic τ²-bench retail data. The team says its score led the retail entries in the October 12 snapshot, though the comparison is IBM’s report of that snapshot rather than a claim about later leaderboard changes.
Airline results show that the method’s best configuration depends on the task. On held-out airline tasks, a compact set of 14 guidelines reached 45% Pass⁴ with GPT-4.1, while the larger set reached 40%, against a 35% baseline. In a separate leaderboard comparison, IBM reports 44% for the GPT-4.1 agent and 82% for an agent using GPT-5.5; the latter topped the airline entries in the snapshot.
Memory improves the prompt, not the underlying model
The study’s claim is narrower than saying memory makes an agent universally more capable: it shows improvements on two policy-driven customer-service benchmark domains under IBM’s specified test setup. Results vary with the model, the amount of guidance and the metric; airline reliability, for example, favored the smaller GPT-4.1 rule set on held-out tasks.
For enterprise teams, the reported trade is concrete. ALTK-Evolve adds a reusable playbook without fine-tuning or manual labeling, but the longer retail prompt increases the measured per-conversation cost. The GPT-4.1 retail result IBM highlights is 67.5% Pass⁴ on its full base-split comparison, against 61.4% for RAFT-30B-A3B.