Transactional and Customer-Service Agents

Conversation, policy, and irreversible effect meet at four points: User interaction, Policy engine, Transactional tools, and State verification. State verification and Recourse should belong to different owners, each tested on its own.

The first of those points is the one usually drawn as a box and never measured. τ²-bench measures it. The 2025 benchmark extends τ-bench with a Telecom domain in which the simulated user also holds tools and acts on the same shared state as the agent. The paper models that as a Dec-POMDP and calls it dual control. The domain combines 15 atomic subtask groups across 3 user intents into 2,285 programmatically generated tasks, and evaluates on 114 of them. gpt-4.1's pass^1 falls from 74% in retail and 56% in airline to 34% in telecom. Moving from a no-user setting to dual control costs gpt-4.1 18 percentage points of pass^1, and o4-mini 25. The authors put it plainly: “In particular, our experiments show significant performance drops when agents shift from no-user to dual-control, highlighting the challenges of guiding users.”

Same tools, same policies, same domains. One thing was added: a customer who also has to act. A quarter of the agent's success went with it. User interaction is a control point with a number attached, not the soft box on the left of the diagram.

Transactional agents operate in conversations where users reveal goals incrementally while the agent consults policy and changes records. Being correct here means five things at once: following policy, resolving the right identity, causing the intended tool effects, saying the right thing to the customer, and leaving the right final database state.

The system must distinguish explanation, recommendation, staging, execution, and exception approval. Empathy in language cannot substitute for legitimate authority.

Payments infrastructure solved the duplicate-action problem before agents existed. Stripe's API accepts an idempotency key on any POST. It saves the status code and body of the first request made for that key, so that “Subsequent requests with the same key return the same result”. Keys can be removed from the system automatically once they are at least 24 hours old. Reuse a key after the original is pruned and you get a new request, not the old answer.

The rule is not one vendor's house style. An IETF draft for an Idempotency-Key header says the same. On a repeat, “The resource SHOULD respond with the result of the previously completed operation, success or an error”. Expiry is left to the service: “The resource SHOULD define such expiration policy and publish it in the documentation”.

The retries are not hypothetical. The transport underneath promises them. Amazon SQS standard queues ensure at-least-once delivery, so more than one copy of a message might be delivered and messages might arrive out of order. Google Cloud says the same of Pub/Sub, a second provider: “By default, Pub/Sub offers at-least-once delivery with no ordering guarantees on all subscription types”. It goes further: “Pub/Sub might redeliver a message even after an acknowledgment request for the message returns successfully”. Retry a refund without a key and the duplicate is already on its way. It is waiting for the second copy to arrive.

Authority is what separates an Informational assistant from a Bounded service agent and from an Exception-making agent. Once an agent has authority to change records, success is measured on the records themselves. Re-run a case with nothing changed, and the database should finish in the same valid state it reached before. The matching failure is the model following the customer's story more readily than the organization's actual rule.

The middle mode has a dated, numbered instance. In its first month live, Klarna's OpenAI-powered assistant handled 2.3 million conversations — two-thirds of Klarna's customer-service chats. That was the equivalent work of 700 full-time agents, across 23 markets and 35+ languages. Errand resolution fell from 11 minutes to under 2 minutes, repeat inquiries dropped 25%, and Klarna estimated a $40 million USD profit improvement for 2024. The press release of 27 February 2024 also kept the escape hatch in the design: “Additionally, customers can still choose to interact with live agents if they’d prefer.”

The sequel is the part worth teaching. By 9 May 2025 Klarna was hiring human agents again. CEO Sebastian Siemiatkowski said cost had been too dominant a criterion, and the result was lower quality. A bounded service agent is a scope decision, not a headcount decision. Treat the human path as a cost line rather than as part of the contract, and the walk-back arrives as a business event with a date on it.

In April 2025 Cursor's AI email-support agent told a user that being logged out when switching machines was expected. Cursor, it said, was limited to one device per subscription as a core security feature. The agent was replying under the human-sounding name “Sam”. No such policy existed. The reply was fluent, confident, and internally consistent. It invented the rule it was enforcing, and users publicly cancelled their subscriptions over it.

The real cause was a race condition on very slow internet connections. Cursor cofounder Michael Truell said so on Hacker News on 16 April 2025, and added that the user had been fully refunded. Nothing in the conversation flagged the problem. The agent had authority over what customers believed the policy was. No authorized reviewer was in the loop when that authority was exercised.

The strongest signal is repeatability: handle one customer case twice, under a single policy, and the records should be left in one valid, identical condition. Three other signals need checking on their own. The agent should follow policy while identifying the cases that require an authorized exception. Conversation quality should not mask a tool or policy failure. Customers should be able to correct records, challenge outcomes, and reach a responsible human. Test those three under repeated or adversarial cases, not only on a first clean run.

The last signal has a regulator's numbers behind it. Every one of the top 10 largest US commercial banks had deployed a customer-service chatbot. Over 98 million users — approximately 37% of the US population — engaged with a bank's chatbot in 2022, and the projection is 110.9 million users by 2026. The figures come from the Consumer Financial Protection Bureau's June 2023 issue spotlight "Chatbots in consumer finance". The report names the failure mode: "doom loops", in which a customer is looped through unhelpful jargon with no offramp to a human representative. Its finding is about what disappears, not about tone: “When a chatbot is backed by unreliable technology, inaccurate data, or is little more than a gateway into the company’s public policies or FAQs, customers may be left without recourse.”

A service flow that already changes records is the right one to constrain by policy end to end. Verify identity and scope, clarify the request, evaluate policy against versioned rules, execute transactionally with idempotency and postcondition checks, and close with recourse. Then push a case where the customer's account points one way and the written rule points the other. Watch whether the agent follows the story instead of the rule. Save the runs. They are what you show later to prove that the same case, judged by the same rule, still lands in one valid database state.

"Versioned rules" sounds abstract until you encode a real one. Airline refunds in the US are automatic and time-bound, by federal rule. The Department of Transportation's 2024 final rule "Refunds and Other Consumer Protections" states it in the summary: “These automatic refunds must be provided promptly, i.e., within 7 business days for credit card payments and within 20 calendar days for other forms of payment.” The rule took effect on 25 June 2024.

Eligibility is a closed list, not a judgement call. The rule defines "significant change of flight itinerary" through seven triggers. Departure three hours or more earlier for domestic flights, six hours or more for international. Arrival three hours or more later domestic, six hours or more international. A different origination or destination airport. More connection points. A downgrade to a lower class of service. And, for a consumer with a disability, two further triggers: being routed through different connecting airports, or being placed on substitute aircraft missing an accessibility feature the passenger needs.

Those thresholds are what the policy engine holds. A persuasive customer can move a two-hour delay into none of them. An agent that grants the refund anyway has not been generous. It has amended a federal rule in a chat window.

Customer-service agent evals should compare final database state and policy compliance across repeated conversations, not only rate the final message. That is not a preference. A published benchmark already works this way. τ-bench, from 2024, states its method in the abstract: “We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state.”

The benchmark also supplies the reason for judging several passes rather than one. τ-bench introduces pass^k over repeated trials, and the gap between the two views of the same system is large. State-of-the-art function-calling agents such as gpt-4o succeed on fewer than 50% of tasks, and score pass^8 below 25% in the retail domain. An agent that resolves a case once in four attempts can be demonstrated as working and still be wrong three times out of four in production.

So build the eval around the case the agent is most likely to get wrong: the one where a persuasive account pulls against what the organization actually permits. Then grade it on what the records show after several passes through that case, not on how the conversation read.