TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents

AI summary

Overview

Research area: Natural Language Processing — evaluation of tool-using conversational agents (LLM agents) in a regulated customer-service setting, specifically Indian retail banking.

Technical level: Intermediate. The paper is readable without deep technical background, but it assumes familiarity with agent benchmarks, tool calling, and pass/fail scoring of multi-turn sessions.

Scope (one sentence): The paper introduces TauIndianBankBench, a 1,000-task benchmark of Indian retail-banking servicing conversations that scores not only whether an assistant left the correct records but also whether it took every procedure-mandated step and made no unrequested change, and it uses this to evaluate six open-weight models.

What This Paper Is About

Indian banks currently serve customers through FAQ pages and scripted chatbots, and language-model assistants could respond faster and actually carry out requests such as blocking a card — but only if they follow the bank's written rules. Those rules prescribe a procedure: a set of required steps, many of which (such as obtaining a closure quote before closing a deposit) leave the database unchanged. Existing benchmarks usually judge only the records left at the end of a conversation, so an assistant that skips a required step can still be scored as successful. The paper's goal is to build a benchmark that measures both the procedure and the outcome, and to show quantitatively how much the two measures disagree.

Key Contributions

1. An Indian retail-banking environment: a synthetic bank of 944 customers (accounts, cards, deposits, loans, standing instructions, KYC status, transaction history), a catalogue of 34 tools, a written policy, a knowledge base of 24 documents, and 1,000 tasks drawn from 50 task families, released with base, lite (200 tasks), train (150), and test (50) splits.
2. An action check that complements the final-state check: it requires every call in the task's gold chain — including read-only calls that the written policy mandates — to appear in the session, and it adds a write purity condition that fails a session if the assistant makes any change the task did not request, even if that change is later reverted.
3. A combined pass criterion: a session passes only if it ends normally, makes every gold-chain call, makes no unrequested change, and leaves records matching a replay of the gold chain. No partial credit is given; making three of four required calls scores the same as doing nothing.
4. An evaluation of six open-weight models under both checks, showing that the final-state check alone accepts procedural violations for every model and that each check fails sessions the other passes.

Main Findings

• Final-state scoring inflates pass rates by 6.5–12.0 points. For every model, 6.5–12.0% of sessions are judged successful when only the final records are checked, although the assistant skipped a required step or made an unrequested change.

• Reference pass rates (both checks, 1,000-task base split): Qwen3.8-27B 0.849; Gemma 4 31B 0.776; Gemma 4 26B-A4B 0.714; Gemma 4 12B 0.699; Gemma 4 E4B 0.607; Gemma 4 E2B 0.479. By category, the same six models score 0.921 / 0.868 / 0.802 / 0.732 / 0.662 / 0.455 on sequencing, 0.774 / 0.718 / 0.662 / 0.639 / 0.500 / 0.426 on edge cases, and 0.757 / 0.559 / 0.500 / 0.533 / 0.480 / 0.388 on rare tools.

• Model size helps within the dense Gemma family. Each larger dense Gemma model passes more tasks than the next smaller one (p < 0.005 for each pair, paired sign test), and the ranking holds within each of the three graded categories.

• Final-state-only pass rates are higher for every model: 0.920 (Qwen3.8-27B), 0.896 (31B), 0.821 (26B-A4B), 0.764 (12B), 0.725 (E4B), 0.598 (E2B).

• The two checks fail different sessions, and neither substitutes for the other. For the three strongest models (Qwen3.8-27B, Gemma 4 31B, 26B-A4B), the number of sessions with correct records but a failed action check (b = 71, 120, 107) exceeds the number that pass the action check but end with incorrect records (c = 13, 25, 37), all p < 10⁻⁸. For the weakest model, E2B, the reverse holds (b = 118, c = 180, p < 10⁻³). For the 12B (b = 65, c = 88, p = 0.075) and E4B (b = 118, c = 109, p = 0.596) the difference is not significant.

• Rare tools are the hardest category for every model, with Gemma models clustered narrowly between 0.388 and 0.559, while Qwen3.8-27B exceeds the 31B by 0.198 in that category — its largest margin over that model in any category.

• Procedure length is not uniformly harder. Four-call tasks are the hardest for every model except E2B. The gap between the strongest and weakest model is widest on five-call tasks (0.614, against 0.024 on one-call tasks). E4B falls from 0.810 on short sequencing tasks to 0.445 on long ones, whereas none of the three strongest models changes by more than 0.086 on sequencing or edge-case tasks.

• The control category shows no size trend. Across the four dense Gemma models, control pass rates stay between 0.804 and 0.929 without a trend in size, and the highest rate belongs to the 12B — evidence that the graded categories reward procedural competence rather than general capability.

• Trivial agents score zero. Four scripted agents were evaluated and none passes any task: one only converses, one reads records but makes no required call, one claims completion in prose, and one makes every available change (failing write purity).

• Scorer reliability checks. An independent re-implementation of the scorer agreed with the released scorer on all 600 recorded sessions and all 705 paired tool calls; the packaged scorer reproduced every score on 5,000 recorded sessions; the harness's judge model agreed on 87.5% of sessions and gave less credit in every disagreement.

• Failure causes differ by model strength. A missing required write is the most frequent action-check failure for four of the six models, and a missing required read for the two smallest (E4B and E2B). Unrequested writes are the least frequent cause for every model except Qwen3.8-27B.

• Violations concentrate in loans. The violation rate is highest in loan tasks, because no loan tool is a write and the final-state check therefore cannot observe an omitted loan step.

• Run-to-run variation. A second run of the five Gemma models changes each pass rate by at most 2.7 points, although 14–26% of tasks change outcome between the two runs.

Methodology in Plain English

The authors build a fake but internally consistent bank and let a model under test talk to a model playing the customer, while both act on a shared database. Each session starts from a fresh seeded copy of a bank with 944 customers, so the clock and generated identifiers are fixed and any session can be replayed exactly.

Each task comes with a gold chain — the reference sequence of tool calls that resolves the request, plus the arguments those calls must match. Gold chains average 3.54 calls and range from one to six. Crucially, the gold chain includes the informational reads the bank's written policy requires (for example, fetching a closure quote before closing a deposit), not just the changes.

Scoring happens twice on the same recorded session, deterministically, with no judge model in the released score. The final-state check, inherited from the τ-bench and τ²-bench family, replays the gold chain on the seeded starting records and compares the resulting database with the session's final database, after normalising away run-dependent fields such as generated references and free-text notes. The action check searches the transcript for each gold-chain call in any order, with a defined rule for which arguments must match. Its write purity condition fails the session if any write does not correspond to a gold-chain write.

Tasks were drafted by a language model against the live environment and admitted only if they passed four filters: no first message appears in more than 40% of tasks within its family and at least eight distinct first messages per family; graded tasks must have a gold chain of at least three calls; no task appears in both train and test; and the replayed gold chain must pass both checks. Call order is fixed by task design rather than by the scorer — some calls need values returned by earlier calls, and the simulated customer raises numbered steps one at a time and asks the assistant to return to a step it skipped.

Evaluation runs one attempt per task on the 1,000-task base split, with MiniMax-M2.7 playing the customer at temperature zero for every model. Sessions terminate when the customer says the request is resolved, when the assistant ends the session or transfers to a human, or with a score of 0 at 200 steps, 10 failed tool calls, or an error. The 95% Wilson confidence interval of a pass rate is approximately ±3 points over the full split and at most ±8 points within a single category.

Why This Matters

Impact on research. The paper makes a concrete, quantified case that terminal-state evaluation — the dominant paradigm in agent benchmarks, including τ-bench — systematically overstates success in domains where policy-mandated steps are read-only. It also supplies a reusable design pattern (gold chains that include required reads, plus write purity) and a validity argument that reviewers increasingly expect: trivial-agent baselines that score zero, task splits that separate adaptation from reporting, and an audit against an independent scorer re-implementation.

Real-world applications:

• Bank assistant pre-deployment testing. A bank could measure whether a candidate assistant actually fetches the closure quote, minimum-balance rule, or card status before acting, rather than only checking that the account ended in the right state.
• Compliance and audit trails for regulated procedures. KYC renewal, tax deducted at source on deposit interest, and statutory eligibility rules for government savings schemes are explicit regulatory steps; the action check turns them into testable requirements.
• Before/after measurement of domain fine-tuning. The separate 150-task train and 50-task test splits let a team tune a model on banking tasks and report on held-out tasks with the same two-check criterion.
• Regression testing of agent scaffolds and tool interfaces. Because the scorer is deterministic and re-scoring the same recorded sessions isolates the scoring rule from a new run, teams can compare agent designs on fixed transcripts.

Industry relevance. The benchmark is released by the National Payments Corporation of India and targets Indian retail banking specifically, a domain where the paper notes existing Indian and financial evaluation suites cover language coverage, domain knowledge, numerical reasoning, or financial tool use — but not customer servicing under Indian banking rules. It runs as a standard domain of the τ²-bench harness against any OpenAI-compatible endpoint, which lowers the cost of adoption for banks and vendors already using that stack. The paper is explicit that a pass does not establish deployment readiness and that further safety, privacy, and regulatory review would be required.

Future Directions

• Extending beyond English and beyond retail banking. Version 1.0.0 covers Indian retail banking in English only; other languages and domains are left to future versions.
• Measuring and controlling simulator error. The authors state they did not measure the simulated customer's own error rate, even though every score depends on it — a natural next step is calibrating or cross-checking the customer simulator.
• Replacing synthetic edge-case frequencies with real traffic distributions. All customer data are synthetic, so how often each type of edge case appears reflects the generated records rather than actual bank traffic.
• Making full evaluation cheaper. Each session is a live multi-turn conversation of up to 200 steps for both assistant and customer, and estimating pass^k multiplies that cost by k; the paper offers the 200-task lite split for development and ablations but leaves the cost problem open for commercial-API settings.
• Broadening the check design. The paper's comparison table positions its scoring against benchmarks that use partial credit (ToolSandbox milestones), judge models (TRAJECT-Bench, TRACE), or pre-generated tool responses (JourneyBench), raising the open question of how strict binary procedure scoring trades off against diagnostic granularity.

Target Audience

Researchers and engineers working on tool-using LLM agents and agentic benchmarks; evaluation and risk teams at banks, payment networks, and financial-software vendors considering LLM customer service; and anyone building domain adaptations of the τ-bench / τ²-bench harness. It is also useful to readers interested in benchmark validity methodology — gaming, trivial baselines, and the gap between process and outcome metrics — even if they work outside finance.

Authors’ abstract

Many Indian banks serve customers through FAQ pages and scripted chatbots. Assistants based on language models could respond faster and carry out requests such as blocking a card, but only if they give correct information and follow the bank's rules. These rules prescribe a procedure, a set of required steps, such as obtaining a closure quote before closing a deposit. Adapting models to this domain requires measuring how well they follow the procedure, before and after training. Benchmarks usually judge only the records left at the end of a conversation. Many required steps, however, leave the records unchanged, so an assistant that skips them can still be judged successful. We present TauIndianBankBench, an action-oriented benchmark of 1,000 Indian retail-banking tasks where an assistant helps a simulated customer. Besides the final records, it checks that the assistant took every required step and made no unrequested change. For every model, 6.5-12.0% of sessions are judged successful when only the final records are checked, although the assistant skipped a required step or made an unrequested change. Scoring both procedure and outcome shows more accurately how well language models perform in Indian retail banking.