Coding-Agent Benchmarks Should Match Their Users' Task Flows

AI summary

Overview

Research area: Evaluation methodology for AI coding agents — specifically, how benchmarks are constructed and whether their interaction patterns reflect real user behavior. The paper sits at the intersection of software engineering, LLM agent evaluation, and human–AI interaction analysis.

Technical level: Intermediate. The qualitative argument is accessible to beginners, but the alignment metric (TFAS), the distributional formalism, and the experimental protocol assume familiarity with benchmark design and agent harnesses.

Scope: An opinion/position paper (workshop track, arXiv:2610.09633v2, category cs.LG) arguing that coding-agent benchmarks must be calibrated to a named, measured target use case, illustrated with an analysis of 4,782 real IDE sessions and a transformation approach applied to 700 SWE-Bench Pro tasks.

What This Paper Is About

Existing coding-agent benchmarks such as SWE-bench and SWE-Bench Pro present each task as a single, polished issue description and score one final patch. The authors show that this interaction pattern looks nothing like how real engineers actually talk to coding agents, who ask questions, plan, review, and switch between task types mid-session. Their goal is to argue that benchmarks should explicitly name their target user population, measure that population's interaction distribution, and reshape existing verified tasks to match it — without discarding the repositories and tests those benchmarks already provide.

Key Contributions

1. A measured "Task Flow" target from production data. The authors analyze 4,782 real agent sessions in JetBrains IDEs (roughly 14,000 user messages), classifying each message into one of 16 intents, and define a corpus's Task Flow as a triple of distributions: session lengths, task types over messages, and directed transitions between adjacent messages.
2. SWE-TaskFlow, a transformation approach. Rather than building new benchmarks, it rewrites the interaction layer of existing verified tasks while leaving repository state, requested change, environment, and final tests fixed. It does this through prompt splitting (one issue becomes K=3 ordered turns) and verifiable repository QA turns (zero to three per task).
3. TFAS, a quantitative alignment score. The TaskFlow Alignment Score compares a benchmark's length, type, and transition distributions against a target using base-2 Jensen–Shannon divergence, producing a score in [0, 100], with 100 meaning an exact match on all three distributions. It is used to select among generated trajectory candidates.
4. A per-turn evaluation design and a pilot. Instead of one terminal binary score, a transformed trajectory yields a test-suite verdict after each issue turn and a judged answer per QA turn. A pilot on 700 SWE-Bench Pro tasks tests whether this changes measurement.

Main Findings

• Real sessions are long and mixed. Of the 4,782 Production Sessions, 50.9% contain a single user message and 33% contain at least three; the 75th and 90th percentiles are three and eight messages. All subsequent analysis conditions on the three-or-more-message slice. First messages average 30.7 words versus 16–19 at later positions.
• Benchmark prompts are far narrower than real prompts. In SWE-Bench Pro, new features (54.7%) and bug fixes (39.1%) account for 93.8% of tasks. In Production Sessions the same two intents cover well under half of user messages; bug fixes lead at 25.9% and new features at 18.9%, with explain requests at 6.1% and general programming questions at 9.7%.
• Explanation is the missing intent. Requests to explain how project code works account for 6.1% of messages in long production sessions, remain common at every position in the session, and are essentially absent from issue-derived benchmarks — motivating the QA augmentation.
• Users switch task types mid-session. A session opening with a feature request often continues with a question about existing behavior, a planning step, and a review request. Issue-derived benchmarks ignore this entirely by delivering one detailed issue and scoring one final patch.
• No public corpus is a universal ground truth. The same classification pipeline applied to long-session samples (1,000 classified sessions each) from SpecStory, DataClaw, and SWE-chat produced visibly different Task Flows from each other and from Production Sessions. SpecStory sits closest to Production Sessions but has a larger planning share; DataClaw is dominated by execution and short operational turns; SWE-chat is dominated by execution and planning.
• Splitting roughly doubles cost without a stable resolve-rate change. With mini-swe-agent as a common scaffold across four models, splitting tasks into sequential steps approximately doubled agent cost; resolve-rate shifts were mixed across models and largely comparable to run-to-run variation, so the runs establish no success ordering.
• Calibrated QA turns substantially raise alignment. Tuning the number and placement of QA turns maximized TFAS against the long-session production target, raising it from 59.6 for pure Split to 77.8. Fixed placements (QA-first, QA-last, QA-random) already reach 75.6–77.1; optimization adds mainly transition alignment. Component scores for the final selection were S_L = 84.0, S_T = 81.5, S_R = 68.8.
• Single-turn prompts collapse the score. Original single-turn prompts score S_L = 0 against the three-message target because their length bin has zero target mass, which collapses the geometric mean to zero.
• Question placement does not appear to change difficulty. In a 64-instance ablation with 114 Terra-generated questions, resolve-rate differences between QA-first, QA-last, and QA-random showed no consistent ordering across models and were comparable to run-to-run variation. Answer accuracy shifted by a few points with no consistent direction (78.1–86.8% for GPT-5.4 Mini, 86.6–93.8% for GPT-5.6 Luna, 93.8–98.2% for GPT-5.6 Terra). This is the license TFAS optimization relies on.
• QA answers provide a second verified signal. Answer accuracy on the selected benchmark spans 69.8% to 83.0% across the four models, computed on the 255 issues carrying questions rather than all 700.
• Classifier reliability is moderate. Three open-weight models (Qwen 3.6-35B, Gemma 4-31B, GPT-OSS 20B) agreed exactly on task type a mean of 76.1% of the time over 100 sessions (pairwise 79.2%, 73.7%, 75.5%). The authors note this measures inter-model robustness, not accuracy, since no human-labeled test set exists.
• Splitting is conservative on eligibility. 700 of the 731 public SWE-Bench Pro tasks survived the splitting and evaluation-context checks; issues that fail the contract stay single-turn.

Methodology in Plain English

The authors start from the observation that benchmark realism is usually asserted rather than measured. To measure it, they take server-side records of real developer–agent conversations inside JetBrains IDEs, strip everything except the user's side, and have an open-weight LLM label each message with one of 16 intents (bug fix, new feature, explain, planning, code review, execute, and so on). Labels are assigned independently per message, so a session becomes an ordered sequence of intents. From that sequence they compute three things: how long sessions are, what fraction of messages fall in each intent, and which intent tends to follow which — the "Task Flow." The identical pipeline is applied to benchmarks and to three public datasets, so all comparisons rest on the same footing.

Next they build a bridge from existing verified benchmarks to that measured target. The repository, the requested change, and the tests stay untouched; only the conversation is regenerated. First, an LLM splits a single benchmark issue into three ordered, self-contained user turns, under a strict contract: preserve every original requirement, introduce none, keep technical artifacts verbatim, and make each turn doable in the state left by earlier turns. Second, it generates up to three questions about how the repository behaves, mined from a trajectory that solved the original issue. Each question must be answerable from the repository, remain valid before and after the patch, not leak the solution, and pass an executable "proof script" run in a clean copy of the repo at creation time.

Because different placements and counts of QA turns produce different Task Flows, they need a way to say which candidate is "closest" to the target. That is TFAS: for session lengths, intent types, and intent-to-intent transitions, compute the Jensen–Shannon divergence between the target distribution and the benchmark's distribution, convert each to a similarity, and take the geometric mean scaled to 0–100. Three labels (greeting, meta-question, other) are excluded as non-instantiable in a repository, and transitions are recomputed after those messages are removed. A randomized local search then picks, per task, how many questions to keep and where to place them, maximizing TFAS.

Finally they run a pilot: models driven by mini-swe-agent inside isolated task containers receive the full ordered sequence of prompts, can edit and test the repository, and are scored by the original benchmark's test script and by a model-judge comparing QA answers to hidden references. A "Concat" control delivers the same rewritten text as one message, separating the effect of rewriting from the effect of sequential disclosure.

Why This Matters

Impact on research. The paper reframes benchmark realism as a measurement problem rather than an assumption. If Task Flows differ measurably across products and populations, then claims that a benchmark is "realistic" are under-specified unless the target population, session filter, and measurement date are stated. The TFAS formulation gives benchmark builders an optimizable and auditable target, and the per-turn scoring design turns one task into several verified signals, which the authors connect to finer-grained diagnosis and ranking-based training.

Real-world applications.

• Evaluating coding assistants before shipping: a team can calibrate an internal benchmark to its own telemetry rather than borrowing a public leaderboard's interaction pattern.
• Comparing assistants across deployment surfaces (IDE plugin versus chat versus terminal), where the same underlying model may face very different intent mixes.
• Building training data and reward signals from trajectories that contain graded, per-turn verified outcomes instead of one final pass/fail.
• Auditing vendor claims: the TFAS components reveal whether a reported gain came from better code generation or from an easier interaction protocol.

Industry relevance. The data comes from JetBrains IDEs, the paper is authored by JetBrains Research, and the generation code and dataset are released publicly (GitHub under Apache-2.0 and a Hugging Face dataset with per-condition splits, content hashes, and provenance manifests for exact replay). The central practical warning is that the interaction protocol is a real evaluation variable: splitting approximately doubled agent cost in the pilot, meaning cost and effort comparisons across benchmarks with different interaction layers are not apples-to-apples.

Future Directions

1. Augmentations for the remaining intents. SWE-TaskFlow currently covers prompt splitting for implementation intents and repository QA for explain. General questions, planning, code review, refactoring, execution, and debugging without a fix still need their own augmentation — and automatic verification of a planning or code-review turn remains an open problem.
2. Combining static and reactive regimes. Fixing all user messages before inference enables exact replay and paired comparisons, but cannot measure adaptation to feedback. Reactive designs (withheld requirements, simulated users, injected edits, help-seeking) measure exactly that at the cost of simulator assumptions and trajectory variance; a full suite could pair a calibrated static core with targeted reactive slices.
3. Establishing ground-truth labels. The intent classifier has no human-labeled accuracy set; the reported 76.1% is inter-model agreement. A human-annotated evaluation set would let the field distinguish taxonomy ambiguity from measurement error.
4. Statistical rigor and target drift. Each pilot condition ran three times with min–max ranges rather than confidence intervals, and no comparison carries a significance test, so resolve-rate readings are directional only. The authors also note that Task Flows drift as products and models change, so targets need measurement dates and periodic re-measurement.

Target Audience

Benchmark builders and evaluation leads are the primary audience, especially those maintaining SWE-bench-style suites who want to add multi-turn interaction without rebuilding repositories and tests. Researchers studying agent evaluation methodology, LLM-as-judge scoring, and human–AI interaction will find the Task Flow formalism and TFAS reusable. Product teams at IDE and developer-tool companies can use the calibration procedure to construct internal benchmarks that reflect their own telemetry. Readers looking for strong empirical claims about which interaction protocol produces better resolve rates will not find them here — the authors explicitly report only directional, non-significant pilot results.

Authors’ abstract

The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In a pilot on 700 SWE-Bench Pro tasks, solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation.