RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

AI summary

Overview

• Research area: Evaluation of LLM-based coding agents (software engineering agents) on repository-level tasks, specifically benchmarking how the communication of a task—not just its difficulty—affects agent performance.
• Technical level: Intermediate. The paper is readable without deep systems knowledge, but familiarity with SWE-bench-style evaluation, resolution rates, and ablation studies helps.
• Scope: The paper characterizes how real developer requests differ from SWE-bench problem statements, builds a data-grounded benchmark and configurable framework (RealSWE) of 381 multi-variant task families, and evaluates seven LLMs to isolate the effects of information content and linguistic style.

What This Paper Is About

Coding agents are mostly measured on SWE-bench-family benchmarks, whose tasks are built from curated GitHub issues that are long, structured, and information-rich. Real user requests are typically short, casual, and sparse. The paper quantifies this mismatch using real prompts from SWE-chat against problem statements from SWE-bench Verified and Pro, then introduces RealSWE—381 task families that hold the underlying task and gold patch fixed while varying only the information composition and linguistic style of the request—to measure how those changes affect agent performance.

Key Contributions

1. Empirical characterization of real SWE requests. A six-category information taxonomy and four linguistic dimensions are applied to 718 user-authored SWE-chat prompts and 1,229 SWE-bench Verified and Pro problem statements, quantifying how real requests differ from benchmark problems in both information composition and style.
2. A data-grounded benchmark and configurable framework. Guided by that analysis, the authors construct 381 multi-variant task families (192 bug fixes, 189 feature requests) and release them as RealSWE-bench, a fixed benchmark matching the empirical distribution of real user inputs, and RealSWE-framework, which exposes the full variant suite for custom configurations and controlled ablations.
3. Controlled evaluation and actionable findings. Seven LLMs are evaluated to quantify the benchmark-reality gap, identify Desired Behavior and Motivation as key signals, and distill guidance for users of coding agents.

Main Findings

• Real requests are information-sparse. Requests containing only a problem statement ([P]) alone or with limited additional context ([PA]) account for 88% of real prompts (85.5% of bug-fix and 91.1% of feature requests), compared with just 7% of benchmark problems (8.0% for bug fixes, 3.9% for feature requests).

• Real requests are also linguistically different. 87% of user prompts use a casual tone and 51% contain imperative sentences, whereas 94% of benchmark problems use a formal tone and 89% rely on declarative sentences. In the detailed distributions, 84.8% of SWE-bench Verified and 100% of SWE-bench Pro prompts are formal. Certainty and perspective show no consistent separation between the two sources.

• Realistic inputs lower resolution rates. All seven evaluated models resolve fewer tasks under RealSWE-bench. Resolution drops by 6.4 percentage points (pp) on average, with absolute drops ranging from 4.0 pp for MiniMax M3 (34.1% to 30.1%) to 8.0 pp for DeepSeek V4 Pro (53.9% to 45.9%). Relative declines fall within 10.3–16.2%, with a mean of 13.6%.

• Task type matters. Bug-fix resolution decreases by 9.1 pp on average, whereas feature-request resolution decreases by only 3.7 pp, suggesting underspecified requests are especially challenging for bug repair.

• Rankings change. MiMo V2.5 Pro moves from fourth place on original inputs to second place under RealSWE-bench, overtaking Qwen3.7 Plus and DeepSeek V4 Flash; the shift in the MiMo–Qwen gap is significant (+3.7 pp, 95% CI [+0.7, +7.3]). MiMo V2.5 Pro is also roughly 2.5× cheaper per task than Qwen3.7 Plus (6.5 versus 16.1 cents).

• The performance distribution reshapes. Among the four models clustered near the top (Qwen3.7 Plus, DeepSeek V4 Flash, MiMo V2.5 Pro, MiMo V2.5), the performance range nearly doubles from 1.7 to 3.3 pp, while the strongest-to-weakest range narrows from 19.8 to 15.8 pp.

• Realistic requests cost slightly more. Six of the seven models incur higher cost under RealSWE-bench (6.2% on average) and take more steps (1.8% on average). Claude Haiku 4.5 is the only exception, showing marginal decreases in both.

• Linguistic style barely matters. Comparing original problem statements with all-field rephrased variants, style changes range from −1.6 to +0.9 pp for bug fixes (average 0.0 pp) and from −3.9 to +0.7 pp for feature requests (average −1.8 pp). None of the eight style contrasts is significant (Holm-adjusted p ≥ .35).

• Desired Behavior [D] is the dominant bug-fix signal. Removing [D] from [PD] lowers resolution by 7.1–8.9 pp (8.0 pp on average; significant for all four models, Holm-adjusted p < .01). This is more than four times the combined decrease from removing [A], [E], and [R].

• Reproduction Steps, Environment Information, and Additional Information add little. Progressively removing [A], [E], and [R] from bug fixes changes resolution rate by only 1.8 pp on average; no removal step yields a significant decrease (Holm-adjusted p ≥ .44), and the only significant change is an improvement (+6.2 pp for MiMo V2.5, p = .017).

• Motivation [M] drives feature requests, but less uniformly. Comparing [PM] with [P], removing Motivation reduces resolution for all four models by 3.4 pp on average, and [M] is the only feature-request field whose average effect is distinguishable from zero (95% CI [+0.2, +8.2] pp). DeepSeek V4 Flash loses 7.1 pp (Holm-adjusted p = .001), whereas the other models lose 1.8–2.8 pp (not significant).

• What users provide is not what agents need. Real prompts largely omit high-value fields: only 5% of real user prompts state Desired Behavior or Motivation. [PD] outperforms [PA] across all four models by 3.8–6.1 pp on bug-fix tasks, and [PM] outperforms [PA] by 1.4–3.5 pp on feature-request tasks.

• Description length aligns with reality. RealSWE-bench has an average task-description length of 1,417 characters, closely matching the 1,427 average in SWE-chat and well below the 1,672–2,776 characters of conventional SWE benchmarks (SWE-bench Verified, Multilingual, Pro, and DeepSWE).

• Selection is not biased toward easy tasks. The selected 381 tasks are harder than the excluded 848 (53.9% versus 63.0% resolution rate for DeepSeek V4 Pro on original statements), larger (134 lines versus 94), and spread more evenly (13.9 versus 11.0 effective repositories).

Methodology in Plain English

The authors started from two observation sets: 718 real first-turn developer prompts from SWE-chat (filtered down from more than 6,000 developer-agent sessions) and 1,229 problem statements from SWE-bench Verified and Pro. They labeled everything using a taxonomy that splits a request into a Problem Statement, Desired Behavior, Reproduction Steps, Environment Information, Motivation, and Additional Information, plus four style dimensions (Formality, Sentence type, Certainty, Perspective), using an LLM-assisted pipeline with GPT-5.4.

Guided by the measured distributions, they transformed benchmark problems into multi-variant task families with a three-step pipeline: classify each task as a bug fix or feature request, decompose the original problem statement sentence by sentence into taxonomy fields without rewriting the text, and then rephrase it into the casual style observed in SWE-chat while preserving technical literals such as code, error messages, and file paths. Only tasks containing every required field were eligible, leaving 403 candidates.

Two annotators independently checked 100 sampled instances per LLM-driven stage to establish human ground truth, and GPT-5.6 Terra audited all candidates with the same three-point rubrics. Classification accuracy was 0.95; judge agreement with humans was 0.97 accuracy / 0.83 macro-F1 for decomposition and 0.99 accuracy / 0.75 macro-F1 for rephrasing. Excluding 22 candidates that scored lowest on any critical criterion left 381 task families. The fixed RealSWE-bench samples one variant per family in proportions matching SWE-chat ([P] 74% and [PA] 26% for bug fixes; [P] 72% and [PA] 28% for feature requests), yielding 142 [P] and 50 [PA] bug-fix tasks plus 136 [P] and 53 [PA] feature-request tasks.

Evaluation used seven models (DeepSeek V4 Pro and Flash, MiMo V2.5 Pro and MiMo V2.5, Claude Haiku 4.5, Qwen3.7 Plus, MiniMax M3) with reasoning enabled, all run through the same mini-SWE-agent v2 Bash-only scaffold in each source benchmark's execution container, with a maximum of 100 agent steps and no cost limit. Each model-condition pair ran three times, with means and standard deviations reported. Ablations used cumulative field-removal paths ([PDREA]→[PDRE]→[PDR]→[PD]→[P] for bug fixes; [PMA]→[PM]→[P] for feature requests), with [PA] added as a comparison condition.

Why This Matters

Impact on research. The paper shows that benchmark scores are partly a function of how a task is communicated, not only how hard it is. Because RealSWE varies information composition and style independently while holding the repository task and gold patch fixed, it enables per-field attribution that earlier realistic benchmarks—which jointly alter multiple properties—could not provide. It also reframes prompt length as an imprecise proxy for the benchmark-reality gap: which information a request contains matters more than how much.

Real-world applications:

• Coding-agent interfaces can prompt users with targeted clarification questions, or infer missing Desired Behavior or Motivation before implementation, to make sparse requests more resolvable.
• Model and agent selection should account for ranking instability: a leaderboard choice can steer users toward a model that is roughly 2.5× more expensive per task without being measurably better under realistic requests.
• Budget and latency planning can anticipate that realistic requests raise cost (6.2% on average) and steps (1.8% on average) for most models while still losing accuracy.
• Developer onboarding and bug-report templates can encourage users to state the intended behavior for bug fixes and the motivation for feature requests, the two fields with measurable value.

Industry relevance. Benchmark leaderboards are used as de facto standards for comparing coding models. The finding that realistic inputs reduce resolution rates by 6.4 pp on average,

Authors’ abstract

Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues: long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-chat and problem statements from SWE-bench Verified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduce RealSWE, 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and gold patch while differing only in information composition and linguistic style. Evaluating seven contemporary LLMs with RealSWE, we find that i) realistic inputs reduce resolution rates by 6.4 pp on average and can change model rankings. Controlled analysis further shows that ii) including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; iii) linguistic style has only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation, which most real prompts omit, substantially improves the LLM's software engineering performance.