Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages

AI summary

Overview

Research area: Software engineering evaluation for AI coding agents — specifically repository-level issue resolution benchmarks covering underrepresented programming languages.

Technical level: Intermediate to Advanced. Readers should be comfortable with the SWE-bench task formulation (problem statement, base commit, reference patch, FAIL_TO_PASS / PASS_TO_PASS tests) and with LLM agent evaluation metrics such as Pass@1 and Pass@3.

Scope: The paper introduces Longtail-SWE, a 202-task benchmark of real GitHub issue-resolution tasks across 13 programming languages poorly covered by existing repository-level benchmarks, built with a solver–judge–repairer pipeline that repairs defective task descriptions and tests instead of discarding them.

What This Paper Is About

Repository-level coding benchmarks exist mainly for Python and a handful of other ecosystems, leaving many languages with almost no rigorous evaluation data. The obvious fix — harvesting historical issues and pull requests from these languages — runs into a quality problem: many candidates have vague descriptions or tests that reject valid alternative implementations, and the usual response of filtering them out shrinks an already tiny pool. The authors instead build a pipeline where a solver attempts each task, a judge diagnoses whether failures come from the agent or from the task itself, and a repairer fixes descriptions and tests so that difficult but repairable tasks can be kept and reliably scored.

Key Contributions

1. Task recovery and refinement pipeline. A solver–judge–repairer loop that coordinates revisions to both evaluation tests and task descriptions, so scarce repository-level tasks with repairable defects are retained rather than filtered out. Revisions are append-only for descriptions and must preserve the unchanged reference patch.

2. The Longtail-SWE benchmark. 202 curated tasks from 129 repositories across 13 languages: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift — plus a companion set of easier tasks (the difficulty-selected tasks not retained in the final 202) intended to support evaluation of smaller models.

3. Empirical analysis of task recovery. A construction ablation from a common pool of 3,261 candidates showing that expanding semantic screening to include B2/B5 candidates and combining environment installation with test auditing raises strict execution-validation passes from 377 to 1,247.

4. Evaluation of nine model configurations plus a human audit of the judge. Reported Pass@1/Pass@3/All-3, per-language breakdowns, token and step usage, repeated-attempt scaling, and a 65-task human audit giving the judge 87.5% precision and 96.1% recall on identifying defective tasks.

Main Findings

• Best overall result: Opus 5.5 (high) has the highest point estimates on all three success metrics — 81.52% Pass@1, 89.11% Pass@3, and 73.76% All-3 — exceeding runner-up GPT 6.1 Sol by 3.14 percentage points on Pass@1. Even so, 10.89% of tasks remain unsolved by Opus after three attempts.

• Wide spread across configurations: Scores range from 81.52% Pass@1 (Opus 5.5) down to 16.67% (Qwen 3.6 27B thinking). The ordering is GPT 6.1 Sol 78.38, GPT 5.6 Sol 75.74, GPT 5.6 Luna 71.45, Kimi K3 63.70, DeepSeek V4 Flash 56.11, GLM-5.3-Flash 48.18, Qwen 3.8 27B 42.74.

• Resource usage varies far more than accuracy: GPT 6.1 Sol uses 1.75M mean total tokens and 27.4 mean steps, versus 38.28M tokens and 313.7 steps for GLM-5.3-Flash. GPT 5.6 Luna uses roughly three times as many tokens as Opus despite a lower Pass@1, and Kimi K3 uses more tokens and steps than Luna with a lower Pass@1 (16.84M tokens, 151.4 steps).

• Single attempts understate capability: On the final 202 tasks with ten attempts per task, Qwen 3.8 27B's Pass@k rises from 43.8% at k=1 to 73.3% at k=10, with diminishing marginal gains; 26.7% of tasks remain unsolved after ten attempts.

• Language difficulty varies about 1.5-fold: Averaging Pass@1 equally across the nine configurations, Erlang is highest at 73.5%, followed by Haskell at 72.8% and R at 69.0%; Clojure (48.1%), F# (50.2%), and Elixir (50.4%) are lowest. Per-language task counts range from 12 to 27.

• Per-language leaders differ: Opus 5.5 leads outright in four language subsets and ties in others (Erlang with GPT 6.1 Sol; Lua with both Sol configurations; Swift with GPT 5.6 Sol and Luna). GPT 6.1 Sol leads on Elixir, Haskell, and Julia; GPT 5.6 Sol leads on F# and ties Luna on Dart; Kimi K3 leads on R.

• Construction changes dramatically increase yield: On the same 1,096 label-A candidates, strict execution-validation passes rise from 377 (34.4%) with the SWE-rebench V2 install agent to 539 (49.2%) with the combined auditor. Adding B2/B5 candidates expands the pool to 2,640 and yields 1,247 passes (47.2%).

• End-to-end retention more than doubles: From the common pool of 3,261, the final benchmark holds 202 tasks (6.2%) versus 68 (2.1%) through the SWE-rebench V2 path. Counting only tasks re-annotated as A after repair, 196 versus 66 remain — a factor of 2.97 in benchmark tasks. Only 68 of the released tasks would have survived the original funnel.

• Quality of the discarded path is poor: A single GPT 5.6 Luna (xhigh) judge found at least one quality issue in 328 of 364 (90.1%) tasks passing SWE-rebench V2 image build and revalidation; 300 (82.4%) had hidden tests requiring undocumented APIs or exact outputs and 323 (88.7%) had insufficient test coverage. Only 36 (9.9%) were accepted — an upper bound on what the full three-judge protocol would pass without repair.

• Test repair alone lifts scores substantially: On 63 released tasks whose descriptions were unchanged, replaying the same three Opus 5.5 trajectories gives Pass@1 of 29.1% pre-repair versus 67.2% post-repair (+38.1 points), Pass@3 of 36.5% versus 77.8%, and All-3 of 20.6% versus 54.0%. On ten Qwen 3.8 27B attempts over the same 63 tasks, Pass@1 goes from 15.2% to 31.0% and Pass@10 from 36.5% to 68.3%.

• Judge audit performance: Two engineers labeled 65 tasks, marking 44 and 41 as problematic with agreement on 40 (Cohen's κ = 0.15); consensus produced 14 accepted and 51 rejected tasks. The three-judge protocol rejected 56 tasks, correctly identifying 49 of 51 problematic ones (87.5% precision, 96.1% recall), but accepted-gold specificity was only 7/14 (50.0%), meaning 12.5% of rejected tasks went to repair unnecessarily.

• Repair effort: Across the 237 difficulty-selected tasks, repair took 4.2 rounds on average (median 4, range 0–14). Post-evaluation validation excluded 35 tasks, producing the final 202.

• Task composition: Core feature development is the most frequent annotation (67.3%), followed by major bug fixes (27.2%) and integration features; labels are not mutually exclusive. Problem statements average 513.25 words, solutions touch 13.14 files with 414.82 reference added lines and 645.87 test added lines, and tasks average 23.97 F2P and 42.91 P2P tests.

Methodology in Plain English

The authors start from SWE-rebench V2, a pipeline that mines merged pull requests from GitHub, builds a Docker environment for each repository at its pre-fix commit, and defines passing and failing tests by applying the PR's test patch before and after the reference fix. They loosen two things and add one new stage.

First, candidate screening. Instead of keeping only "A" candidates whose descriptions look aligned with their tests, they also keep two repairable categories: B2 (tests need names, signatures, or values the statement never mentions) and B5 (tests may depend on incidental details of the reference implementation). This grows the pool from 1,096 to 2,640 candidates.

Second, construction. A Kimi K3 auditor agent does the job that used to belong to an installation script: it reads the repository's docs, config, and CI, works out the native build and test commands, installs dependencies, and also repairs the test patch — generalizing implementation-specific checks or adding missing coverage. It may also blank out implementation hints in the issue text with [HIDDEN], which happened in 56 tasks (113 occurrences). Every candidate is then rebuilt from scratch in a clean container and must show at least one failing-to-passing test with no regressions.

Third, the solver–judge–repairer loop, which is the paper's core idea. A solver model attempts the task. A judge model then reads the trajectory, the solution patch, the execution results, the statement, the reference patch, and the tests, and decides whether a failure is the agent's fault or the task's fault. If it is the task's fault, a repair agent may revise the test patch or append clarifications to the statement — never changing the reference patch, never touching the environment, and never adding clarifications that leak the reference implementation. Repairs are validated by re-executing at the base commit and with the reference patch, and the task cycles back for re-judging, in sets of up to five rounds. A task is accepted only after unanimous acceptance by three different judge models: GPT 5.6 Luna (xhigh), GPT 5.6 Terra (high), and Kimi K3 (high).

Finally, difficulty calibration and release. Each accepted task is attempted three times by Qwen 3.8 27B (xhigh) with a 262K-token context; tasks solved in all three are dropped unless that would leave fewer than 15 tasks in a language. After evaluation, the authors re-audited the 237-task pool using 5,688 stored solver attempts from eight model configurations, fixed evaluator defects, revisited tests that rejected valid patches or accepted invalid ones, and excluded 35 tasks — giving 202 final tasks, 12 to 27 per language.

Why This Matters

Impact on research. Benchmark coverage shapes what the field can claim about coding agents. The paper shows that the standard practice of filtering for "clean" descriptions and unambiguous tests does not just change benchmark size — it conservatively discards roughly two-thirds of the tasks that a repair pipeline can salvage (196 versus 66 re-annotated A tasks). It also quantifies a reliability problem in automatically mined suites: a judge flagged issues in 90.1% of tasks passing a standard construction path, and fixing tests raised measured Pass@1 on unchanged descriptions from 29.1% to 67.2%. That means reported scores on un-repaired benchmarks may reflect overly strict tests, not model failure.

Real-world applications:

• Choosing coding agents for non-mainstream stacks. Teams working in Elixir, Scala, Kotlin, Dart, R, or Julia get per-language evidence rather than extrapolating from Python results — for example, Kimi K3 leads on R while Opus 5.5 leads on Clojure and reaches 100.0% on Erlang.
• Cost-aware model selection. The token and step columns (1.75M tokens and 27.4 steps for GPT 6.1 Sol versus 38.28M and 313.7 for GLM-5.3-Flash) let practitioners trade accuracy against inference budget.
• Test-suite auditing in CI. The judge-for-defect approach transfers to internal test suites that reject valid refactors or accept broken code, and the "hidden test requires exact output" pathology flagged in 300 of 364 tasks is a common real-world testing smell.
• Building domain-specific benchmarks. The reusable pipeline, prompts, and tooling are released, so an organization can construct its own curated issue-resolution set instead of relying on public data.

Industry relevance. Organizations that maintain libraries in these 13 ecosystems now have an executable, containerized evaluation target with published environments and trajectories, and a documented method for keeping the task pool large enough per language to be usable (12–27 tasks per language here) when the supply of eligible pull requests is thin.

Future Directions

• Separating the effects of each refinement step. The authors note that the fixed-patch ablation isolates test refinement on 63 tasks with unchanged descriptions, but description repair is not separately evaluated, and further ablations are needed to disentangle the contributions of the other stages.

• Measuring construction-time bias. Several evaluated models (GPT 5.6, Kimi K3) also participated in construction and judging, so they may benefit from the resulting task interpretations. The authors flag this as an open question and plan to study how a model's involvement in building a benchmark affects its score.

• An independent audit of the released benchmark's error rate. The 65-task human audit measures judge reliability on a sample from multiple pipeline stages, not the defect rate of the final 202 tasks, and its low specificity (7/14) shows the protocol sends acceptable tasks through unnecessary repair.

• Contamination control through date-filtered collection. Because tasks come from public issues and PRs, the pipeline's ability to collect tasks created on or after a specified date is proposed as a way to restrict evaluation to tasks postdating a model's training cutoff, though undisclosed cutoffs make this hard to guarantee.

Target Audience

Benchmark builders and evaluation researchers who need to construct high-quality issue-resolution datasets from scarce or noisy sources; applied ML engineers choosing and budgeting coding agents for non-Python stacks; maintainers and tooling teams in the 13 covered language ecosystems; and reviewers who need to judge the reliability of reported SWE-bench-style scores. Readers new to agent evaluation will find the methodology section accessible, but the results tables assume familiarity with Pass@k and SWE-bench task structure.

Authors’ abstract

Challenging repository-level benchmarks remain limited for many programming language ecosystems, where filtering scarce candidates with ambiguous descriptions or defective tests can discard valuable engineering problems. We introduce Longtail-SWE, a benchmark of 202 issue-resolution tasks across 13 languages with limited coverage in existing repository-level benchmarks: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Its…