One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents

AI summary

Overview

Research area: Software engineering (cs.SE) — post-training of large language model agents for repository-level software engineering (SWE), combining category-aware reinforcement learning, supervised fine-tuning, and multi-teacher on-policy distillation.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning for LLM agents (RLOO, policy-gradient objectives, on-policy distillation), executable SWE benchmarks, and expert merging/distillation methods.

Scope: The paper proposes and evaluates a framework — executable task construction, a hierarchical "SWE Labeler" taxonomy, category-specific expert training via Refresh–Repair–Expand, and label-routed multi-teacher on-policy distillation — for turning heterogeneous repository-level SWE tasks into one deployable agent that improves both aggregate and per-category resolution.

What This Paper Is About

Repository-level software engineering (SWE) covers many different kinds of work — service and data-layer bug fixes, user-facing interface changes, infrastructure and tooling changes, performance or security patches — yet most agent training treats them as one pooled distribution. The authors observe a "category see-saw": when a single policy is trained on a pooled mixture, gains in some task categories coincide with regressions in others, and a single aggregate resolution score hides this redistribution. The goal is to build a single deployable SWE agent that improves across all categories at once, by training category-specific experts from the same base model and then consolidating them into one policy without any external model supplying solution trajectories or action targets.

Key Contributions

1. Category-aware specialization and integration within SWE. The authors state this is the first study to organize repository-level SWE tasks into semantic repository-domain categories, train category-specific RL experts, and consolidate them into a single policy through label-routed multi-teacher on-policy distillation (MOPD). Category-level evaluation connects the motivating see-saw observations to a practical objective: improving overall and per-category resolution in one deployable policy.
2. Self-improving experts through Refresh–Repair–Expand (RRE). Instance-level diagnostics show that average initial-RL gains coexist with observed regressions. RRE couples policy development with data selection: each updated expert refreshes instance mastery, reuses verified successes from its own preceding RL trajectories for Repair SFT, and selects a new training frontier from the broader pool. Agentic-miniRL supplies the long-horizon RL recipe.
3. SWE Labeler, an evidence-grounded multi-axis labeling system. Source-grounded label definitions and explicit decision rules apply both to issue–change instances and to agent interaction trajectories. Two hierarchical semantic axes and three ordinal scale axes provide a common foundation for category-level analysis, training-pool construction, and teacher routing.
4. An evaluation framework that exposes hidden redistribution. The paper quantifies category-level progress with the minimum category gain over the base model (G_sim) and the "see-saw gap" (SSG = Δ_all − G_sim), making regressions visible that aggregate resolution obscures.

Main Findings

• Aggregate scores hide opposing category-level changes. Under Pooled RL, gains in some categories coincide with regressions in others, and positive overall gains can coexist with a negative minimum category gain. The paper calls this pattern the category see-saw.
• Final policy results. The final MOPD policy achieves a mean resolution of 58.04% on Pro-618 and 59.00% on SWE-bench Multilingual, improving over the base model by 5.39 and 2.78 percentage points, respectively.
• Pro-618 construction. The authors evaluate on 618 of the 731 SWE-bench Pro instances, excluding instances flagged in community reports for faulty environments, broken container images, or unreliable evaluation logic. The subset splits into Pro-A (service/data/security, 221 tasks), Pro-B (user-facing applications, 201 tasks), and Pro-C (systems, tooling, and runtimes, 196 tasks).
• Pooled RL baseline setup. The motivating Pooled RL experiment trains a single policy on the full 6,723-task mixture and evaluates it on the fixed Pro-618 split, with step 0 anchoring the base model.
• Initial expert RL is uneven. In initial category-specific RL runs, average training-instance success rates improve, but some instances regress while others improve — which is the stated motivation for explicit consolidation of successful behavior and repeated reassessment of which tasks provide useful training signal.
• SWE Labeler taxonomy scale. The final taxonomy contains 26 Task Type L1 families with 119 L2 labels and 21 Repository Domain L1 families with 108 L2 labels. The three scale axes each have four ordered levels, for 12 scale levels in total. The label contract is instantiated for all 119 Task Type L2 labels, 108 Repository Domain L2 labels, and 12 scale levels.
• Label profiles differ sharply across benchmarks. Applied to 1,531 instances from three corpora: SWE-bench Verified (500 instances, 1 language, bug-fix 87.0%, web_backend 46.4%, 86.2% single-file); SWE-bench Pro (731 instances, 3 languages, bug-fix 47.3%, devops_infra 33.2%, 59.2% cross-module); SWE-bench Multilingual (300 instances, 8 languages, bug-fix 89.3%, lang_runtime 24.7%, 66.0% single-file).
• Routing configuration. For the main configuration, 18 routable Domain L1 families are coarsened into three broad engineering contexts (A, B, C), instantiating category-aware specialization under a fixed RL budget while retaining the full label vector for analysis.
• Training substrate. The executable task pool contains approximately 32K executable candidates; each instance has a base revision, an issue-style problem statement, an environment specification, and executable tests, validated by running tests before and after the gold patch.
• No external teacher trajectories. Expert training and policy integration use no external model to provide solution trajectories or action targets.
• Benchmark references. SWE-bench Verified curates a human-validated subset of 500 Python instances; SWE-bench Multilingual extends evaluation to 300 tasks across 42 repositories and nine programming languages.
• Not reported in the available content. The truncated text does not report the numeric scores for the Pooled RL and Balanced RL baselines, the individual expert-gain recovery figures, the exact RL budget or hyperparameters, the identity of the base model beyond a model listing reading "Logics-SWE-Qwen3.6-27B," or the full results of the ablation studies referenced in Section 5.

Methodology in Plain English

The authors take one domain — repository-level SWE — and break it into pieces rather than training on everything at once.

First, they build an executable training set from merged repository pull requests and linked issues. Each instance gets a base revision, a rewritten problem statement, a Docker environment, and tests. They check for information leakage (statements that reveal source paths, test paths, or patch code), deduplicate, and audit for contamination. They validate each instance by running its tests before and after the gold patch, keeping only fail-to-pass and pass-to-pass tests, and they filter flaky or time-decayed verifiers. The result is roughly 32K executable candidates.

Second, they classify tasks with SWE Labeler, a hierarchical, evidence-grounded labeling system. Every label is a decision record linking an authoritative source (for example, ISO/IEC 25010, ISO/IEC/IEEE 14764, MITRE CWE, or Fowler's refactoring catalog) to an operational rule, observable positive signals, and boundaries to neighboring labels. It uses two semantic axes (Task Type, Repository Domain) plus three ordinal scale axes (modification scope, cognitive complexity, estimated resolution time), with annotation done hierarchically: predict L1 first, then choose L2 labels only from that L1's children and assign scale levels. The same taxonomy handles both static issue–change records and interactive agent trajectories through two input adapters.

Third, they use Repository Domain L1 as the routing axis and coarsen 18 routable families into three engineering contexts (A, B, C). Each category gets its own expert, all starting from the same base model, trained in a Refresh–Repair–Expand loop: the updated policy refreshes which instances it has mastered, reuses its own verified successful trajectories for Repair SFT, and reselects tasks for further RL. The RL phase uses Agentic-miniRL, a long-horizon multi-turn variant of MiniRL with behavior-policy correction, RLOO advantages, the K1 reference penalty in the reward path, and turn-aware loss reduction over assistant actions separated by tool observations.

Finally, the experts are merged into one deployable student via label-routed multi-teacher on-policy distillation (MOPD), in which the student's own long-horizon trajectories are used and a ReLU-gated reward extrapolation keeps only each teacher's improving direction over the reference. Evaluation happens at two levels at once: overall resolution plus per-category resolution, the minimum category gain, and the see-saw gap.

Why This Matters

Impact on research. The paper reframes SWE agent post-training as a heterogeneity problem rather than a single scalar optimization problem. It introduces measurable diagnostics (G_sim, SSG) for detecting when aggregate gains conceal per-category regressions, applies cross-domain expert splitting and fusion — previously studied across domains like mathematics, code generation, and instruction following — inside a single domain where the partition must be established by observable labels rather than given by domain boundaries, and shows a self-improvement loop that uses no external teacher trajectories.

Real-world applications

• Enterprise engineering agents that must handle a mixed backlog spanning service backends, user-facing applications, and infrastructure/tooling code without regressing on any one class of work.
• Multilingual and multi-ecosystem maintenance, where repository ecosystems and language runtimes differ substantially, as the SWE-bench Multilingual results illustrate.
• Training-data curation and evaluation auditing for SWE datasets, using the SWE Labeler's cross-benchmark profiles to detect benchmark imbalance (for example, the 47.3% bug-fix share on SWE-bench Pro versus 87.0% on SWE-bench Verified) and to exclude instances with faulty environments.
• Monorepo or platform teams that maintain code across three broadly different engineering contexts and want one deployable model rather than several specialized ones.

Industry relevance. The work is an industrial-scale study (Alibaba Group affiliation) that operates on approximately 32K executable candidates and reports results on public benchmarks. Its routing scheme, which assigns training streams by observable repository domain, is directly transferable to production agent platforms, and its category-level reporting offers a template for internal model evaluation that does not let a single headline number hide regressions.

Future Directions

• Determine whether category separation alone produces stronger experts. The authors explicitly frame this as an open question: initial expert RL improved average training success while leaving uneven instance-level progress, and the paper motivates RRE as the response.
• Extend the routing granularity beyond three routes. The authors note the full annotation is deliberately richer than a direct expert router, since assigning one expert per Repository Domain L1/L2 label would fragment RL examples and optimization budget; how finer routing trades off against data and compute is left open.
• Investigate the optimization-budget imbalance identified in prior multi-teacher work. The related work cites analysis indicating optimization-budget imbalance as a source of incomplete multi-teacher integration, which is directly relevant to how the A/B/C teachers are consolidated by MOPD.
• Test the label space outside its construction benchmarks. The paper notes that the same label space can organize training and held-out tasks independently of whether they originate from Pro-618, leaving open how well the categories and the routing rules generalize to other repositories, languages, and task families.

Target Audience

Researchers and engineers working on LLM agents for software engineering, reinforcement learning for long-horizon tool-using agents, multi-teacher distillation and expert merging, and benchmark design for executable SWE tasks. It is also relevant to practitioners building production code agents who need per-category rather than aggregate evaluation, and to dataset curators interested in evidence-grounded, auditable labeling taxonomies. The paper is written at an advanced level and assumes prior familiarity with RL post-training and SWE benchmarks.

Authors’ abstract

Repository-level software engineering (SWE) comprises heterogeneous task categories, whose progress under pooled agentic reinforcement learning can be uneven: gains in some categories coincide with regressions in others, while aggregate resolution obscures these changes. Motivated by this category see-saw, we develop a category-aware expert-training and policy-integration framework. Executable task construction and SWE Labeler, an evidence-grounded multi-axis labeling system, organize the training pools. Initial category-specific RL improves average training success while leaving uneven instance-level progress, motivating explicit consolidation of successful behavior and policy-adaptive task selection. Same-origin category experts alternate long-horizon Agentic-miniRL with Refresh-Repair-Expand (RRE): the updated policy refreshes instance mastery, reuses its own verified successful trajectories for Repair SFT, and reselects tasks for further RL. Label-routed multi-teacher on-policy distillation (MOPD) consolidates the experts into one deployable student, with ReLU-gated reward extrapolation keeping only each teacher's improving direction over the reference. Expert training and policy integration require no external model to provide solution trajectories or action targets. We evaluate Pooled RL and Balanced RL, expert development, and single-model integration through aggregate and per-category resolution, the minimum category lift over each joint-RL baseline, and expert-gain recovery. The final MOPD policy achieves mean resolution of 58.04% on Pro-618 and 59.00% on SWE-bench Multilingual, improving over the base model by 5.39 and 2.78 percentage points, respectively.