SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents

AI summary

Overview

Research area: Software engineering agents / post-training of small language models for long-horizon code repair (cs.SE, cs.LG adjacent).

Technical level: Intermediate. Familiarity with supervised fine-tuning, reinforcement learning for LLMs (GRPO), and agent scaffolds like SWE-agent helps, but the paper's core idea — pairing a small model with an occasional expert — is explained without heavy math.

Scope in one sentence: The paper proposes SWE-Protégé, a two-phase post-training pipeline that teaches a 7B model to remain the primary decision-maker on SWE-bench Verified while learning when to ask a much larger expert model for help, achieving 42.4% Pass@1 with roughly 4 expert calls and about 11% of total tokens coming from the expert.

What This Paper Is About

Small language models (defined in the paper as models with ≤10B parameters) are cheap and fast, but on long-horizon software engineering benchmarks such as SWE-bench they stall out, repeat the same commands, and resolve only about 10% of tasks. The authors argue that the fix is not simply scaling up training data — they show that progressively scaling SWE-smith training can even cause regressions — but rather teaching the small model to collaborate selectively with a strong expert model, the way a junior engineer might occasionally ask a senior colleague for guidance while still doing the work themselves.

Key Contributions

1. A reframing of software repair as expert–protégé collaboration. Instead of routing a whole task to a big model, the small model stays the sole decision-maker and learns to invoke an expert only when it is stalled, using a single added tool called ask_expert.

2. A two-phase post-training pipeline. Phase I is rejection-sampling supervised fine-tuning on expert-augmented trajectories; Phase II is on-policy agentic reinforcement learning with shaped rewards that penalize degenerative looping and shallow or unused expert collaboration.

3. Strong results with minimal data and compute. Lightly post-training Qwen2.5-Coder-7B-Instruct yields 42.4% Pass@1 on SWE-bench Verified with SWE-agent, a +25.4% improvement over the prior SLM state of the art, using the same roughly 5K-trajectory SFT dataset size as SWE-smith and a short single-node RL phase over only 100 tasks for 160 steps.

4. A quantitative characterization of the failure modes being fixed. The paper measures action looping, expert token share, cost, step counts, and failure-mode shifts, and shows that Phase II RL — not Phase I SFT — is what converts stalled states into forward progress.

Main Findings

• Headline accuracy: SWE-Protégé-7B reaches 42.4% Pass@1 on SWE-bench Verified with SWE-agent (Opus 4.1 expert), compared with 40.2% for SWE-agent-LM-32B, 17% for SWE-agent-LM-7B, and 18.2% for Lingma-SWE-GPT-7B. The paper reports this as exceeding SWE-agent-LM-32B by +2.2%, the SWE-agent-LM-7B baseline by +30.6%, and Lingma-SWE-GPT-7B by +24.2%.

• Expert backends matter: In Table 1, the Sonnet 3.7 variant scores 30.8%, the Sonnet 4.5 variant 41.0%, and the Opus 4.1 variant 42.4%.

• Phase II RL helps consistently: Table 2 reports absolute gains over the corresponding SFT checkpoint of +1.2 (Sonnet 3.7, listed at 30.6%), +6.2 (Sonnet 4.5, 41.0%), and +2.8 (Opus 4.1, 42.4%), an average gain of 3.4%.

• Expert use is sparse: Roughly 4 expert calls per task, with expert tokens making up about 11% of total tokens (11.9% with Sonnet 4.5). Only 6 expert calls are permitted across all rollouts.

• Expert replies are short, expert inputs are long: Median / p95 / max expert reply lengths are 500 / 937 / 1,657 tokens, conditioned on experts receiving 8,885 / 20,716 / 43,031 token inputs (the query plus the most recent 5 messages).

• Cost savings: Direct expert execution has median per-task costs of $0.54 (Sonnet 3.7) and $1.24 (Sonnet 4.5), with outliers at $3.04 and $2.88. Under SWE-Protégé, median expert cost falls to $0.13 / $0.15 (4.2× and 8.2× lower) and to $0.65 with Opus 4.1 — even though Opus 4.1 is 5× and 4.54× more expensive per token.

• Total tokens: Per-task total tokens stay roughly constant across experts at about (3–3.2) × 10⁵, and Phase II reduces total token usage by about 40%.

• Data scaling works for the protégé but plateaus for the baseline: SWE-Protégé-7B improves monotonically, e.g. 19.0% (1.3k) → 33.4% (4.8k) with Sonnet 3.7 (+14.4%) and 23.6% → 39.4% with Sonnet 4.5 (+15.8%). SWE-agent-LM-7B goes from 2.2% at 100 trajectories to 17.0% at 2.4k, then regresses to 11.8% at 5.0k.

• Looping is largely eliminated by Phase II: Post-SFT (Phase I), 31.0% of trajectories contain a repeated-action run longer than 10 steps (19.0% >20, 8.0% >40), comparable to SWE-agent-LM-7B/32B at 33.6%/24.4%. After Phase II, only 0.8% exceed 10 and none exceed 20, versus 1.8% for both Sonnet 3.7 and Sonnet 4.5.

• Behavioral shaping dynamics: Under loop-aggressive shaping, mean loop penalty rises from about −1.3 toward about 0 and the loop-negative rate falls from about 0.50–0.65 to near zero by about 100 global steps, yet accuracy improves only marginally. Gains appear after switching to follow-aggressive shaping, where mean follow-through rises from about 0.2 to about 0.8 and negative follow-through falls from about 0.30–0.40 to about 0.05.

• Trajectories get shorter, not prematurely terminated: Mean steps per task drop from about 60 after SFT to about 20 after Phase II, while expert calls stay stable at around four. On average 10.8% of instances are resolved after ≥40 steps.

• Autonomy beats forced escalation: Removing the policy's ability to request help hurts — fixed-interval invocation drops to 19.6% and random invocation to 24.2%. A loop heuristic helps only when the expert sees the full interaction history: 33.4% (Loop ✓, Ctx ✗) versus 29.0% (Loop ✗, Ctx ✗), and provides no measurable benefit under Ctx ✓ (29.4% either way).

• In-house experts underperform frontier experts: Experts post-trained from SWE-Protégé-7B and SWE-Protégé-32B checkpoints using 5,623 examples yield task resolution of 17.0% and 20.8% respectively under the same protégé policy.

• In-place data modification fails: Injecting expert calls into existing SWE-smith traces performs worst at 14.2%, below the 17.0% SWE-agent-LM-7B baseline, suggesting gains come from coherent interactions rather than merely adding expert tokens.

• Contamination check holds up: On a held-out SWE-smith-style subset of 400 tasks excluded from training and released after the expert models became available, the 7B model achieves 32.0% without expert calls and 40.3% with Sonnet 3.7; the 32B variant achieves 41.5% and 43.0% respectively.

• Generous step limits do not help experts: Directly running Sonnet 3.7 and Sonnet 4.5 on SWE-agent with step limits of 8 and 16 causes performance to drop sharply to 18.2% and 26%.

Methodology in Plain English

The researchers start with an off-the-shelf SWE-agent scaffold and add exactly one tool: the small model can emit a structured call asking an expert for help, together with a query. The expert sees only a compact state — the most recent 5 messages — while the small model always acts from the full history.

Phase I teaches the mechanics. A strong code-capable model (Claude Sonnet 3.7) is placed in the same environment, prompted to call the expert only when appropriate, and used to generate trajectories. That model serves as its own expert and is briefly allowed to see the ground-truth solution, which produces more correct trajectories (38% more usable trajectories than without the ground-truth patch). The small model is then fine-tuned on this data with standard next-token cross-entropy over about 4.8K resolved tasks, sequence length 32,762, batch size 32. No auxiliary loss encourages or discourages expert calls; sparsity emerges from imitating trajectories where calls are rare.

Phase II teaches judgment. Starting from the SFT checkpoint, the team runs a short on-policy GRPO phase with a composite trajectory-level reward: a penalty for command repetition, a shaping term for deferral quality scored by the expert acting as an in-trajectory judge, a follow-through term scored by whether the model acted on guidance and reported back, and gates that downweight correctness-based rewards when looping or follow-through are pathological. The shaping schedule runs in two stages: loop-aggressive for steps 1–80, then loop+follow-aggressive for steps 81–160. A hard −10 penalty applies when no expert call is made in the second stage. Training uses 100 tasks from SWE-Gym, 6 rollouts per prompt, batch size 16, and 160 steps on a single node with 8 NVIDIA A100/H100 80G GPUs.

Evaluation is on SWE-bench Verified, a human-vetted 500-instance subset drawn from 12 GitHub repositories, reporting single-rollout Pass@1 with no test-time sampling or majority voting.

Why This Matters

Impact on research: The paper challenges the assumption that long-horizon software engineering is inherently out of reach for small models, and shows that the interaction pattern between models — not just data volume — is a lever worth studying. It also provides evidence that naive data scaling of distilled trajectories can regress, and that reinforcement learning with collaboration-aware rewards is what converts stalling into progress.

Real-world applications (as motivated in the paper):

• Cost- or quota-constrained application services such as Cursor or OpenClaw, where frontier-model tokens are a binding constraint and the paper envisions a SWE-Protégé-like paradigm shining.
• Software engineering workflows where agentic assistance needs to run at lower latency and compute cost, making it accessible where budget or infrastructure precludes always-on frontier models.
• Layered assistance systems in production engineering, where a small model handles routine exploration and tool use while escalation paths and checks are invoked selectively.
• Other long-horizon agentic domains such as data analysis, which the authors note the techniques are not tied to SWE-agent or SWE-bench and could in principle be applied to.

Industry relevance: The economics are the story. Expert tokens make up only about 11% of total tokens, expert calls average about four per task, and median expert cost per task falls to $0.13–$0.15 for Sonnet backends — up to 8.2× lower than letting the expert solve the whole task. Gains are achieved with the same roughly 5K SFT trajectories as SWE-smith and a short single-node RL phase, which keeps the recipe reproducible outside of very large compute budgets.

Future Directions

• Expert post-training and co-adaptation. The authors treat the expert as a fixed black-box backend and explicitly leave more principled expert post-training and co-adaptation as an important future direction. Their in-house experts (17.0% and 20.8%) underperform frontier backends, suggesting this is non-trivial.
• Richer collaboration strategies. The paper does not exhaustively explore design choices, naming expert interrupts, richer bidirectional control, and other collaboration strategies as open areas; fixed-interval and random invocation both underperformed learned escalation.
• Hyperparameter and model-family sweeps. Phase I/II hyperparameters, broader student model families, and the observation that LoRA and QLoRA underperformed full SFT on SWE-bench Verified remain relatively unexplored.
• Domain transfer. Although the focus is SWE-bench within the SWE-agent framework, the authors suggest the techniques could apply to other domains such as data analysis, and note the evaluation focuses primarily on Python-based repositories.

Target Audience

Researchers and practitioners working on software engineering agents, LLM post-training, reinforcement learning for agents, and efficient deployment of language models. It is especially relevant to engineers who need strong coding-agent performance under cost, latency, or rate-limit constraints, and to researchers interested in multi-model collaboration, model routing, and small language model capability building on long-horizon tasks.

Authors’ abstract

Small language models (SLMs) offer compelling advantages in cost, latency, and adaptability, but have so far lagged behind larger models on long-horizon software engineering tasks such as SWE-bench, where they suffer from pervasive action looping and low resolution rates. We introduce SWE-Protégé, a post-training framework that reframes software repair as an expert-protégé collaboration problem. In SWE-Protégé, an SLM remains the sole decision-maker while learning to selectively seek guidance from a strong expert model, recognize stalled states, and follow through on expert feedback. Our approach combines supervised fine-tuning on expert-augmented trajectories with agentic reinforcement learning that explicitly discourages degenerative looping and unproductive expert collaboration. We lightly post-train Qwen2.5-Coder-7B-Instruct to achieve 42.4% Pass@1 on SWE-bench Verified, a +25.4% improvement over the prior SLM state of the art, while using expert assistance sparsely (~4 calls per task and 11% of total tokens).