SWE-IF: Aligning Code Evaluation with Human Preference

AI summary

Overview

Research area: Code LLM evaluation, instruction following, and human-preference alignment Technical level: Intermediate Scope: The paper introduces a taxonomy of 30 verifiable code instructions (VeriCode) and an augmented evaluation testbed (SWE-IF) that measures code instruction following alongside functional correctness, then shows that combining the two correlates better with human preference than functional correctness alone.

What This Paper Is About

Code evaluation today is dominated by pass@k, which only checks whether generated code passes unit tests. That ignores many things users actually care about during "vibe coding" — clean style, sensible logic patterns, good documentation, proper error handling, and respect for prior intent across multi-turn edits. The authors argue that instruction following (IF) is the missing measurable signal behind human preference, and they build tools to quantify it at scale.

Key Contributions

1. VeriCode taxonomy — A curated set of 30 verifiable, non-functional code instructions organized into five categories (Coding Style & Conventions, Logic & Code Patterns, Documentation & Commenting, Error Handling & Exception Management, Library & API Constraints). Each instruction pairs with a deterministic verifier (mostly Ruff linter rules, plus AST analysis and regex), so success is a binary pass/fail rather than a subjective judgment. A Parameters field (e.g., line_length, max_branches, docstring convention) lets the 30 core instructions expand programmatically into hundreds of variants.

2. SWE-IF testbed — Two augmented benchmarks derived from established suites: Big-SWE-IF (from BigCodeBench's 1,140 real-world programming tasks) and Live-SWE-IF (from LiveCodeBench v1–v6, 1,055 contest-style problems). Each instance is augmented with an LLM-selected, non-conflicting subset of five instructions plus parameter values, yielding over 10K instruction-level evaluations. The testbed measures functional correctness via original unit tests and instruction following via verifiers.

3. Dual-protocol evaluation design — Two interaction settings that mirror real usage: single-turn generation (all constraints in one prompt) and multi-turn editing (constraints revealed one at a time, with the model updating its solution each round). Functionality is measured as functional regression FR_k relative to the base score; IF is measured at both instruction level (average pass rate) and task level (all constraints satisfied).

4. Large-scale empirical study — Evaluation of 31 LLMs across 10 model families (Gemini, Claude, OpenAI, DeepSeek, Qwen, Grok, Gemma, Mistral, MiniMax, Kimi), with correlation analysis against LMArena coding Elo ratings from over 800K human votes.

Main Findings

• Non-functional instructions cause measurable functional regression. Adding constraints that target style, documentation, or library usage (not functionality) lowers pass@1 across every model. Under five instructions, average pass@1 drops 5.85% on Big-SWE-IF and 6.61% on Live-SWE-IF. On Big-SWE-IF multi-turn with five instructions, every model regresses more than 5% except Gemini 2.5 Flash and Claude 4 Opus; on Live-SWE-IF, o4-mini and Kimi K2 exceed 10% regression in more than half of the tested configurations.

• Following multiple instructions remains hard. Even the best models reach only 46.75% (Big-SWE-IF) and 40.95% (Live-SWE-IF) task-level IF under five instructions. With three or more instructions, most advanced models fall below 50, and on Live-SWE-IF five of the seven leading models fall below 30 in the single-turn setting. Because each added instruction multiplies the pass probability, task-level success decays exponentially.

• Single-turn preserves functionality; multi-turn follows instructions better. Average regression on Big-SWE-IF climbs from 2.48% to 5.76% (single-turn) versus 3.18% to 9.31% (multi-turn) as instructions go from one to five. Conversely, multi-turn editing holds a 3–4.5% IF advantage on Big-SWE-IF and roughly 8% on Live-SWE-IF, suggesting iterative edits let models make targeted revisions while single-turn generation prioritizes overall correctness.

• Position bias affects instruction following. On Big-SWE-IF, instruction-level IF traces a U-shape — the classic "lost-in-the-middle" pattern — despite prompts being only a few hundred tokens long. Single-turn shows primacy bias (first instruction followed best); multi-turn shows recency bias (last instruction followed best). While the U-shape does not replicate on Live-SWE-IF, the primacy/recency directions hold across both benchmarks.

• Human preference reflects a blend of both metrics. Correlating LMArena coding Elo with a composite α·IF + (1−α)·Func peaks at intermediate α on both benchmarks, outperforming either metric alone. On Big-SWE-IF, the optimum is α = 0.4 for Pearson and α = 0.7 for Spearman; the optimum for Spearman and Pearson on Live-SWE-IF is similar.

• Which factor matters most depends on task type. For real-world programming tasks (Big-SWE-IF), pure IF correlates over 0.1 points higher with human preference than pure functional correctness on Spearman. For algorithmic contest tasks (Live-SWE-IF), the opposite holds — functional correctness dominates.

• The taxonomy is robustly applicable. A manual spot-check of 500 selected instructions (100 instances) yielded a 100% validity rate for both relevance and non-conflict. The main source of noise is parameter selection, with a low invalid rate of roughly 1–3% (0.96% for Claude 4 Opus as selector versus 2.47% for Gemini 2.5 Pro).

Methodology in Plain English

The authors start from an observation: on platforms where human programmers pick their preferred code snippet (like Copilot Arena), model rankings don't line up well with functional benchmark scores. They hypothesize that non-functional instruction following explains much of the gap. To measure that, they build a taxonomy: they source candidate rules from Ruff, a Python linter with 800+ rules, then consolidate overlapping rules, filter to broadly applicable ones, and drop any instruction that a strong model already follows more than 90% of the time without hurting its pass rate. Each remaining instruction gets a deterministic verifier so evaluation is pass/fail rather than a subjective judgment call.

Next, they take two existing benchmarks (BigCodeBench and LiveCodeBench) and, for each problem, run an LLM selector that picks five relevant, non-conflicting instructions with parameter values. They validate parameters with rule-based checks. Then they run 31 models on the augmented suites in two protocols: all instructions in one prompt, or revealed one per round. They compare functional scores against the originals and compute instruction-following scores at both per-instruction and all-instructions-satisfied granularity. Finally, they correlate model performance against LMArena Elo ratings to see whether a blend of the two metrics matches human preference better than either alone.

Why This Matters

Impact on research: The paper challenges the dominance of pass@k as the reward signal in code LLM evaluation and RLVR training. It shows that optimizing purely for functional correctness steers models away from a quality notion that humans actually apply, and it provides a verifiable, scalable alternative signal that could serve both evaluation and post-training.

Real-world applications:

• Coding assistants (Copilot, Cursor, and similar tools) could use instruction-following scores to select better responses in interactive sessions, where users care about style and intent preservation, not just passing tests.
• Automated code review and linting pipelines could adopt the taxonomy to check whether generated patches respect project conventions, documentation standards, and targeted-edit constraints.
• Model leaderboards and benchmarks could report composite scores that better predict which models users will actually prefer on day-to-day programming vs. competitive coding.
• Reinforcement learning from verifiable rewards could use the linter-backed verifiers as dense, deterministic rewards to train models toward non-functional code quality without human labels.

Industry relevance: The taxonomy is grounded in industrial linters (Ruff) and real style guides, making it directly usable by teams that already enforce such standards internally. The clear distinction between real-world programming and algorithmic contest settings tells developers which model strengths matter for their specific workload.

Future Directions

• Extending VeriCode beyond Python. The framework is language-agnostic in principle, but the current instantiation covers only Python; applying it to other languages would require mapping to their respective linters and standards.

• Using IF verifiers as training signals. Given the low cost and determinism of the verifiers, a natural follow-up is to test whether RL or SFT with these rewards improves human alignment without harming functional correctness.

• Understanding and mitigating position bias. The "lost-in-the-middle" pattern in instruction following at short context lengths is surprising and unexplained; work on ordering strategies or attention mechanisms could address it.

• Better handling of parameter selection. Automated selection of instruction parameters is the main error source in the augmentation pipeline, so improving or validating parameter generation is a practical open problem.

• Broadening human-preference validation. The correlation analysis uses LMArena Elo as the preference proxy; validating against votes from domain-specific developers or real IDE interactions would strengthen the claim.

Target Audience

Researchers and engineers working on code LLM evaluation, code generation, and post-training with verifiable rewards; benchmark designers looking for metrics beyond functional correctness; and product teams building or selecting AI coding assistants who need a measurable notion of code quality that aligns with real developer preferences. The paper is accessible to readers with basic familiarity with LLM benchmarks, though fluency with code evaluation terminology (pass@k, unit tests, linters) helps.

Authors’ abstract

Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check. Vibe check reflects human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check besides functional correctness. To quantify models' code instruction-following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in SWE-IF, a testbed to assess both instruction following and functional correctness. Evaluating 31 LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit functional regression. Most importantly, a composite score of functional correctness and instruction following correlates best with human preference, with instruction following emerging as the primary differentiator among LLMs. Our code, data, and taxonomy are available at https://github.com/maszhongming/SWE-IF.