Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models

AI summary

Overview

Research area: Empirical software engineering / code-generation machine learning — specifically, how the choice of output format (whole-file generation vs. iterative diff-based edits) affects the quality of LLM-based code editing.

Technical level: Intermediate. The paper assumes familiarity with fine-tuning language models, perplexity-style metrics (bits-per-byte), static analysis, and diff/search-replace edit formats, but its arguments are presented without heavy mathematical machinery.

Scope: A single-domain, within-dataset comparison of two output regimes across two architecturally different Flutter/Dart code models, evaluated on roughly 1,790 held-out editing tasks per model using compilation pass rate, bits-per-byte, similarity, and blinded LLM-judge ratings.

What This Paper Is About

Production coding tools generally prefer diff-style output — having a model emit localized search/replace edits rather than rewriting an entire file — because diffs are shorter to generate and less likely to corrupt untouched code. Whether that preference also holds as a training objective, rather than just an inference-time interface, is unsettled: prior work argues in both directions. This paper isolates the question by training two very different backbones (a ~100M-parameter model built from scratch and a fine-tuned Qwen2.5-Coder-0.5B) in both a direct whole-file regime and an iterative "steps" diff-based regime, holding the data, tokenizer pipeline, and evaluation harness fixed, then measuring which regime actually produces better code.

Key Contributions

1. A controlled four-arm comparison. The paper trains rainbow-pony-direct, rainbow-pony-steps, qwen-direct, and qwen-steps on the same underlying Flutter/Dart task pool and evaluates all four under greedy decoding, isolating output regime from confounds such as different base models or datasets.

2. A negative result with a failure attribution. Direct generation beats steps mode on every measured metric for both architectures, and the paper demonstrates that most steps-mode failures occur inside trajectories that completed normally rather than from step-budget exhaustion or mechanical apply errors.

3. Robustness checks against selection bias. A matched-ID comparison (comparing steps and direct on identical sample IDs, restricted to clean trajectories) and a blinded gpt-4.1 judge restricted to code that passes dart_pass on both sides show the gap persists — and is larger, not smaller, once task-difficulty selection is accounted for.

4. The "task locality" explanation. The paper unifies a category-level finding (steps wins concentrate in refactoring_edits and error_handling_and_edge_cases) and a trajectory-length finding (steps wins concentrate at short num_steps) into a single variable: how spatially narrow and self-contained the required change is.

Main Findings

• Direct generation wins in aggregate by a wide margin. On dart_pass, Rainbow-Pony-100M direct scores 0.802 [0.783, 0.820] versus steps 0.347 [0.325, 0.369] (n = 1,789 each); Qwen2.5-Coder-0.5B direct scores 0.900 [0.885, 0.913] versus steps 0.501 [0.478, 0.524] (n = 1,792 each). That is a 45.5-percentage-point gap for Rainbow-Pony and 39.9 points for Qwen. Direct also wins on bits_per_byte (0.107 vs. 0.180 for Rainbow-Pony; 0.088 vs. 0.142 for Qwen) and similarity_ratio (0.511 vs. 0.439; 0.568 vs. 0.493).

• Most steps-mode failures are silent content corruption, not process failures. The large majority of steps trajectories complete normally: stop_reason=done accounts for 1,525 of 1,789 (85.2%) rainbow-pony-steps rows and 1,456 of 1,792 (81.3%) qwen-steps rows. Yet dart_pass within done trajectories is only 0.356 (Rainbow-Pony) and 0.569 (Qwen) — both far below their direct counterparts. Approximately 84% of all Rainbow-Pony steps-mode failures and approximately 70% of Qwen steps-mode failures occur in trajectories that completed normally.

• Ambiguous-edit fallback is a major failure driver. Within done-only trajectories, dart_pass is 0.570 without fallback versus 0.123 with fallback for Rainbow-Pony (795 vs. 730 rows), and 0.800 versus 0.175 for Qwen (919 vs. 537 rows). Fallback-affected rows account for roughly 65% (Rainbow-Pony) and 71% (Qwen) of all failures within the done bucket. The paper attributes this to its first-occurrence disambiguation heuristic for search spans that match multiple locations.

• The gap survives matched-ID comparison, and grows. Restricting to clean steps trajectories (done, no fallback, fewer than 20 steps) gives 792 clean IDs for Rainbow-Pony with a clean-steps dart_pass of 57.2% versus matched-direct 82.1% (a 24.9-point gap), and 919 clean IDs for Qwen with 80.0% versus 92.4% (a 12.4-point gap). Matched-direct also exceeds each arm's own full-population rate, confirming that clean steps trajectories skew toward intrinsically easier tasks.

• A blinded judge confirms a genuine residual quality difference. On matched-ID rows where both modes pass dart_pass (n = 390 for Rainbow-Pony, n = 688 for Qwen), the fully blinded gpt-4.1 judge rated direct higher on all six comparisons (three dimensions × two architectures), with every pairwise difference significant under Welch's t-test at p < 0.001. Examples: Rainbow-Pony code quality 3.96 (steps) vs. 4.46 (direct), t = −6.87; Qwen correctness 4.36 vs. 4.75, t = −8.17 (1–5 scale).

• Steps mode does win on a real, non-random subset. Per-row win rates range 3.5%–4.8% for dart_pass, 0.1%–0.3% for bits_per_byte, 33.7%–34.8% for similarity_ratio (which the authors flag as a metric artifact, since both the steps output and the reference were built by incremental edits from the same initial_code), and 4.8%–9.2% for the judge dimensions. The authors treat the dart_pass and judge win rates as the meaningful signal.

• File length does not explain the wins. The initial_code_len effect has opposite signs between architectures (Rainbow-Pony wins skew toward 6% longer initial files; Qwen wins skew toward 11% shorter ones), and both effects are small.

• Steps wins concentrate in two categories. Relative to a ~11.2% baseline share, refactoring_edits accounts for 18.6% (Rainbow-Pony) and 14.3% (Qwen) of dart_pass wins, and error_handling_and_edge_cases for 15.1% and 15.9%. These are the only two categories consistently overrepresented in both architectures. Win counts are small (n = 63–86 total wins per architecture).

• Steps wins concentrate at short trajectories. For rows where steps wins on at least two of three judge dimensions, mean num_steps is 5.54 (median 5) versus 8.94 (median 8) for all other rows in Rainbow-Pony (t = −8.60), and 6.65 (median 6) versus 9.02 (median 8) for Qwen (t = −4.54). All six per-dimension tests are significant at p < 0.001. Win rate declines from roughly 10–18% at num_steps 1–6 down to approximately 0% by num_steps 14–19.

• The two findings are one mechanism. The two win-overrepresented categories are, independently, the two lowest mean-num_steps categories out of nine in both architectures: refactoring_edits (median 2–3) at 4.14 (Rainbow-Pony) and 4.49 (Qwen), and error_handling_and_edge_cases at 6.84 and 7.24. The authors name this underlying variable task locality.

Methodology in Plain English

The setup is a within-dataset A/B test. Both models are fine-tuned on the same pool of 14,600 hand-designed Flutter/Dart tasks (bbidpa/flutter-full-examples-v1, goal/initial_code/final_code triples, 36 task types across three complexity tiers, Apache-2.0). Direct mode draws 5M tokens from this pool and learns to emit the entire modified file in one pass. Steps mode draws 50M tokens from a step-decomposed derivative dataset (bbidpa/flutter-diff-steps-v1, 100K–1M rows), where each full-file example is expanded into a chain of individual search/replace edits, and learns to emit one <ACTION> plus <CHANGES> hunks per turn until it emits a stop action or hits a 20-step budget. Notably, the two regimes are not token-matched — steps mode receives roughly 10× more fine-tuning tokens, which the authors flag and argue cuts against their central finding.

At inference, each steps-mode edit is applied by exact match: a search span occurring once is replaced, zero times triggers an apply_failed step, and more than once invokes a first-occurrence fallback heuristic. Evaluation uses a held-out set of about 1,790 examples per architecture (1,789 for Rainbow-Pony, 1,792 for Qwen, with the difference attributed to tokenizer behavior under a fixed 1,024-token block size) under greedy decoding, scoring dart_pass (Dart static analysis), bits_per_byte against the reference final_code, similarity_ratio, stop reason, trajectory length, and fallback count.

To rule out the possibility that steps mode simply fails on harder tasks, the authors define a "clean" subset (done, zero fallbacks, fewer than 20 steps) and compare it only against direct-mode results on the same sample IDs. They then further restrict to rows passing dart_pass on both sides and score those with a fully blinded gpt-4.1 judge on a 1–5 scale for goal_fulfillment, correctness, and code_quality — the judge sees only the instruction, the initial file, and one candidate output, never the model name, mode, or dart_pass outcome. The judge ran over essentially the full held-out set: 1,792/1,792 for both Qwen arms, 1,789/1,789 for rainbow-pony-direct, and 1,788/1,789 for rainbow-pony-steps (one row skipped with judge_error=empty_output_code), totaling 7,161 judged rows.

Why This Matters

Impact on research. The paper contributes direct evidence to an open question about whether edit-sequence training is generally beneficial (as LintSeq argues for code synthesis) or format-dependent (as the adaptive-format work of Cheng et al. argues). Its answer is that on Flutter/Dart editing of an existing file, edit-based training underperforms badly, and that the conditions where it does win map onto a single measurable variable — task locality — rather than to category labels. That reframes the LintSeq result as reconcilable rather than contradictory: constructing a new program through lint-verified incremental edits is a different problem from locating a change inside an already-correct file, where a wrong disambiguation silently breaks working code.

Real-world applications:

• Choosing an output format for IDE-integrated Flutter/Dart assistants, where short localized fixes (refactors, edge-case fixes) may favor diff output but larger changes favor whole-file generation.
• Designing training-data pipelines for small, domain-specific code models: step-decomposition inflates a training corpus roughly 10× in rows without producing a better editor in this domain.
• Stress-testing edit-application harnesses — the paper's first-occurrence fallback heuristic accounts for 65%–71% of failures within completed trajectories, pointing at ambiguity rejection as a concrete harness improvement.
• Setting expectations for agentic coding tools that emit patches: the paper's steps mode is a no-feedback, single-pass approximation of that loop, so its results bear on the "generate the patch, then apply it" stage specifically.

Industry relevance. The finding that a ~100M from-scratch model reaches 0.802 dart_pass in direct mode but only 0.347 in steps mode, while a fine-tuned 0.5B model reaches 0.900 versus 0.501, suggests that output-regime choice is a first-order design decision comparable in size to model scale or pretraining pedigree for this task. The paper also discloses an unintentional learning-rate scheduler bug in qwen-direct (its LR stayed at 2.9–3.0 × 10⁻⁵ rather than annealing to the 3 × 10⁻⁶ floor) — a reproducible-training anecdote with practical value.

Future Directions

• Token-matched fine-tuning. The steps arms received roughly 10× more fine-tuning tokens than the direct arms (50M vs. 5M). The authors note it remains possible, particularly for the task-locality result, that longer trajectories within the steps-mode set are specifically undertrained relative to short ones.
• A stricter apply harness. Because the ambiguity fallback is a known source of silent errors, the paper suggests a harness that rejects ambiguous search spans outright might show a smaller direct-vs-steps gap.
• Broadening the domain. All results are on Flutter/Dart code editing in self-contained snippets; the authors explicitly make no claim that the ranking generalizes to other languages or to larger multi-file repositories.
• Agentic loops with execution feedback. The steps mode used here has no compiler or test feedback between edits, unlike SWE-bench-style iterative repair agents, so whether feedback changes the picture is untested.
• Human validation of the judge. The blinded gpt-4.1 judge was not independently validated against human raters; the paper leaves a human-agreement spot-check to future work.

Target Audience

Researchers and engineers working on code-generation models, code-editing agents, and LLM training-data design — particularly those deciding between whole-file and diff-style output formats, or building small domain-specific code models. It is also relevant to practitioners who build or evaluate edit-application harnesses, since a substantial portion of the reported failure mode traces back to ambiguous search-span resolution rather than model behavior alone. Readers looking for benchmark-leaderboard style comparisons across many public models will not find them here; the design is deliberately narrow, single-domain, and self-contained.

Authors’ abstract

Large language models used for code editing can be trained and deployed in at least two output regimes: direct generation, where the model emits the entire modified file in one shot, and iterative diff-based generation ("steps"), where the model emits a sequence of localized search/replace edits applied one at a time until it signals completion or a step budget is exhausted. The diff-based regime is attractive because it mirrors how developers edit code and should require far fewer generated tokens per turn. We train two code models - a 100M-parameter model trained from scratch (Rainbow-Pony-100M) and a fine-tuned Qwen2.5-Coder-0.5B - in both regimes on a shared Flutter/Dart dataset, and evaluate all four resulting models on a held-out set of approx 1,790 tasks per model. Direct generation substantially outperforms diff-based generation on every metric we measure - compilation/static-analysis pass rate, bits-per-byte, character-level similarity to the reference, and blinded LLM-judge ratings of goal fulfillment, correctness, and code quality - and the gap persists after controlling for task difficulty via a matched-ID comparison and when restricting to code that compiles on both sides. We then identify a single, architecture-independent mechanism behind the conditions where diff-based generation does win: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in our dataset. We term this task locality and discuss its implications for when an edit-based training regime is and is not the right choice for a code-editing model.