CROCODIL: Cross-Model Code Editing with LLMs

AI summary

Overview

Research area: Natural language processing, specifically large language model (LLM) post-training for automated code editing and repair.

Technical level: Intermediate. Readers should be comfortable with LLM reinforcement learning (policy, reward, GRPO) and standard code-editing evaluation (edit distance, build/test pass rates), though the paper explains each concept plainly.

Scope: The paper measures how five LLMs edit code written by other LLMs versus their own code, and introduces Crocodil, a reinforcement-learning post-training framework that reduces excessive edits on foreign code without losing functional correctness.

What This Paper Is About

Development teams frequently use several different LLM coding assistants, so the code any one model is asked to edit was often originally written by a different model with a different coding style. The authors ask whether this "cross-editing" situation causes models to make more changes than necessary, and they show that it does: models overwrite foreign code far more aggressively than their own.

The paper then proposes Crocodil (Cross-model Code Editing with LLMs), a training method that rewards edits for both similarity to the original code and successful builds and tests, so the model learns to make smaller but still working edits.

Key Contributions

1. A new data-collection framework for cross-model evaluation. The authors gather pre-edit and post-edit function pairs from merged GitHub pull requests on popular Rust crates, together with the tests that exercise those functions, and then use several different LLMs to independently re-implement the pre-edit code. This lets them vary both the author (implementor) of the pre-edit code and the model performing the edit, which prior code-editing benchmarks such as Can It Edit, EDIT-Bench, SWE-agent, and Agentless do not support.

2. Empirical evidence that LLMs over-edit foreign code. Using five LLMs (Qwen3.5 35B A3B, GPT-OSS 20B, Olmo3 7B, Olmo3.1 32B, and Haiku 4.5), the authors find that the four open-weight models edit their own implementations up to 14% less than they edit other models' implementations, on 14 of 16 model pairings.

3. Crocodil, a post-training framework with a composite reward. Crocodil combines a similarity reward that penalizes large edits with an execution reward that scores build and test success, multiplied together and optimized with Group Relative Policy Optimization (GRPO).

4. Evidence that prompt engineering is not enough. A hand-written "strict" system prompt that asks for minimal edits cannot consistently shrink edits or improve edit success, while Crocodil roughly halves edit distances and improves build and test pass rates on foreign code.

Main Findings

• Models edit foreign code more than their own. Among 16 self-versus-cross comparisons, 14 show self-editing producing the smaller change, and 7 of these are statistically significant at p < 0.05 by a one-sided Mann-Whitney U test. The two exceptions are both Olmo3 7B, whose self-editing change is larger than its cross-editing change on Olmo3.1 32B's and Haiku 4.5's implementations.

• Olmo3 7B is the most aggressive editor, editing about twice as much as the other models on implementations by Qwen3.5 35B A3B and GPT-OSS 20B. That gap shrinks substantially on Olmo3 7B's own implementations.

• Haiku 4.5 breaks the pattern. The one closed-source model shows little difference between editing its own code and foreign code; its smallest normalized edit distance falls on GPT-OSS 20B's implementations rather than its own. The authors leave the reason as an open question.

• The corpus is substantial. From Rust crates on GitHub the authors collected 4,514 pull requests and 8,235 functions across 356 repositories, with usable tests for 2,458 functions (1,404 PRs, 207 repositories). The collected tests execute 87.9% of the lines of the developer's post-edit function on average, with a median of 94.5%.

• Crocodil shrinks edits. On the "all tasks" columns, Crocodil halves Olmo3 7B's normalized edit distance on functions implemented by each of the four open-weight implementors. Restricting to tasks where both the base model and the Crocodil-trained model pass every test, Crocodil still edits less on every implementor, showing the reduction is not simply a side effect of failing more tasks.

• Crocodil improves success on foreign code. The Crocodil-trained Olmo3 7B outperforms both the untrained base model and the strict-prompt variant on build rate, all-tests pass rate, and mean fraction of tests passed for every implementor other than Olmo3 7B itself. For example, on Haiku 4.5 implementations the build rate rises from 32.67% (base) and 35.14% (strict) to 45.75% (RL), and the all-tests rate from 12.36% and 19.31% to 20.85%.

• The self-edit exception is expected and mitigable. Crocodil is not expected to improve self-editing success because the model already makes few excessive edits on its own code. The authors note a user could unload the LoRA adapter and fall back to the base model for that case.

• The build advantage is not just refusing to edit. A noticeable portion (19.94%) of the Crocodil model's outputs leave the implementation unchanged. After excluding those and looking only at tasks where the model actually modifies the code, the build rate is still higher than the base model's for every implementor except Olmo3 7B.

• Qualitative failures reveal a trade-off. In cases where only the Crocodil model succeeds, the base model tends to add out-of-scope code (for example hallucinating an enum Architecture with invalid syntax in a function named endianness), while Crocodil stays inside the requested match block. But the similarity penalty can also cause under-editing: in a function named cooked_byte_string, Crocodil failed to propagate an identifier rename from LexError to Reject, which the base model handled.

Methodology in Plain English

The authors build their evaluation set from small merged pull requests on Rust projects, breaking each PR into single-function updates and keeping only functions that exist both before and after the PR and are between 20 and 200 non-empty, non-comment lines. They gather context that a model could realistically use — function signature, the rest of the PR diff, callee functions, call sites, use statements, the enclosing impl and struct headers, and LLM-generated descriptions of the repository, the PR, and the function — and they collect tests from three commits (base, head, and the last main-branch commit where the function still matches the post-edit form), filtered by line coverage so only tests that actually exercise the function survive.

Each of the five LLMs then acts as an implementor, rewriting each developer function from context alone (never seeing the original code). Drafts that fail to build or test go through an iterative repair loop, where one agent writes a natural-language repair plan and a second agent applies it, up to 6 rounds (2 rounds for Haiku 4.5 to bound API cost). Implementations are discarded if they overlap more than 50% with the original developer code, if they already pass every post-edit test, or if their context exceeds the relevant token window.

Every model then acts as an editor, applying the original PR's change to every implementation, including its own. Edit size is measured as character-level Levenshtein distance after both sides are run through rustfmt and stripped of comments and blank lines, then min-max normalized within each task across the five editors. Each editor is paired against each other implementor with a one-sided Mann-Whitney U test.

For training, the same pipeline is run on a separate pool of repositories with no project-level overlap, yielding 355 cross-edit tasks for training and 39 for validation. The reward is the product of two terms: a similarity reward that shrinks as the model's change magnitude (characters changed times lines changed) grows relative to the developer's own change magnitude, and an execution reward that blends build success (weight 0.2), regression tests the implementation already passed (weight 0.4), and new post-edit tests (weight 0.4). Olmo3 7B is trained with LoRA (rank 32, alpha 64) using the VERL framework for 3 epochs, batch size 32, 8 rollouts per prompt, temperature 1.0, learning rate 3×10⁻⁵, and KL coefficient 0.001.

Why This Matters

Impact on research. The paper identifies a failure mode — excessive editing of code written by a different model — that existing code-editing benchmarks structurally cannot measure, because they vary the editor but not the author of the pre-edit code. It also shows that a reward combining a similarity term with an execution term can change model behavior in a way prompt engineering cannot, offering a template for training models to be conservative editors rather than just capable ones.

Real-world applications:

• Mixed-tool development teams where developers use different LLM assistants and pass code between them.
• Code review workflows, since fewer changes per pull request means faster review and fewer tests to run for that change.
• Agentic coding pipelines that chain models, where one model's output becomes another model's input.
• Building local adaptation layers: because Crocodil uses a LoRA adapter over frozen base weights, a team could load the adapter only for cross-model editing tasks and fall back to the base model for self-edits.

Industry relevance. The problem is directly caused by the way real teams adopt LLMs — different developers pick different assistants, and the same developer switches models as latency, cost, or task demands shift. Low success rates on this benchmark (build rates in the 28–46% range across all rows, which the authors attribute to the small base model rather than the training) also indicate how much room remains before LLM code editing is reliable on unfamiliar code.

Future Directions

1. Extending beyond Rust. The corpus and training data come exclusively from Rust crates on GitHub. The authors explicitly state they have not verified that the self-edit/cross-edit gap or Crocodil's reward generalizes to languages with weaker typing or different grammar, such as Python or JavaScript.

2. Scaling past function-level edits. Each PR was decomposed into per-function tasks of 20–200 lines. Edits that span multiple functions or multiple files are outside the scope of this work and Crocodil's behavior there is not studied.

3. Broadening the model pool and training targets. Due to budget constraints the study uses five LLMs, only one closed-source, and only Olmo3 7B was trained with Crocodil. The authors note that larger closed-source models such as GPT-5 and Claude Opus are not evaluated and that the framework is not tested on them.

4. Understanding Haiku 4.5's behavior. The paper notes that why the closed-source model shows little difference in edit distance between its own code and foreign code is an interesting question that it leaves as an exciting direction for future work.

5. Taming the under-editing failure mode. The qualitative analysis shows the similarity penalty can make the model refuse seemingly minor but necessary edits, especially propagating identifier renames. Reducing that without losing the edit-size gains is an open problem the paper surfaces but does not solve.

Target Audience

This paper is most useful to researchers and engineers working on LLM-based code generation, code editing, and agentic software engineering; to practitioners applying reinforcement learning post-training to code models; and to teams that deploy multiple LLM coding assistants and want to understand how those models interact when they hand code to one another. Readers evaluating code-editing benchmarks will also find the cross-model evaluation setup relevant, since it exposes an axis those benchmarks do not currently vary.

Authors’ abstract

Large language models (LLMs) have become ubiquitous tools for code generation and editing. However, development teams often use multiple LLM assistants. Different developers may prefer different models, and individual developers may switch between models across different coding sessions. Because of this, the edits any one model makes are frequently applied to foreign code originally generated by another model. These LLMs are often trained on different datasets, and as a result have different stylistic preferences. Do LLMs behave differently when they edit foreign code originally written by a different LLM with a different coding style? We find that models tend to make more, and often excessive, edits on foreign code. We introduce CROCODIL (Cross-model Code Editing with LLMs), a post-training framework for reducing excessive edits while preserving functional correctness. CROCODIL's similarity reward penalizes large changes, while its execution reward scores build and test success. We use the product of these two rewards to encourage the policy to decrease the edit size without decreasing the edit task success rate. CROCODIL is available at https://github.com/EngineeringSoftware/Crocodil.