QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code

AI summary

Overview

• Research area: Neural compilation — using Large Language Models (LLMs) to translate LLVM intermediate representation (IR) into hardware-specific assembly code — combined with automatic prompt optimization.
• Technical level: Advanced. Comfort with compilers (IR, register allocation, instruction sets such as x86_64 and aarch64), LLM prompting, and benchmarking methodology is assumed by the paper.
• Scope: The paper introduces NeuComBack, a benchmark for IR-to-assembly neural compilation, establishes baselines for five frontier LLMs on it, and proposes a self-evolving prompt optimization method that learns from the model's own self-debugging traces.

What This Paper Is About

Neural compilation — having a language model directly translate a program representation into assembly — could make it far cheaper to build compiler backends for new hardware and could surface optimizations that rule-based compilers miss. Two obstacles block practical use: there is no dedicated benchmark for the IR-to-assembly task, and LLM-generated assembly is neither reliably correct nor consistently fast. This paper builds such a benchmark (NeuComBack), measures existing models on it, and then shows that letting the model iteratively rewrite its own prompt from its past debugging successes substantially raises both correctness and performance.

Key Contributions

1. A dedicated benchmark for IR-to-assembly compilation. NeuComBack contains two tiers: NeuComBack-L1 (200 tasks selected for the longest LLVM IR from a cleaned set of 1,618 ExeBench test programs) and NeuComBack-L2 (all 151 TSVC cases). The paper states that no publicly available dataset previously targeted IR-to-ASM translation specifically.
2. New performance baselines for frontier LLMs. The authors evaluate GPT-4o, O3-Mini, O1, DeepSeek-V3, and DeepSeek-R1 on translating IR to x86_64 assembly, reporting functional correctness (ACC) and correctness-with-superior-performance (ACC+Perf) against clang -O3.
3. A self-evolving prompt optimization method. A single prompt is progressively refined offline by having the model analyze its complete self-debugging trajectories, extract error patterns and effective fixes, and propose and confirm edits. The method learns from full debugging trails rather than only final failures or successes.
4. Evidence that the approach generalizes. Improvements are reported across two architectures (x86_64, aarch64), two program distributions (NeuComBack-L1, L2), and a cross-distribution prompt-transfer experiment.

Main Findings

• Baseline LLMs struggle badly on the task. On NeuComBack-L2 (151 cases, x86_64), DeepSeek-R1 was the strongest baseline at 45.70% ACC (69/151) and 21.85% ACC+Perf (33/151). GPT-4o reached only 1.99% ACC (3/151) and 0.66% ACC+Perf (1/151); O3-Mini 21.19% (32/151) and 5.30% (8/151); O1 19.87% (30/151) and 5.30% (8/151); DeepSeek-V3 14.57% (22/151) and 3.31% (5/151).
• Learned prompts sharply raise correctness on fundamental compilation. On the NeuComBack-L1 test set (40 samples, x86_64, DeepSeek-R1), ACC rose from 50.00% (20/40) with the baseline prompt to 80.00% (32/40), described as a 60% relative improvement.
• Gains appear at initial generation and compound through optimization. On the NeuComBack-L2 test set (25 samples, x86_64, DeepSeek-R1), the learned prompt raised initial-generation ACC from 44.00% (11/25) to 64.00% (16/25) and initial-generation ACC+Perf from 24.00% (6/25) to 40.00% (10/25). After two rounds of iterative optimization, ACC+Perf rose from 28.00% (7/25) with the baseline prompt to 56.00% (14/25) with the learned prompt — a 100% relative improvement.
• Correct programs are frequently faster than clang -O3. Among the 16 x86_64 programs correctly generated with the learned prompt, 14 (87.5%) were further optimized to surpass clang -O3 performance.
• The approach transfers to a second architecture. On aarch64 (NeuComBack-L2 test set, 25 samples, DeepSeek-R1), Table 4 reports ACC rising from 36.00% (9/25) to 72.00% (18/25) and ACC+Perf from 8.00% (2/25) to 28.00% (7/25), i.e., a 100% relative improvement in correctness and a 250% relative improvement in ACC+Perf. The abstract and conclusion describe the aarch64 correctness gain as rising from 36% to 58%, a figure that differs from the 72.00% shown in Table 4.
• Learned prompts transfer across data distributions. A prompt learned on NeuComBack-L2, applied without further learning to NeuComBack-L1 (x86_64), reached 67.50% (27/40) on the test set and 74.50% (149/200) overall, versus 50.00% (20/40) test and 54.50% (109/200) overall for the default prompt — a 35% relative test-set improvement, though below the 80.00% achieved by a prompt learned on L1 itself.
• Fewer debugging rounds are needed. Average self-debug rounds for eventually-correct programs dropped from 0.90 to 0.28 (x86_64, L1, max 1 round), 1.09 to 0.25 (x86_64, L2, max 2 rounds), and 1.44 to 1.22 (aarch64, L2, max 4 rounds) compared with the baseline.
• Case studies show concrete optimizations. On function s452, the LLM eliminated redundancy by precalculating constants, reducing two paddd (packed doubleword add) instructions to one in the inner loop. On s332, where LLVM -O3 uses sequential comparisons without vectorization, the LLM used cmpps (compare packed single-precision floating-point values) to compare four floats at once and bsfl (bit scan forward) to extract the index from the comparison mask.
• Learned rules fall into three categories. Table 7 illustrates formatting rules (function code exclusively in .text, .size directives immediately after function bodies), syntactic rules (.L prefix with exact spelling for local labels, @PLT suffix on external calls such as call dummy@PLT), and semantic rules (clearing return registers with XORL %eax,%eax for void functions, passing stack-based parameters in reverse order with alignment padding).

Methodology in Plain English

The authors define neural compilation formally as a function that maps a source program — here LLVM IR — plus a target architecture into assembly, subject to two requirements: semantic equivalence with the source, and performance at least as good as a compiler run at -O3. Their workflow starts with an initial assembly generation; if that output is not functionally correct, the process ends. Otherwise the model runs up to T rounds of optimization, with self-debugging available after each generation or optimization step.

To build the benchmark, they took existing C benchmarks and adapted them. For Level 1, they rewrote ExeBench test programs to be C/C++ standard compliant (the original authors had relied on g++ -fpermissive), verified that clang could compile each into LLVM IR, then kept the 200 programs with the longest IR sequences. Level 2 is the full TSVC suite, chosen because its nested loop structures stress register allocation and scheduling. The paper reports that TSVC programs average 26.06 variables versus 13.73 for ExeBench. Correctness is measured as the fraction of cases whose output exactly matches a clang -O0 reference; performance is measured only on Level 2. For TSVC inputs, fixed-value array initializations were replaced with deterministic pseudorandom values under a fixed seed. Each program is executed 11 times, with the first three (warm-up) and last three (cool-down) runs discarded and the median of runs 4–8 reported.

The prompt-learning method has an offline and an online stage. Offline, the model starts from an empty prompt template, runs compilation tasks with self-debugging, and then — using a meta-prompt — analyzes the full trajectories of programs that began incorrect and became correct, extracts error patterns and effective fixes, and proposes edits to the prompt. A review step then confirms only the necessary and beneficial edits. The prompt is updated once per mini-batch of tasks, with one self-debugging round per generation for Level 1 and two for Level 2, over three epochs with a batch size of 5. Online, the evolved prompt drives initial generation, self-debugging to correctness, and iterative optimization, with the same prompt supplied throughout to limit error introduction. Data splits were 120/40/40 (train/validation/test) for L1 and 101/25/25 for L2, with the best prompt chosen by validation performance.

Why This Matters

For research, the paper supplies something the field lacked: a defined task formulation, a benchmark with two distinct difficulty tiers, published baselines for named models, and an evaluation protocol separating correctness from performance. It also reframes prompt optimization for compilation as learning from complete debugging trajectories rather than from isolated failures, and shows that such prompts transfer across architectures and program distributions.

Real-world applications suggested by the work:

• Compiler backends for new instruction sets. The paper's stated motivation is that neural compilation can drastically reduce the time and effort needed to build compilers for emerging ISAs.
• Assisted optimization of generated code. The s452 and s332 examples show LLM-driven removal of redundant instructions and vectorization where a mature compiler did not vectorize.
• Embedded and I/O-heavy code translation. A cited portion of the Level 1 ExeBench programs comes from embedded-systems applications with intensive I/O operations.
• Lowering the expertise barrier in compiler work. The authors frame compilers as systems that demand prohibitively expensive human expertise to develop and maintain; an LLM-assisted path offers an alternative entry point.

Industry relevance centers on cost: if a model can produce assembly that is both correct and competitive with -O3 on a meaningful fraction of cases, that capability bears directly on how much specialist engineering time a new hardware target requires, and on how aggressively a vendor can tune generated code without writing hand-crafted passes.

Future Directions

• Closing the remaining correctness gap. Even the best configuration left a substantial share of programs incorrect, so raising ACC and ACC+Perf further is the authors' stated next goal.
• Extending the benchmark. The authors say future work will expand benchmarks to cover more complex real-world compilation scenarios, and note that extremely complex control flow such as deep recursion and concurrency is underrepresented in current sources.
• Scaling to more architectures and distributions. Only x86_64 and aarch64 are tested, and prompt transfer is demonstrated only from NeuComBack-L2 to L1; broader cross-architecture and cross-distribution transfer is untested.
• Determining what makes evolved prompts work. The paper presents learned rules grouped as formatting, syntactic, and semantic, but does not quantify which rule categories drive the measured gains.

Target Audience

Compiler engineers and hardware architects exploring backends for new ISAs; LLM researchers working on code generation, self-refinement, and automatic prompt optimization; and benchmarking researchers who need a defined IR-to-assembly task with published baselines to compare against. Readers without background in LLVM IR or assembly conventions will find the benchmark description approachable but the case studies and learned rules harder to follow.

Authors’ abstract

Compilers, while essential, are notoriously complex systems that demand prohibitively expensive human expertise to develop and maintain. The recent advancements in Large Language Models (LLMs) offer a compelling new paradigm: Neural Compilation, which could potentially simplify compiler development for new architectures and facilitate the discovery of innovative optimization techniques. However, several critical obstacles impede its practical adoption. Firstly, a significant lack of dedicated benchmarks and robust evaluation methodologies hinders objective assessment and tracking of progress in the field. Secondly, systematically enhancing the reliability and performance of LLM-generated assembly remains a critical challenge. Addressing these challenges, this paper introduces NeuComBack, a novel benchmark dataset specifically designed for IR-to-assembly compilation. Leveraging this dataset, we first define a foundational Neural Compilation workflow and conduct a comprehensive evaluation of the capabilities of recent frontier LLMs on Neural Compilation, establishing new performance baselines. We further propose a self-evolving prompt optimization method that enables LLMs to iteratively evolve their internal prompt strategies by extracting insights from prior self-debugging traces, thereby enhancing their neural compilation capabilities. Experiments demonstrate that our method significantly improves both the functional correctness and the performance of LLM-generated assembly code. Compared to baseline prompts, the functional correctness rates improved from 44% to 64% on x86_64 and from 36% to 58% on aarch64, respectively. More significantly, among the 16 correctly generated x86_64 programs using our method, 14 (87.5%) surpassed clang-O3 performance.