QiMeng-SALV: Signal-Aware Learning for Verilog Code Generation

AI summary

Overview

• Research area: Machine learning for electronic design automation (EDA), specifically LLM-based Verilog (hardware description language) code generation, with a focus on reinforcement learning from preference data.
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning, Direct Preference Optimization (DPO), abstract syntax trees, and digital hardware concepts such as signals, wires, and gates.
• Scope: The paper introduces a signal-level reinforcement learning method that extracts functionally correct signal implementations from partially incorrect Verilog modules to provide denser, more meaningful training rewards.

What This Paper Is About

Large language models can write Verilog code, but making them produce functionally correct circuits is hard because there is no reliable way to reward them for correctness during training. Existing reward signals either compare code structure (which fails when two different implementations are both correct) or require an entire module to be right before any learning happens. This paper solves that by recognizing that Verilog signals are largely independent of each other, so even a broken module can contain correctly implemented signals that are worth learning from.

Key Contributions

1. Signal-level reinforcement learning for Verilog. The first fine-grained RL algorithm that operates on individual hardware signals rather than whole modules, shifting the optimization granularity from module-level to signal-level.

2. Automated signal-aware verification without testbenches. A pipeline using Yosys to generate random input stimuli and compare output signals between generated and reference modules, producing a preference dataset even when no testbench exists in the training data.

3. AST-based signal code extraction. A method that builds a signal dependency topology graph from the abstract syntax tree and performs backward traversal from target output signals to isolate the exact code segments responsible for each signal.

4. Signal-aware DPO loss. A modified DPO objective that computes token probabilities only over code segments belonging to contrasting signals, filtering out noise from incorrect signals in otherwise useful samples.

Main Findings

• State-of-the-art open-source results. QiMeng-SALV achieves 62.6% pass@1 and 75.1% pass@5 on RTLLM v1.1, and 81.4 pass@1 on the VerilogEval1.0 Machine subset — the best among open-source models.

• 7B parameters match a 671B model. The 7B-parameter QiMeng-SALV matches DeepSeek-v3 (671B parameters) on RTLLM v1.1 (62.6% vs 62.0% pass@1) and surpasses it on pass@10 (81.1% vs 72.4%).

• Large gain over same-data SFT baseline. Compared to CodeV (Qwen2.5), trained on the identical dataset with the same base model, QiMeng-SALV improves pass@1 by roughly 59.7% on RTLLM v1.1.

• Outperforms module-level RL. Against VeriPrefer, which uses module-level functional rewards, QiMeng-SALV wins on every reported metric, confirming that signal-level supervision extracts more learning signal from the same data.

• Filtering incorrect signals is essential. Ablations show that naively mixing partially correct modules into standard DPO degrades performance (55.3% pass@1 vs. 57.4% for fully correct data alone), but adding the signal-aware filter raises it to 62.6%.

• Training stages compound. SFT adds roughly 15 percentage points over the base model, and signal-aware DPO adds another 10–14 points on top.

• Scaling rewards helps. Performance rises consistently as the signal-aware preference dataset grows, indicating that more functional rewards translate directly into better models.

• Negligible computational overhead. Simulation takes 0.0391 s and AST parsing 0.0957 s per sample on a single core, dropping to 0.65 ms and 1.59 ms respectively when parallelized across 60 CPU cores.

Methodology in Plain English

The approach starts with an ordinary LLM fine-tuned on 135k cleaned Verilog samples from GitHub (the CodeV dataset, reduced from 165k after filtering out syntactically invalid modules). For each design prompt, this model generates five candidate module implementations.

Step 1 — Verification. The tool Yosys reads the reference module from the training data and auto-generates many random input signal streams. These same inputs are fed to both the generated candidates and the reference module, and output signals are compared. A signal counts as correct only if it matches the reference across every input set. This produces pairs where one module has a correct signal that the other module gets wrong.

Step 2 — Extraction. Each module is parsed into an abstract syntax tree. The system builds a graph of which signals depend on which other signals, then walks backward from a target output signal to find all the intermediate signals it needs. The corresponding code lines are pulled out, giving a clean code segment for the preferred module and a matching segment for the dispreferred one.

Step 3 — Signal-aware DPO training. Standard DPO would push up the probability of the entire preferred module and push down the entire dispreferred module. Since preferred modules here may still contain errors, the authors instead compute the loss only over the tokens in the extracted contrast-signal code segments. The rest of the module is ignored. This prevents wrong signals from contaminating the gradient while still letting the model learn from the correct piece.

Training uses the Qwen2.5 Coder Instruct 7B base model, two epochs of full-parameter SFT followed by about 7,000 steps of LoRA-based signal-aware DPO. Evaluation uses pass@k on the VerilogEval and RTLLM benchmarks.

Why This Matters

Research impact. This work reframes reinforcement learning for code generation as a granularity problem rather than a reward-modeling problem. It shows that the bottleneck in RL for hardware code is not the absence of correct samples, but the failure to look inside incorrect samples. The signal-independence property of Verilog is what makes this possible, and the same logic may extend to other domains where outputs decompose into independently verifiable units.

Real-world applications:

• Automated chip design assistants. Engineers can generate and refine register-transfer level (RTL) modules faster, with the model learning from near-misses rather than requiring perfect examples.
• Verification-aware EDA toolchains. The signal verification stage can be plugged into existing design flows to give per-signal correctness feedback during development.
• Hardware design education. Students and instructors get a tool that flags which specific signals in a design are wrong, not just whether the whole module fails.
• Low-resource hardware teams. A 7B open-source model matching a 671B proprietary model means smaller organizations can run competitive HDL generation locally.

Industry relevance. RTL design is a major bottleneck in semiconductor development. Reducing iteration cycles with reliable LLM assistance has direct commercial value, and the fact that this runs locally on modest GPUs makes it deployable inside companies with strict IP constraints.

Future Directions

• Dependence on reference quality. The method assumes the reference modules in the training set are correct. If the dataset contains wrong references, the verification step produces wrong labels. Building noise-robust verification or automated reference repair is an open problem.
• Extending to sequential and stateful logic. The approach relies on the relative independence of signals, which holds best for combinational logic. Modules with complex clocking, reset behavior, or state machines may need a different decomposition strategy.
• Broader signal-level objectives. The current work builds signal-aware DPO specifically. Exploring signal-level variants of GRPO or PPO, or combining signal-level and module-level rewards, could yield further gains.
• Generalization beyond Verilog. The underlying idea of extracting correct sub-components from incorrect whole outputs may transfer to other structured code generation tasks, such as SQL queries, program synthesis, or multi-file software generation.

Target Audience

This paper is best suited for machine learning researchers working on reinforcement learning from human or verifier feedback, EDA and hardware verification engineers interested in LLM-assisted design, and graduate students with background in both deep learning and digital circuit design. Readers should already understand DPO, transformer language models, and basic Verilog semantics to follow the technical details. Practitioners seeking a deployable Verilog generation model will also find the empirical results and open-source release directly useful.

Authors’ abstract

The remarkable progress of Large Language Models (LLMs) presents promising opportunities for Verilog code generation which is significantly important for automated circuit design. The lacking of meaningful functional rewards hinders the preference optimization based on Reinforcement Learning (RL) for producing functionally correct Verilog code. In this paper, we propose Signal-Aware Learning for Verilog code generation (QiMeng-SALV) by leveraging code segments of functionally correct output signal to optimize RL training. Considering Verilog code specifies the structural interconnection of hardware gates and wires so that different output signals are independent, the key insight of QiMeng-SALV is to extract verified signal-aware implementations in partially incorrect modules, so as to enhance the extraction of meaningful functional rewards. Roughly, we verify the functional correctness of signals in generated module by comparing with that of reference module in the training data. Then abstract syntax tree (AST) is employed to identify signal-aware code segments which can provide meaningful functional rewards from erroneous modules. Finally, we introduce signal-aware DPO which is optimized on the correct signal-level code segments, thereby preventing noise and interference from incorrect signals. The proposed QiMeng-SALV underscores the paradigm shift from conventional module-level to fine-grained signal-level optimization in Verilog code generation, addressing the issue of insufficient functional rewards. Experiments demonstrate that our method achieves state-of-the-art performance on VerilogEval and RTLLM, with a 7B parameter model matching the performance of the DeepSeek v3 671B model and significantly outperforming the leading open-source model CodeV trained on the same dataset. Our code is available at https://github.com/QiMeng-IPRC/QiMeng-SALV.