EvolVE: Evolutionary Search for LLM-based Verilog Generation and Optimization

AI summary

Overview

Research area: AI for electronic design automation (EDA) — specifically, using large language models (LLMs) with evolutionary search to generate and optimize Verilog register-transfer-level (RTL) hardware descriptions.

Technical level: Intermediate. Readers should be comfortable with basic LLM concepts (prompting, inference-time search) and have some familiarity with digital hardware terminology (RTL, testbenches, synthesis, PPA).

Scope: The paper introduces EvolVE, an evolutionary search framework that operates at inference time (without fine-tuning) to generate functionally correct Verilog and to optimize it for Power, Performance and Area, together with two new evaluation artifacts — the rectified Mod-VerilogEval v2 and the industry-scale IC-RTL benchmark.

What This Paper Is About

LLMs are trained on sequential text and code, but Verilog describes massively concurrent hardware synchronized by a clock, so directly prompting an LLM often produces functionally incorrect or non-synthesizable designs. Because high-quality Verilog data is scarce and enlarging models yields diminishing returns without more data, the authors shift from a learning-centric to a search-based paradigm: treating hardware generation as state-space exploration where the LLM iteratively self-corrects against formal testbench feedback. The paper's goal is to determine which evolutionary search strategy suits which chip-design task, and to supply the benchmarks needed to measure that, including PPA optimization rather than only functional pass rates.

Key Contributions

1. EvolVE framework. An evolutionary framework offering two search strategies — Idea-Guided Refinement (IGR) and Monte Carlo Tree Search (MCTS) — that lets LLMs iteratively explore the design space without massive domain-specific datasets or prohibitively large models. The authors describe it as the first framework to analyze multiple evolution strategies on chip design tasks, with DeepSeek-R1-FP4 reaching 92% on RTLLM v2 and 98.1% on VerilogEval v2, and Siliconmind-7B reaching 92.3% on the latter.

2. IC-RTL benchmark and Verilog PPA optimization. A new high-complexity suite drawn from the Taiwan National IC Design Contest and expert-crafted custom designs, addressing the scarcity of rigorous PPA evaluation. On it the framework surpasses reference implementations authored by contest participants, reducing the PPA product by up to 66% in Q5 (Huffman Coding) and 17% in the geometric mean across all problems.

3. Structured Testbench Generation (STG). A detachable, automated, high-coverage verification engine that categorizes signals automatically and returns a continuous correctness score, moving the cost of iterative debugging from expensive LLM reasoning to free EDA simulation tools.

4. Mod-VerilogEval v2. A rectified version of VerilogEval v2 (described in Appendix B) created because the authors found inconsistencies in the original benchmark's golden models and problem specifications that degrade evaluation precision. The evolved codebase, Mod-VerilogEval v2, and evaluation scripts are stated to be released shortly, and the IC-RTL source is available at https://github.com/weiber2002/ICRTL.

Main Findings

• Strategy-task match. MCTS excels at maximizing functional correctness, while IGR proves superior for optimization. The two are chosen to target complementary goals: IGR ensures global exploration of the architectural space to escape local optima for PPA optimization, whereas MCTS performs precise exploitation to resolve local functional and timing constraints. Across most architectures, MCTS also showed higher stability and superior peak performance than IGR.

• State-of-the-art functional results. DeepSeek-R1-FP4-MCTS reaches 98.1% on VerilogEval v2 and 92.0% on RTLLM v2 at a 300-node budget. At only 15 nodes it already achieves 94.2% on VerilogEval v2, matching the best reported VerilogCoder result (GPT-4-Turbo with 100 nodes, 94.2%). This exceeds REvolution's 95.5% obtained with Deepseek-V3.

• Base-model uplift. Gemini-2.0-Flash's baseline saturates at 80.1% on VerilogEval v2; evolutionary search raises it to 86.5% with IGR and 87.8% with MCTS. GPT-OSS-120B-IGR reaches 96.8% on VerilogEval v2 and 90.0% on RTLLM v2 versus baselines of 94.2% and 76.0% respectively.

• Scaling rather than rejection sampling. The framework keeps extracting gains as the node budget grows, unlike baselines that plateau. Siliconmind-7B-MCTS climbs from 85.3% at 15 nodes to 92.3% at 300 nodes in Table 1, a monotonic increase the authors attribute to correcting subtle logic errors in hard problems. (The scaling discussion in the text cites 92.9% at 300 nodes, while Table 1 and the contribution list report 92.3%.)

• Sample efficiency of strong models. DeepSeek-R1-FP4 resolves 84.6% of benchmark problems at the very first node and requires an average of only 2.49 nodes to reach a solution.

• STG speeds convergence, not just accuracy. On Mod-VerilogEval v2, the average node count for DeepSeek-R1-FP4 drops from 4.78 to 2.49, and total output token consumption falls by over 60% (from 38,818 to 15,440). At the 300-node ceiling, STG and the unguided variant reach comparable accuracy; the gain is in the search trajectory, latency and inference cost.

• Toolchain speed. Formulating the objective as an Area-Latency product evaluated with open-source Yosys and Icarus Verilog yields a speedup of over 30x compared to commercial workflows such as Synopsys VCS and Design Compiler, while avoiding the need to generate FSDB or VCD files for power evaluation. The resulting optimizations were validated with PrimeTime and the commercial suite.

• PPA outcomes on IC-RTL (IGR with DeepSeek-R1-FP4, synthesized with Synopsys Design Compiler at TSMC 180nm, clock periods swept over 3 to 7 ns).

  • Q1_LBP (Local Binary Patterns): converged to a design matching the manual baseline, with latency and area within 1% of the initial implementation, suggesting the human design already sits at a local optimum.
  • Q2_GEMM (Systolic Array): 12% latency reduction with neutral area and lower power, from rescheduling control logic to improve data reuse and reduce pipeline bubbles.
  • Q3_CONV (Image Convolution): 6% area reduction at fixed latency, but power regressed from 0.45 mW to 0.50 mW, neutralizing the overall PPA gain.
  • Q4_JAM (Job Assignment Machine): 31% area and 26% power reductions without sacrificing latency, a 36% composite PPA improvement.
  • Q5_HC (Huffman Coding): 66% peak PPA improvement, driven by simultaneous reductions in area (30%) and power (the paper text is truncated at this point).

• Why the clock sweep matters. The authors argue that measuring timing slack on a design synthesized with a relaxed clock is meaningless because commercial tools alter circuit topology (standard-cell drive strengths, arithmetic architectures) based on the clock target, and the global PPA optimum rarely coincides with maximum frequency.

• Benchmark gaps confirmed. A recent survey (Yang et al., 2025) cited in the paper notes that existing benchmarks are small-scale (fewer than 100 samples) and focus on basic modules, and that current metrics rely solely on functional pass rates, ignoring PPA.

Methodology in Plain English

The authors treat writing Verilog as a search problem rather than a one-shot generation task. A "node" in the search pairs a candidate Verilog file with a score and diagnostic feedback. Starting from a problem description, the LLM writes an initial design, which is scored against a testbench. From there, the system repeatedly picks a parent node, asks the LLM to improve the code using that node's feedback, and scores the new version.

Two ways of picking and refining parents are studied. Idea-Guided Refinement first asks the LLM to brainstorm k distinct high-level architectural ideas (optionally informed by referenced papers via retrieval-augmented generation), then runs each idea down its own refinement chain, applying Aider-style differential edits instead of regenerating whole files — which saves tokens and keeps consistent state in the context window. Monte Carlo Tree Search instead grows a tree: it selects nodes with an upper-confidence-bound rule that always prioritizes unvisited children first, generates an improved child from the selected leaf, and backpropagates the new score up to the root, incrementally updating visit counts and quality values.

Scoring depends on the task. For generation, the score is simply the fraction of test cases passed, with a large negative constant (−10^5) for compilation or simulation failures. For PPA optimization, only designs passing every test case score positively, and the score is the negative of Area times Latency divided by a scaling factor of 10^5 — a fast proxy that correlates with power while allowing quick open-source simulation.

The feedback text also changes by situation: raw error messages after a failure, a design summary for generation tasks, and optimization guidance plus a design summary for PPA tasks.

Verification runs through Structured Testbench Generation, which does three things. It parses the design's ports with regular expressions and classifies signals into Clock/Reset, Control and Datapath groups based on common industry naming (clk, rst, valid, ready). It then generates stimuli: exhaustive 2^w coverage for control signals of width 8 or less, constrained-random sampling for wider control buses to prevent runtime explosion, and random sampling seeded with corner cases (zero, max value, alternating bits) for datapath buses. Finally, it checks outputs against a golden reference at critical state transitions such as post-reset and shortly after clock edges, producing a continuous pass-rate score between 0 and 1 that acts as a functional gradient so partially correct designs can be prioritized. The generated testbenches follow IEEE 2005 Verilog and work with both Icarus Verilog and commercial simulators like VCS.

Experiments used Siliconmind-7B, GPT-OSS-120B, DeepSeek-R1-FP4 and the proprietary Gemini-2.0-Flash, with a standardized budget of 300 nodes, LLM temperature 0.6, IGR configured for k=60 initial ideas refined over m=5 steps, and MCTS using an expansion rate of 3 children and exploration constant c=1.4. Self-hosted models ran on a cluster of 8 NVIDIA H100 GPUs, with DeepSeek-R1-FP4 using tensor and expert parallelism within its Mixture-of-Experts architecture and Siliconmind-7B using pure data parallelism.

Why This Matters

The work argues that hardware generation does not have to wait for better fine-tuning data or larger models: test-time search can extract capability that a fixed model already contains. It also pushes evaluation beyond functional pass rates into system-level PPA, which is the metric industry actually optimizes, and it supplies the benchmark suite and the open-source simulation path needed to measure it affordably.

Real-world applications:

• Image and signal processing accelerators: the Local Binary Patterns, convolution and distance transform tasks map directly onto vision pipelines, with the convolution task covering zero-padding, 3x3 convolution, bias addition, ReLU and 2x2 max-pooling in Q4.16 fixed point.
• Matrix multiplication engines: the systolic array task targets data reuse, operand broadcast across processing elements, and reduced I/O bandwidth for AI accelerators.
• Combinatorial optimization hardware: the Job Assignment Machine solves an 8-worker/8-job assignment by exploring all 8! (40,320) permutations with a pipelined cost-accumulation path.
• Data compression and coding hardware: the Huffman Coding task requires hardware sorting and tree construction, e.g. parallel sorting networks or priority queue structures, within limited clock cycles.

Industry relevance: the paper targets the labor-intensive, expertise-heavy RTL design cycle and reports results validated on commercial Synopsys tooling (VCS, Design Compiler, PrimeTime) at the TSMC 180nm node, while keeping the optimization loop runnable on free open-source tools. Its benchmarks are derived from the Taiwan National IC Design Contest, and it compares against reference implementations authored by contest participants rather than only synthetic toy modules — addressing the survey-cited gap of benchmarks under 100 samples with no PPA dimension.

Future Directions

• Extending and diversifying IC-RTL. The benchmark currently spans a small set of hand-crafted tasks plus contest problems; scaling it to broader industrial design classes is a natural next step the paper's framing invites.
• Deciding when to use IGR versus MCTS. The paper establishes MCTS for correctness and IGR for optimization, but does not report a hybrid or automatic selector that picks a strategy per problem during a single run.
• Closing the PPA gap on tasks where the optimizer failed to win. Q1_LBP hit a 1% band around the human baseline and Q3_CONV's 6% area saving was canceled by a power regression, so the conditions under which search beats human design are still only partially characterized.
• Broadening model coverage and releasing artifacts. Results are reported for four models; the EvolVE codebase, Mod-VerilogEval v2 and evaluation scripts are stated to be released shortly, which will let others test whether the scaling behavior generalizes. The paper does not report power figures beyond the truncated Q5 discussion, nor does the available content report hardware measurement on silicon for the optimized designs.

Target Audience

This paper suits graduate students and researchers working at the intersection of LLMs and electronic design automation, EDA and RTL engineers curious about automated design-space exploration, and practitioners evaluating whether inference-time search can substitute for domain-specific fine-tuning. Benchmark designers and competition organizers will also find the IC-RTL and Mod-VerilogEval v2 contributions relevant, as will anyone assessing the cost-efficiency tradeoffs of commercial versus open-source verification loops.

Authors’ abstract

Verilog's design cycle is inherently labor-intensive and necessitates extensive domain expertise. Although Large Language Models (LLMs) offer a promising pathway toward automation, their limited training data and intrinsic sequential reasoning fail to capture the strict formal logic and concurrency inherent in hardware systems. To overcome these barriers, we present EvolVE, the first framework to analyze multiple evolution strategies on chip design tasks, revealing that Monte Carlo Tree Search (MCTS) excels at maximizing functional correctness, while Idea-Guided Refinement (IGR) proves superior for optimization. We further leverage Structured Testbench Generation (STG) to accelerate the evolutionary process. To address the lack of complex optimization benchmarks, we introduce IC-RTL, targeting industry-scale problems derived from the National Integrated Circuit Contest. Evaluations establish EvolVE as the new state-of-the-art, achieving 98.1% on VerilogEval v2 and 92% on RTLLM v2. Furthermore, on the industry-scale IC-RTL suite, our framework surpasses reference implementations authored by contest participants, reducing the Power, Performance, Area (PPA) product by up to 66% in Huffman Coding and 17% in the geometric mean across all problems. The source code of the IC-RTL benchmark is available at https://github.com/weiber2002/ICRTL.