Pareto-Optimal RTL Code Generation via Multi-Objective Reinforcement Learning with Large Language Models

AI summary

Overview

Research area: AI for electronic design automation (EDA), specifically LLM-based generation of register-transfer level (RTL) hardware descriptions, trained with reinforcement learning with verifiable rewards (RLVR) and extended to multi-objective (Power, Performance, Area) optimization.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (GRPO), Hypervolume-based multi-objective optimization, and the RTL-to-PPA tool flow (Icarus Verilog, Yosys, OpenROAD).

Scope: The paper proposes a training pipeline and reward function that make an LLM generate a set of RTL circuits spanning diverse PPA trade-offs rather than optimizing a single scalarized objective, and evaluates it on the RTLLM v2 benchmark with two open base models.

What This Paper Is About

Existing RLVR methods for RTL generation collapse power, delay, and area into one scalar reward, so a trained model tends to produce one kind of circuit rather than a spread of trade-off options. Because real designers weigh power, latency, and area differently depending on product requirements, the authors argue the goal should be a diverse, high-quality candidate set from which a preferred trade-off can be picked after generation. Their goal is to train an LLM so that the entire set of circuits it produces across multiple sequential generations covers the PPA Pareto front well.

Key Contributions

1. A set-level problem formulation. The authors state they are the first to propose a learning method for LLM-based circuit generation that optimizes the quality and diversity of the whole circuit set obtained through multiple generations in the PPA objective space, rather than the quality of a single circuit.

2. A generation-based training pipeline. Coverage improvement of an entire Pareto solution set is reformulated as a sequential decision problem: the model takes a specification plus the history of previously generated circuits and their PPA values, and generates the next circuit to add. This makes the problem trainable with GRPO.

3. A Hypervolume Contribution (HVC) reward. Each generated circuit is rewarded by how much Hypervolume it adds to the existing circuit set (relative to the origin as reference point), after testbench verification and PPA estimation, which explicitly encourages new candidates to expand the Pareto front.

4. A new evaluation protocol and metric. The paper defines relative Hypervolume (rHV), inspired by NHGR, using a per-problem pseudo Pareto front built from the non-dominated union of all models' and all trials' solutions, alongside Win Rate and a circuit-diversity count, and reports gains on RTLLM v2.

Main Findings

• rHV gains on VeriReason-3B: At iter=10 the proposed Sequential HVC reaches 32.4% rHV versus 19.1% for the unmodified Base model (+13.3 percentage points) and 18.0% for Single-Objective PPA (+14.4 percentage points). Intermediate iterations: 22.9% at iter=1 and 30.7% at iter=5, versus Base's 13.1% and 17.5%.

• rHV gains on CodeV-R1-Distil-7B: At iter=10 the proposed method reaches 43.2% versus Base's 38.4% (+4.8 points) and Single-Objective PPA's 37.8% (+5.4 points). At iter=1, however, it scores 26.7%, below Sequential TB (28.4%) and Sequential TB+PPA (27.7%), though above Base (23.1%) and Single-Objective PPA (23.5%). Its advantage appears at iter=5 (40.6%) and iter=10.

• Absolute rHV remains low: Even the best configuration reaches 43.2% of the reference archive's Hypervolume, meaning the generated sets still cover less than half of the normalized pseudo-front — a point the authors do not dwell on but which the table shows.

• Win Rate dominance and generalization past the training horizon: The proposed method has the highest Win Rate at almost all iterations on both base models; the only exception is iter=1 for CodeV-R1-Distil-7B, where Sequential TB is marginally higher. The lead widens with more iterations and holds up to iter=10, even though training only used circuit histories of up to length 5.

• Reward hacking in scalar-reward sequential training: Sequential TB and Sequential TB+PPA decline in Win Rate as iterations increase and end with the lowest Win Rate at later iterations on both base models. The diversity results show why: they frequently produce only a single unique circuit per trial, i.e., the model regenerates circuits it already produced.

• Diversity ranking: Sequential HVC generated the most diverse circuits on both base models. Base and Single-Objective PPA sit in the middle, with their relative order depending on the base model. The authors note an ablation adding an explicit repetition penalty to the sequential baseline is provided in Appendix E.4 (the appendix content is truncated in the supplied text).

• Single-objective PPA training can hurt: On VeriReason-3B, Single-Objective PPA scores below the untrained Base model at every reported iteration (10.4% vs 13.1%, 15.4% vs 17.5%, 18.0% vs 19.1%), suggesting scalarized PPA rewards can degrade set-level coverage.

Methodology in Plain English

The authors take an existing RTL-generating LLM and fine-tune it with GRPO, but change what the model sees and what it is rewarded for.

What the model sees. Instead of just the natural-language spec, the prompt also contains a "circuit history": previously generated circuits for that spec together with their measured PPA. This lets the model see which regions of the trade-off space are already occupied.

How a reward is computed. For a spec, the model samples a group of candidate Verilog designs. Each is simulated with Icarus Verilog against its testbench. Designs that fail get reward 0. Designs that pass get 1.0 plus a Hypervolume Contribution term. Passing designs are pushed through Yosys for logic synthesis and OpenROAD for PPA estimation, the raw values are normalized against the ground-truth ("golden") circuit's PPA so that 1.0 means matching the reference and above 1.0 means beating it, and the HVC is the increase in Hypervolume when the candidate is added to the set of circuits generated so far. HVC rewards are clipped to [0, 3.0] because of high variance.

Dataset reconstruction between rounds. After a training round, the authors record every generated circuit with its PPA, its HVC, and the history it was conditioned on. For each original sample (same spec and same history), they keep only the candidate with the highest HVC, append it to that sample's history, and add the resulting entry to the next round's training set. Samples whose history reaches length 5 are dropped, and at most 1,000 samples per history length are kept, with the oldest removed first. The original history-free dataset is always added back. Over rounds, the training set therefore contains histories of length 0, 1, 2, and so on, teaching the model to iterate.

Evaluation. At test time, the model generates a circuit, evaluates it, appends it to the history, and repeats for up to 10 iterations per problem. Sampling uses temperature T = 0.5 with multiple trials per problem. Each problem's reference front is the non-dominated set of the union of all solutions from all models and all trials, min–max normalized per objective; rHV is the Hypervolume of a method's normalized set divided by the reference archive's Hypervolume.

Experimental controls. Two base models with different training recipes are used to show the effect is not base-model-specific: VeriReason-3B (Qwen2.5-Coder-3B-based) and CodeV-R1-Distil-7B (Qwen2.5-Coder-7B-Instruct-based). Five conditions are compared: Base, Single-Objective PPA, Sequential TB, Sequential TB+PPA, and Sequential HVC. Training data came from VeriReason's hard level (1,149 problems): 889 problems whose ground-truth passed verification and PPA estimation were used for Single-Objective PPA, and 78 problems showing negative correlation (r < 0) between area and delay were used for the generation-based methods, since degenerate Pareto fronts give weak multi-objective signal. Evaluation used 45 of the 50 RTLLM v2 problems where verification and PPA estimation succeeded.

Why This Matters

Impact on research. The paper reframes RTL generation RLVR from "make one good circuit" to "make a good set of circuits," and shows that a set-level metric (Hypervolume) can be used directly as a per-sample reward. It also documents a concrete failure mode — scalar rewards plus sequential generation collapse into regenerating the same circuit — that is relevant to anyone doing multi-turn or history-conditioned RL for code or design.

Real-world applications:

• Product-specific trade-off selection: a design team can generate one candidate pool per specification and pick low-power, low-latency, or small-area variants depending on the product's requirements, instead of retraining per objective weighting.
• Upstream design-space exploration: diverse RTL candidates can be screened before committing to logic synthesis and physical design, which the paper notes can take minutes to hours per candidate.
• Seed pools for evolutionary search: the authors connect diverse high-quality RTL to methods like AlphaEvolve, which keep a database of prior programs as parents and inspiration, and cite its finding that retaining candidates that are good under different criteria stimulates more diverse generation.
• Early RTL-stage optimization: the paper cites the argument that manipulating RTL directly moves optimization earlier in the design flow, before costly downstream tools are invoked.

Industry relevance. The work comes from Hitachi's Research & Development Group and targets a practical bottleneck in industrial HDL design flows. It uses standard open tools (Icarus Verilog, Yosys, OpenROAD) and two publicly available base models, so the pipeline is reproducible without proprietary infrastructure — though the small 78-problem training set limits how far the results generalize (see below).

Future Directions

• Scaling beyond a single benchmark and 78 training problems. The authors explicitly flag that evaluation is limited to RTLLM v2 and that generation-based training used only 78 problems; verifying on other benchmarks, more diverse specifications, and larger datasets is stated as future work.
• Higher-dimensional objective spaces. The method was only verified with 3 objectives (P, P, A). Because Hypervolume must be computed for each rollout during training, handling the added computational cost as the number of objectives grows is described as an open challenge.
• Longer histories under context limits. Input length caps how much circuit history can be shown, so problems with many Pareto-optimal solutions cannot have their full history supplied, making full enumeration difficult; more compact history representations would address this.
• Transfer beyond circuits, and resisting reward hacking. The authors note application to other engineering fields such as control engineering is unverified, and their results suggest further work on reward designs that keep exploration alive over long generation horizons — the repetition-penalty ablation in the truncated appendix is a first step in that direction.

Target Audience

Researchers and engineers working at the intersection of LLMs and hardware design — particularly those building RLVR or GRPO pipelines for Verilog/RTL generation, and EDA practitioners interested in multi-objective design-space exploration. It is also useful for multi-objective RL researchers curious about applying Hypervolume-based rewards to a real sequential generation task, and for ML engineers who want a worked example of how scalarized rewards cause mode collapse in history-conditioned generation. Readers should already be comfortable with policy-gradient methods and Pareto/Hypervolume concepts; the paper does not tutorialize these.

Authors’ abstract

Recent studies have proposed reinforcement learning with verifiable rewards (RLVR) methods that enable LLMs to generate high-quality RTL circuits from natural-language specifications. Existing approaches optimize circuit PPA (Power, Performance, Area) by aggregating multiple objectives into a single scalar reward, making it difficult to simultaneously generate diverse Pareto-optimal solutions across different trade-offs such as low power, low latency, and small area. However, in practical design, the relative importance of each objective varies depending on product requirements and design constraints; thus, obtaining a diverse and high-quality candidate set is beneficial in that designers can choose a preferred trade-off a posteriori. In this work, we are the first to propose a learning method for LLM-based circuit generation that optimizes, not the quality of a single circuit, but the quality and diversity of the entire circuit set obtained through multiple generations in the PPA objective space. The proposed method consists of (1) a generation-based training pipeline that reformulates coverage improvement of the entire Pareto solution set into a sequential circuit generation problem, and (2) a reward function based on Hypervolume Contribution (HVC) that measures how much a generated circuit contributes to the Pareto front. Evaluation on the RTLLM v2 benchmark shows that the proposed method achieves +13.3 percentage points in relative Hypervolume over a baseline without PPA optimization and +14.4 percentage points over a single-objective reinforcement learning method, while also demonstrating superiority in circuit diversity.