SLIC: Reinforcement Fine-Tuning Small LMs for Multi-Turn Analog Circuit Optimization

AI summary

Overview

Research area: Machine learning for electronic design automation (EDA), specifically reinforcement learning with verifiable rewards (RLVR) applied to analog integrated circuit sizing and optimization.

Technical level: Advanced. The paper assumes familiarity with policy-gradient/GRPO-style reinforcement learning, circuit simulation, and analog design metrics such as gain, bandwidth, and phase margin.

Scope: The paper introduces SLIC, a framework that fine-tunes a 4B-parameter language model with multi-turn, simulator-grounded reinforcement learning so it can iteratively tune analog amplifier circuits locally, without sending proprietary design data to hosted LLM services.

What This Paper Is About

Analog IC design is an iterative, expert-driven task where an engineer repeatedly adjusts transistor dimensions, simulates the circuit, and reasons about coupled trade-offs among gain, power, bandwidth, and phase margin. Foundries restrict sharing process design kits (PDKs), so hosted large-model workflows are often impractical, and off-the-shelf LLMs perform poorly on this multi-turn task. The paper's goal is to train a small (4B-scale), locally deployable model that can perform closed-loop circuit optimization directly against a circuit simulator, and to release an open benchmark for evaluating such models.

Key Contributions

1. A locally deployable, reasoning-driven training framework for analog design. SLIC is trained entirely on simulator-grounded synthetic data generated from public 65 nm CMOS predictive technology models (PTM), avoiding proprietary PDK data. It pairs structured XML circuit representations with discretized relative parameter-update actions, so the model proposes interpretable edits rather than opaque absolute values.

2. Domain-specific adaptations for simulator-interactive RL. The authors combine domain-validity and success-gated multi-objective rewards (including a success-conditional slack bonus) with Reward-Guided Rollout Steering (RGRS), which samples multiple candidate edits per turn, simulates each, uses one selected successor to advance the trajectory, and keeps all candidates for group-relative policy updates.

3. A 4B policy that beats a 763B baseline. Trained on PTM 65 nm, SLIC-Random reaches 87.1% in-distribution four-specification Pass@1 within eight turns, 48.5 percentage points above DeepSeek-V4.1-Flash Think (763B), and exceeds GPT-6-Astra at the four-turn horizon (74.9% vs. 67.1%).

4. An open, goal-conditioned benchmark. The paper releases 41 amplifier topologies with target specifications and initial states, plus standardized protocols for in-distribution validation, held-out topology evaluation, and cross-technology-node transfer spanning 22–130 nm CMOS.

Main Findings

• Large gains over the teacher model: DeepSeek-V4.1-Flash Think generated the SFT data, yet the 4B student starts below it after SFT (26.5% vs. 38.6% at eight turns, in-distribution) and ends well above it after RL (87.1%). DeepSeek reaches 21.2% at four turns and 38.6% at eight turns in-distribution; on held-out topologies it reaches 14.9% and 26.7%.

• Competitive with a strong closed-source model: On 840 in-distribution tasks, GPT-6-Astra scores 67.1% at four turns and 90.6% at eight turns; SLIC-Random scores 74.9% and 87.1%, a 3.5-point gap at eight turns. On 240 held-out-topology tasks, GPT-6-Astra reaches 63.7% and 86.2%, while SLIC-Softmax reaches 72.6% at eight turns, 45.9 points above DeepSeek.

• Continuation policy matters: SLIC-Softmax is best on simple in-distribution topologies (87.7% at four turns, 93.9% at eight) and on held-out topologies (72.6%); SLIC-Random leads on complex topologies (64.8% at four turns in-distribution) and overall. SLIC-Greedy is consistently weaker (66.5% in-distribution, 43.9% held-out at eight turns), suggesting diversity during rollout construction helps generalization. The Softmax-vs-Random gap is reported as not statistically significant.

• Outperforms Tree MT-GRPO overall: Tree MT-GRPO leads SLIC-Random on simple in-distribution topologies at eight turns (96.3% vs. 92.7%) but trails on the overall in-distribution set (81.2% vs. 87.1%) and on held-out topologies (58.3% vs. 70.6%), with the largest gaps on complex topologies. The authors attribute this to ambiguous credit assignment when advantages are aggregated across later turns, plus policy entropy growth and sampling cost that grows exponentially with horizon.

• Cross-technology-node transfer works without fine-tuning: SLIC-Softmax maintains 73.3–79.8% in-distribution and 61.9–71.0% held-out Pass@1 across five unseen PTM nodes, versus 83.5% and 72.6% at the training node.

• Transfer to a commercial PDK: On a TSMC 65 nm CMOS foundry PDK with no additional fine-tuning, SLIC-Random achieves 55.8% in-distribution and 49.9% held-out Pass@1 at eight turns, down from 86.9% and 70.6% on the same tasks with PTM 65 nm. It still beats SFT by about 29 percentage points on both splits.

• Test-time scaling: Additional simulator interactions improve cumulative Pass@1, reaching up to 83.2% held-out-topology Pass@1 at a 32-turn budget, with diminishing returns at longer horizons. Training with only a four-turn horizon remains effective and still improves beyond it, whereas single-turn GRPO saturates earlier.

• Component ablations: Removing the domain-validity reward gates lowers SLIC-Random's eight-turn Pass@1 by 4.3 points in-distribution and 7.0 points out-of-distribution. Adding the success-conditional slack bonus (with dynamic filtering) improves Pass@1 by 8.5 points in-distribution (76.4% to 84.9%) and 12.1 points on held-out topologies (56.7% to 68.8%).

• Backbone generalization: SLIC improves over SFT on all four evaluated backbones. Scaling from Qwen3.5-4B to Qwen3.5-9B raises in-distribution Pass@1 by 1.8 points (87.1% to 88.9%) and held-out Pass@1 by 5.5 points (70.6% to 76.1%). Qwen3-8B reaches 39.8% in-distribution and 29.0% held-out; Qwen2.5-7B-Instruct (Softmax variant, 800 RL updates) reaches 32.2% and 24.9%.

Methodology in Plain English

The authors treat circuit tuning as a game the model plays against a simulator. Each turn, the model sees an XML-structured description of the circuit, the current measured performance (gain, power, −3 dB bandwidth, phase margin) and device operating states, plus a list of permissible edits. It then outputs a short reasoning trace and a set of relative parameter changes — for example, make this transistor wider — rather than absolute values.

Training happens in two phases. First, a strong prompted model (DeepSeek-V4.1-Flash) runs multi-turn optimization, and each trajectory is relabeled with its best-observed performance tuple as the goal, following hindsight relabeling. Valid and performance-improving turns are filtered out to build a supervised fine-tuning set that gives the 4B model basic reasoning and action behavior. Second, the model is trained with multi-turn GRPO, a policy-gradient method that compares groups of sampled trajectories against each other.

Two design choices adapt GRPO to circuits. The reward first checks physical validity — a circuit with non-positive gain gets no credit for low power or wide bandwidth, and peaking above 6 dB suppresses gain, bandwidth, and phase-margin credit. Successful designs are then scored in the range [1, 1.5] with a bonus based on mean normalized specification slack, so groups where everything succeeds still carry a learning signal. Second, Reward-Guided Rollout Steering samples G candidate edits from the same history, simulates all of them, and lets one eligible successor become the parent state for the next turn — Random, Greedy, or Softmax selection — while all candidates contribute to the policy update. This keeps credit focused on the local update rather than smeared across later turns.

The policy outputs succeed only if a valid design meets all four targets within 5% tolerance under the validity gates. Training used a maximum horizon of eight turns and took roughly 39 hours on 2× B200 GPUs; training and in-distribution evaluation used 35 topologies, with six held-out topologies for out-of-distribution tests.

Why This Matters

Impact on research: The paper shows that a 4B model, trained with verifiable simulator rewards, can outperform a 763B general-purpose model on a tightly constrained engineering task. It also introduces a reusable recipe — domain-gated rewards plus reward-guided rollout steering — for multi-turn RL problems where vanilla trajectory-level credit assignment is too coarse, and it contributes an open benchmark for closed-loop analog optimization where none was standardized.

Real-world applications:

• In-house analog sizing with confidential PDKs, where foundry models cannot leave the corporate network and hosted LLM APIs are prohibited.
• Parallel design exploration, since a 4B model is small enough for multiple specialized agents to run concurrently on different sub-blocks such as amplifiers, mixers, and comparators.
• Cross-node design porting, where a policy trained on public predictive models is reused on older (90–130 nm) and more advanced (22–45 nm) nodes without retraining.
• Design-space triage and first-pass tuning, producing interpretable, human-readable edit sequences that engineers can inspect and refine rather than opaque optimizer outputs.

Industry relevance: The transfer result on a commercial TSMC 65 nm PDK, with local inference and simulation keeping proprietary device models and circuit data in-house, directly addresses the confidentiality barrier that blocks hosted-LLM adoption in commercial and national-security settings. It also avoids the token costs of repeatedly calling external APIs in agentic design loops.

Future Directions

• Multi-agent analog design: Extending SLIC from a single sizing agent toward coordinated specialized agents for topology exploration, block-level optimization, verification, layout-aware refinement, and system-level specification decomposition.
• Scaling to larger mixed-signal systems: Long-context management, cross-agent coordination, and maintaining consistent design state across hierarchical blocks remain open problems.
• Closing the PDK transfer gap: Pass@1 drops from 86.9% to 55.8% in-distribution and from 70.6% to 49.9% held-out when moving from PTM to the foundry PDK; the authors attribute this to process-specific device effects that shift achievable trade-offs, and the mitigation is left open.
• Better credit assignment and search: The paper notes Tree MT-GRPO leads SLIC-Random on simple in-distribution topologies (96.3% vs. 92.7%), suggesting room to combine RGRS-style local comparisons with stronger lookahead or branching.

Target Audience

Researchers and practitioners at the intersection of machine learning and EDA: RL researchers interested in multi-turn, verifiable-reward training and credit assignment; analog and mixed-signal designers evaluating automation that respects PDK confidentiality; and teams building locally deployable, domain-specialized small-model agents for engineering workflows. Readers seeking an entry-level introduction to analog design or to GRPO will need supporting background.

Authors’ abstract

Analog integrated circuit (IC) design is a multi-turn, simulator-grounded engineering task that demands iterative reasoning over coupled performance tradeoffs, yet proprietary process design kits (PDKs) make hosted large language model workflows difficult to deploy in practical design settings. We present SLIC (Small LM Iterating on Circuits), a reinforcement fine-tuning framework that trains a 4B-scale, locally deployable language model for closed-loop analog circuit…