LLM-Assisted Optimization for Simulation-Based Redesign

AI summary

Overview

Research area: AI-assisted engineering design optimization — specifically, using large language models (LLMs) to guide black-box, simulation-based redesign of an existing engineering artifact.

Technical level: Intermediate. The paper is readable by someone with general familiarity with optimization concepts (search budgets, constraints, black-box simulators), but it assumes comfort with terms like Bayesian optimization, variable fixing, and warm starting.

Scope (one sentence): The paper builds and tests a framework called WAFER that inserts an LLM into three different stages of a simulation-driven redesign loop, and benchmarks those three strategies against four conventional numerical optimizers on an electric-vehicle battery-pack cost problem across four types of design-requirement change.

What This Paper Is About

Engineering teams rarely design from scratch — they adapt an existing design when requirements, prices, or packaging constraints change. When the design's performance is only computable by a slow external simulator, this "redesign" becomes an expensive black-box reoptimization problem, and conventional optimizers ignore the engineering knowledge that a human would bring (what changed, which physics matter, which parts of the old design are still good).

The paper asks how best to feed that semantic engineering knowledge into the search using an LLM. It compares three placements of LLM guidance — proposing starting points (warm start), deciding which design variables to freeze (variable fixing), and running the whole search itself (ReAct) — against cold-start and reference-warm numerical baselines on a battery-pack redesign case study.

Key Contributions

1. An integrated framework (WAFER — Workflows for AI-Facilitated Engineering Redesign) that places LLM guidance at three distinct stages of simulation-based redesign: before search (warm starting), at problem setup (variable fixing), and at every step of search (ReAct).

2. A controlled benchmark across four structurally different redesign scenarios on an EV battery-pack cost-minimization problem solved with the BatPaC v5.0 simulator, spanning three LLM sizes (4B, 9B, 27B), two levels of supplied engineering information (L1 and L2), and four black-box solvers (Bayesian optimization, genetic algorithm, MADS, random search).

3. Evidence that LLM-assisted reoptimization matches the feasible-design discovery of numerical baselines with 80% fewer simulator evaluations while tripling the fraction of runs that find near-optimal solutions, under a common budget of 20 accepted simulator evaluations.

4. A scenario-conditioned explanation of which integration strategy wins and why — variable fixing changes the dimension of the search, warm starting directs the search toward promising regions, and ReAct can change the search strategy at every step — plus the finding that the benefit of a larger model grows with how much control the LLM has over the search.

Main Findings

• Fewer evaluations, better outcomes. The integrated approach can match the feasible-design discovery of numerical-solver baselines with 80% fewer simulator evaluations and triples the fraction of runs that reach near-optimal solutions.

• Warm start and ReAct dominate in three of four scenarios. In S1, S2, S3, and S4 curves, WS and ReAct lie clearly above the numerical baselines in three of the four scenarios; the exception is S1, where the reference design is already feasible and the reference-warm baseline starts at 100% discovery.

• Sharpest separation in the fast-charge scenario (S3). At the end of the 20-evaluation budget, feasible discovery is 77% for ReAct and 69% for WS, versus 35% for the reference-warm baseline and 17% for cold start. The median index of the first feasible design is 4 for ReAct and 11 for WS, while more than half of the reference-warm and cold-start runs found no feasible design within 20 evaluations.

• Large energy-requirement change (S4) also favors LLMs. Median first-feasible index is 4 for both WS and ReAct versus 17 for reference-warm, and more than half of cold-start runs found none.

• S2 (space constraint) is the hardest scenario for everyone. More than half the runs of every method failed to find a feasible design, though WS and ReAct still reached feasibility more often than the two baselines.

• Adding an LLM does not break the easy case. In S1 the reference design is already feasible, so discovery means submitting it unchanged: reference-warm reaches 100% at the first evaluation, WS reaches 99%, and ReAct reaches 100%.

• Variable fixing is scenario-specific, not generally good. VF reaches 72% feasible discovery in S3, comparable to the other LLM methods, but only 27% in S4 — worse than the 61% of reference-warm and even the 37% of cold start. On near-best attainment (within a 2% optimality gap), VF reaches 53% in S3, close to ReAct's 54% and well above WS's 19%, but rarely reaches the near-best region in S1, S2, and S4.

• Cold start never approaches the best design. Cold start stayed at or below 1% near-best attainment in every scenario.

• Feasibility and quality are different difficulties. S1 was the easiest scenario for finding a feasible design but all methods' near-best curves stay low, showing the two criteria do not track each other.

• Best-known designs shift in interpretable ways. The cost of the best design found across all runs is 97.1 USD/kWh (S1), 97.4 (S2), 100 (S3), and 90.3 (S4). The cobalt price shock in S1 flips the chemistry from NMC811 to LFP at its thickness bound; S2 keeps chemistry and thickness but rearranges to three modules per row and three rows with three fewer cells per module; S3 keeps topology and thins the cathode from 155 to 96.7 μm; S4 combines both changes.

• Model-size benefit scales with LLM control. The benefit of a larger model is largest for ReAct (which controls every proposal) and inconsistent for warm starting and variable fixing (which hand the remaining search to a numerical optimizer).

• Not reported in the supplied excerpt: the specific inference-time and token-use figures from the cost analysis, and the detailed reasoning trace analysis in Section 4.2 (the provided text truncates mid-sentence at Section 4.2.1).

Methodology in Plain English

The researchers picked a realistic redesign problem: an EV battery pack whose cost per usable energy must be minimized subject to energy (≥84 kWh), power (≥239 kW), voltage (200–1000 V), cell capacity (40–120 Ah), and pack length/width limits (≤2200 mm, ≤1500 mm). All evaluation goes through BatPaC v5.0, treated strictly as a black box. A grid search produced one reference design — NMC811 chemistry, 155 μm electrode, 29 cells per module, 2 modules per row, 4 rows — costing 96.5 USD/kWh.

They then defined four redesign scenarios by changing one aspect of the context: (S1) NMC811 cathode material cost rises from 19.63 to 25.75 USD/kg; (S2) allowable pack width drops from 1500 to 1100 mm; (S3) recharge time drops from 60 to 30 minutes; (S4) required energy rises from 84 to 130 kWh with relaxed packaging to 2700 × 1650 mm.

Every method got the same budget: 20 accepted BatPaC evaluations per run. Baselines were cold start (random initialization) and reference-warm (the old design included in initialization), each paired with one of four solvers — Bayesian optimization (via Ax), a mixed-variable genetic algorithm (via pymoo), MADS, and random search. The three LLM strategies differed in where the model acts: warm starting has the LLM propose three initial designs for the solver; variable fixing has the LLM decide which variables to freeze so the solver searches a smaller space; ReAct has the LLM propose every design itself, seeing the objective, constraint values, and feasibility of each prior proposal as feedback. All optimizers saw the same penalized scalar objective (cost plus a fixed penalty and a normalized constraint-violation term).

The LLM was Qwen3.5 at 4B, 9B, and 27B parameters, run locally in bfloat16 with no fine-tuning. Two prompt levels were compared: L1 (problem definition only) and L2 (problem definition plus compact battery engineering relationships). Outputs were parsed into structured actions and validated before simulation.

The design totaled 248 configurations × 50 runs = 12,400 runs. Statistics used two-sided Fisher exact tests with Holm correction, 95% Wilson intervals for proportions, and Spearman rank correlation for analyzing the LLM's variable-release and move probabilities. The 20-evaluation budget was chosen so ReAct's growing history stayed within the models' 32,768-token context window; no run terminated due to context length.

Why This Matters

Research impact. The paper reframes LLM-assisted optimization as a design question about where to insert semantic knowledge rather than whether LLMs help. By holding the simulator budget fixed and varying the LLM's point of intervention, it produces an explanation — dimension reduction versus region targeting versus adaptive control — that transfers to other simulation-driven design problems.

Real-world applications:

• Automotive redesign: adapting a battery pack when material prices shift, packaging shrinks, fast-charge targets tighten, or a pack is carried to a new vehicle model or trim — exactly the four scenarios tested.
• Supplier and material disruptions: the S1 cobalt-price-shock scenario mirrors real supply-chain events that force chemistry or thickness changes.
• Platform reuse across vehicle programs: the S4 energy-scale-up case reflects reusing a pack architecture for a larger vehicle class.
• General simulation-heavy engineering: any setting where the simulator is too complex or expensive for gradient-based or analytic optimization but where engineers already know what changed.

Industry relevance. The paper cites a major automotive manufacturer reporting more than 1,000 engineering change orders per month at roughly 20,000–50,000 Euros per change, with individual changes taking weeks or months. Cutting simulator evaluations by 80% for equivalent discovery directly attacks that cost, and the local-deployment choice (open weights, no hosted API) matters for companies that cannot send sensitive design data outside.

Future Directions

1. Make strategy selection automatic. The best method depends on the structure of the requirement change — variable fixing wins when a thickness-like variable is the lever (S3), warm start/ReAct when topology or feasibility landscape shifts. A meta-controller that classifies the change and picks the LLM role is the obvious next step.

2. Explain and improve variable fixing's failure in S4. VF fell below even cold start (27% vs 37%) when the energy target scaled up, presumably because it froze variables that needed to move. Better criteria for when not to freeze — or adaptive unfreezing — are open.

3. Extend beyond the 20-evaluation budget and single-simulator setting. The budget was capped by ReAct's 32,768-token context. Longer horizons, memory compression, or retrieval over the evaluation history would test whether the LLM advantage persists on harder problems.

4. Quantify the cost-benefit trade-off more broadly. The paper measures inference time and token use alongside gains, and observes that model-size benefit scales with LLM control — but the excerpt does not report the specific cost figures. A full Pareto analysis of solution quality against inference cost, across hardware and model families, remains open.

Target Audience

Engineers and researchers working on simulation-based design optimization who want to know whether and how to add an LLM to an existing optimizer loop; mechanical, aerospace, and energy-systems designers dealing with expensive black-box simulators and frequent redesign cycles; and AI researchers studying LLM-agent integration with numerical solvers who need a carefully controlled, budget-matched benchmark rather than a single-method demonstration.

Authors’ abstract

Engineering design frequently involves redesigning a prior solution to adapt to changes in context, such as material prices or system requirements. Redesign often leverages simulators too complex and/or computationally intensive for traditional numerical solvers, which lack engineering knowledge and context, to ensure global solution quality. Large language models (LLMs) can interpret semantic information, including engineering knowledge about the system and changes in context, to help guide search, and recent studies have shown that incorporating this knowledge into optimization can improve search performance. However, how best to combine LLMs with numerical solvers in redesign remains unclear. We develop and test an integrated LLM-optimization approach for redesign and compare three methods for leveraging LLMs (warm start, variable fixing and ReAct) and four black-box solvers (Bayesian optimization, genetic algorithms, mesh adaptive direct search and random search) on an electric vehicle battery pack cost minimization case study across four types of redesign scenarios. We find that the integrated approach can match feasible design discovery of the numerical-solver baselines with 80% fewer simulator evaluations while tripling the fraction of runs that find near-optimal solutions. We find that the best LLM-solver integration approach and the way the prior solution is leveraged depend on the structure of the design requirement change, and that the benefit of increasing LLM model size grows with the extent to which the LLM controls the search. These results show clear potential for LLMs in redesign tasks for guiding or replacing numerical solvers.