Solver-Independent Automated Problem Formulation via LLMs for High-Cost Simulation-Driven Design

AI summary

Overview

Research area: Natural language processing applied to engineering design automation, specifically the use of large language models (LLMs) to convert natural-language design requirements into executable mathematical optimization models for high-cost simulation-driven design. The paper sits at the intersection of LLM fine-tuning, synthetic data generation, and industrial optimization (demonstrated on antenna design).

Technical level: Advanced. The work assumes familiarity with supervised fine-tuning, prompt engineering, multi-objective optimization concepts (Pareto dominance, non-dominated sorting), Spearman rank correlation, and constrained Bayesian optimization.

Scope: The paper proposes and evaluates APF, a solver-independent framework that generates and filters its own fine-tuning data so that open-source LLMs can formalize ambiguous engineering design requirements into executable optimization formulations without relying on expensive solver feedback.

What This Paper Is About

In high-cost simulation-driven design, engineers describe what they want in vague natural language (for example, "a flat passband" or "deep stopband rejection"), but optimization algorithms need precise mathematical objectives and constraints. Translating between the two is slow, error-prone, and depends on scarce expert knowledge.

Existing LLM methods do not fit this setting: prompt-based methods struggle with vague, domain-heavy requirements, and fine-tuning-based methods normally need solver feedback to filter training data, which is prohibitively expensive when every evaluation requires a high-fidelity physics simulation. The goal of this paper is to automate that translation without ever calling a solver during data construction.

Key Contributions

1. Automated high-quality dataset construction. The authors develop a pipeline combining data generation (LLM-written model equations) with data augmentation (semantic paraphrasing plus order permutation) and test-instance annotation to build a fine-tuning dataset in the absence of solver feedback.

2. A solver-independent evaluation module. Instead of filtering data by running simulations, they use LLM reasoning to produce a reference ranking of test instances and score each generated formulation by the Spearman correlation between the formulation's induced ranking and that reference ranking.

3. Supervised fine-tuning and industrial validation. They apply supervised fine-tuning (SFT) on the filtered dataset and evaluate APF on an antenna design task, reporting that it consistently outperforms existing approaches in formalization accuracy.

4. Robustness analyses. They report a human-vs-LLM-judge study, a listwise-vs-pairwise evaluation comparison, ablation studies on augmentation and selection, and sensitivity analyses on the selection threshold and instance order.

Main Findings

• Large gains over base models. Fine-tuning with APF raised the overall alignment score of LLAMA3.1-8B from 0.2248 to 0.7976. Qwen2.5-7B improved from 0.5292 to 0.7961 and Mistral-7B from 0.3007 to 0.7918.

• Competitive with much larger proprietary models. APF-based LLAMA3.1-8B reached 0.7976 overall, Qwen2.5-7B reached 0.7961, and Mistral-7B reached 0.7918, compared with 0.7923 for Claude-sonnet-4.5, 0.7518 for DeepSeek-V3, and 0.6651 for GPT-4o. Prompting frameworks Chain-of-Experts (0.7252) and OptiMUS (0.6687) scored lower.

• Quality scores of the base dataset were high. Most quality scores clustered between 0.9 and 1.0, and a selection threshold of 0.7 was applied, yielding a final high-quality dataset of 7,879 samples.

• Real antenna optimization validates the approach. When each method's generated formulation was fed to a scalable constrained Bayesian optimization (SCBO) solver, only the curve produced from the APF formulation satisfied all design requirements; formulations from other methods failed the passband and high radiation null requirements.

• LLM judges agree with human rankings. On 30 design requirement sets with 15 test instances each, Spearman correlations with human rankings were 0.8316 (std. 0.1415) for GPT-5, 0.8175 (std. 0.1067) for Gemini-3-Pro, and 0.8007 (std. 0.1560) for GPT-5-mini.

• Listwise evaluation is far cheaper than pairwise with comparable quality. Listwise achieved ρ = 0.8643 using 1 call, 97.66 seconds, and $0.02; pairwise achieved ρ = 0.8536 using 105 calls, 2544.8 seconds, and $0.47.

• Both augmentation and selection matter. Removing data augmentation dropped the overall score to 0.7553; removing data selection dropped it to 0.7653; full APF scored 0.7976.

• Stable under hyperparameter and ordering changes. Varying the selection threshold from 0.6 to 0.8 produced overall scores of 0.7968, 0.7976, and 0.7967. Permuting instance order five times gave a mean ρ of 0.8514 with standard deviation 0.0105.

Methodology in Plain English

The authors first define a consistent way to talk about design problems. Each requirement is written as a triple: the region of the evaluation variable it concerns (for example, a frequency band), the metric measured there (for example, radiation efficiency), and the design intent (a threshold constraint or an optimization goal such as "maximize"). A full design requirement set is a collection of such requirements, and it is derived from real historical simulation records so that every requirement set corresponds to a physically solvable design.

They then build training data. An LLM is prompted to translate each requirement set into executable Python functions, each labeled as either an objective (written for minimization) or a constraint (satisfied when the return value is below zero). This produces the base dataset.

Because engineers phrase things differently, they augment the data. An LLM rewrites each requirement into multiple semantically equivalent phrasings while being constrained to keep all numbers, units, and condition types unchanged. The requirements and their corresponding equations are also randomly permuted so the model does not learn to rely on the position of a requirement in the list.

To decide which generated formulations are actually good without running simulations, they use test instances as a bridge. A test instance is the performance response of a design solution from a high-fidelity simulation, typically a high-dimensional curve. An LLM acting as a domain expert is given a task instruction, a one-shot human expert example, a table of the instances, and the requirement set, and produces a listwise ranking of all instances by how well they satisfy the design intent. This ranking becomes the reference.

Each candidate formulation is then executed on the same test instances to produce its own predicted ranking, computed hierarchically: feasible solutions rank above infeasible ones, and within that, non-dominated sorting based on Pareto dominance applies. The quality score is the Spearman correlation between the predicted ranking and the reference ranking. Samples below the 0.7 threshold are discarded; if a base sample is discarded, all of its augmented variants are discarded with it. The remaining data is used for supervised fine-tuning of open-source models.

For the antenna case study, the radiation efficiency curve is divided into five bands (Low Stopband, Low Radiation Null, Passband, High Radiation Null, High Stopband) with heterogeneous and conflicting goals. From 2,300 sampled requirement sets, 300 were held out for testing and 2,000 were used to build the base dataset.

Why This Matters

Impact on research. The paper shows that supervised fine-tuning data for domain-specific optimization formulation can be constructed and filtered without any solver-in-the-loop, which is the standard requirement in prior work such as ORLM — the authors explicitly excluded ORLM from comparison because frequent interactions with external solvers were prohibitively costly. It also demonstrates that 7B–8B open-source models, once fine-tuned this way, can match or exceed much larger proprietary models on this task, and it provides a reusable recipe (representation, augmentation, ranking-based selection) that other simulation-heavy domains could adapt.

Real-world applications:

• Antenna design, where radiation efficiency curves must satisfy passband, null, and stopband goals simultaneously (the paper's case study, using a three-layer filtering patch antenna).
• Aerospace design, cited by the authors as a high-cost simulation-driven field.
• Microelectronics, likewise cited as a domain where design parameters are tuned against expensive simulated performance distributions.
• Robotics, also cited by the authors as a field with the same requirements-to-formulation bottleneck.

Industry relevance. The bottleneck the paper targets — turning ambiguous engineering intent into executable objective and constraint functions — is described as time-consuming and heavily dependent on expert knowledge. Automating it lets design teams spend expert effort on the physics rather than on translating prose into code, and the reported cost figures for the evaluation step ($0.02 and 97.66 seconds for listwise versus $0.47 and 2544.8 seconds for pairwise) show the pipeline is operationally practical.

Future Directions

• Cross-domain validation. The authors state that evaluation is currently limited exclusively to antenna design and that they plan to validate the framework in broader physics-based engineering fields such as aerodynamics and structural optimization.

• Overcoming context-window limits. The solver-independent evaluation depends on prompts containing detailed test instances, which the authors note is bounded by the context window of current LLMs, potentially limiting the ability to handle highly complex problem descriptions or large-scale validation datasets.

• Extending beyond the two augmentations studied. The paper reports ablations for data augmentation and data selection, but the space of augmentation strategies and their interaction with different requirement structures is not fully explored.

• Scaling the number of heterogeneous requirements. The antenna case uses five conflicting frequency bands; whether the same ranking-based quality signal holds as requirement sets grow larger and more interdependent is an open question the paper does not resolve.

Target Audience

Researchers and practitioners working on LLM-based automated problem formulation, LLM-driven engineering design automation, and synthetic data generation for domain-specific fine-tuning. It is also relevant to industrial engineering and optimization teams in simulation-heavy sectors (antennas, aerospace, microelectronics, robotics) who want to automate the translation from design specifications to optimization models. Because it relies on concepts such as Pareto dominance, non-dominated sorting, Spearman correlation, and constrained Bayesian optimization, readers without a background in optimization or LLM fine-tuning would find the methodology sections demanding.

Authors’ abstract

In the high-cost simulation-driven design domain, translating ambiguous design requirements into a mathematical optimization formulation is a bottleneck for optimizing product performance. This process is time-consuming and heavily reliant on expert knowledge. While large language models (LLMs) offer potential for automating this task, existing approaches either suffer from poor formalization that fails to accurately align with the design intent or rely on solver feedback for data filtering, which is unavailable due to the high simulation costs. To address this challenge, we propose APF, a framework for solver-independent, automated problem formulation via LLMs designed to automatically convert engineers' natural language requirements into executable optimization models. The core of this framework is an innovative pipeline for automatically generating high-quality data, which overcomes the difficulty of constructing suitable fine-tuning datasets in the absence of high-cost solver feedback with the help of data generation and test instance annotation. The generated high-quality dataset is used to perform supervised fine-tuning on LLMs, significantly enhancing their ability to generate accurate and executable optimization problem formulations. Experimental results on antenna design demonstrate that APF significantly outperforms the existing methods in both the accuracy of requirement formalization and the quality of resulting radiation efficiency curves in meeting the design goals.