BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction

AI summary

Overview

• Research area: AI for Engineering / evaluation of large language models on physics-constrained 3D construction tasks (embodied AI, benchmarking).
• Technical level: Advanced.
• Scope: This paper introduces BuildArena, an interactive benchmark that asks LLMs to turn natural-language specifications into physically viable 3D structures, builds them in a physics sandbox, and scores them by simulation outcomes.

What This Paper Is About

Engineering construction automation aims to translate plain-language specifications into structures that actually hold up under physical laws, but no benchmark existed for testing whether LLMs can do this. The authors build a task suite, a construction interface, and a simulation-based scoring pipeline so that language-driven design can be evaluated end-to-end. The paper's guiding question is: how can LLMs be comprehensively evaluated for language-driven, physics-grounded construction automation?

Key Contributions

1. An extendable task design strategy. The authors define three task categories (Support, Transport, Lift), each with quantifiable difficulty levels Lv.1 (Easy), Lv.2 (Medium), Lv.3 (Hard) and corresponding evaluation metrics, framed as a reusable template for adding new categories and levels. The design is organized around six engineering difficulty dimensions: Quantification, Robustness, Magnitude, Compositionality, Precision, and Ambiguity.
2. A 3D Spatial Geometric Computation Library. Because the widely used Besiege simulator's geometric computations are closed-source, the authors built an open-source library that reproduces its building operations, verifies LLM actions, and returns descriptive feedback or blocks invalid actions. A fidelity check on a 49-block machine reports position error below 1.5 × 10⁻⁶ unit length and orientation error below 2.5 × 10⁻⁵ degrees relative to Besiege.
3. An LLM agentic workflow as a baseline evaluation protocol. A fixed, shared workflow of five entities—Planner, Drafter, Reviewer, Builder, and Guidance—plus a Controller for Transport, organizes construction into a Plan Phase, a Draft–Review Loop, and a Build–Guidance Loop. All models are compared under the same agentic setting without model-specific tuning.
4. A simulation-based evaluation pipeline. Completed structures are exported into Besiege-compatible files, loaded into the simulator, and run under task-specific protocols, with each task–LLM pair sampled 64 times and results averaged.

Main Findings

• Frontier models construct, but weakly. Across the three task categories, success rates fall as difficulty rises. On GPT-5, Transport success moves from 78.1% (Lv.1) to 23.4% (Lv.2) and 26.6% (Lv.3); Support from 85.9% to 59.4% and 10.9%; Lift from 95.3% to 10.9% and 17.2%.
• GPT-5 and Grok-4 lead. The paper reports these two as the competitive performers, consistent with existing research. Grok-4 reaches a Support Lv.1 maximum load indicator of 211.4 and a Lift Lv.2 maximum height indicator of 890.6, though its Lift Lv.2 success rate is 31.2%.
• Most models collapse on hard tasks. At the Hard level of all three tasks, most models show very low performance, and for many models Lift Lv.3 success rates drop to zero. The exception the authors name for accuracy-sensitive tasks is GPT-5 and Grok-4.
• Weakness concentrates in four dimensions. Aside from GPT-5 and Grok-4, models show high similarity in their capability distributions, with consistent strengths in Magnitude and Ambiguity and consistent weaknesses in the other four dimensions (Quantification, Robustness, Compositionality, Precision).
• Failure modes are spatial. Overlap Conflict and Face Occupied are the most difficult mistakes to avoid, indicating that agents often fail to track the updated spatial structure. Support tasks show more excess connection errors, while Lift tasks show more misjudgment of face status because structures have fewer modules and fewer redundant faces.
• Structures resemble real engineering practice. Without explicit examples in the prompts, models produce truss-like reinforcement for bridges and multi-wheeled chassis layouts with load-bearing platforms for vehicles; some propose propulsion-powered vehicles or wheel-integrated bridges using automatic braking. The authors read this as latent engineering knowledge activated by the benchmark.
• Zero-shot beats one-shot. On Support Lv.1, supplying a strong one-shot example (reaching a maximum load of 1000) lowered GPT-4o's success rate from 40.6% to 25.0% and indicator from 181.2 to 160.6; GPT-5 fell from 85.9% to 65.6% and from 324.9 to 294.0. Zero-shot is therefore the default protocol.
• More inference does not mean better builds. In all three task categories, the best results often consume moderate token counts, while many failed attempts incur massive token usage.
• Closed-loop feedback helps. On Support Lv.1, simulator feedback brought failed samples to 100% success within five rounds. On Lift Lv.2, adding an analyst agent that reviews trajectories moved all three tested models from a 0% baseline to non-zero success.
• Open-weight models are viable but behind. Qwen3.5-27B reaches 35.9% success on Support Lv.1 and 12.5% on Transport Lv.1 and Lv.3, while overall remaining substantially below GPT-5 and Grok-4, especially at higher difficulty.
• Complexity is a prompt choice, not a system limit. When prompts explicitly emphasize structural complexity and visual elaborateness, LLMs produce substantially more intricate constructions with the same module set.

Methodology in Plain English

The authors start by abstracting the ways engineering tasks get hard—how much numerical reasoning is required, how tolerant a design is to a single failure, how large the structure is, how deeply it nests sub-structures, how precise placement must be, and how ambiguous the instructions are. From these they build three task families. Transport asks for a machine that moves; Support asks for a bridge across a gap; Lift asks for a rocket, an aircraft, or a two-part assembly that launches.

Each task family has three difficulty tiers. Support widens the gap across levels and allows up to three substructures in Lv.2 and Lv.3. Lift goes from a single engine measured by thrust-to-weight ratio, to an integrated rocket aircraft, to a two-substructure build that must be assembled. Transport removes the explicit "four-wheeled vehicle" hint after Lv.1 and adds a cargo load.

Since Besiege only offers a graphical interface for humans, the team wrote an open-source spatial computation library that mirrors its construction logic and physical constraints, so an LLM can issue commands in language and get back either state descriptions or rejection notices. Actions fall into Build, Refine, Query, and Control.

The LLM workflow is a chain of specialized prompt roles: a Planner writes the plan, a Drafter draws schematics that a Reviewer critiques in a loop, and a Builder plus Guidance translate approved schematics into commands until Guidance confirms completion. A Controller handles Transport.

For scoring, finished structures are exported, loaded into Besiege, and run under task-specific protocols—for example, increasingly heavy loads placed on a bridge, or continuous firing of water cannons to simulate launch. Each task–LLM pair is sampled 64 times and averaged. Models are ranked only by success rate and the task-specific performance indicator; the number of parts is reported as a descriptive signal, not a ranking criterion. Cost is tracked through input tokens, output tokens, and total LLM requests.

Why This Matters

This work opens an evaluation frontier that sits between text benchmarks and robotics: it tests whether language models can produce artifacts that survive contact with physics, not just produce plausible-sounding answers. It gives the community a shared, reproducible testbed and a diagnostic map of where current models break down—spatial tracking, hierarchical assembly, and precision.

Real-world applications implied by the paper's framing:

• Civil infrastructure: bridges and load-bearing structures designed from plain-language requirements, as in the Support task.
• Automotive and transportation: vehicle chassis and mobility designs, as in the Transport task.
• Aerospace and propulsion: rocket engines and launch-capable assemblies, as illustrated by the paper's "design a rocket that meets Mars mission requirements" example.
• Manufacturing and assembly planning: end-to-end build plans covering design, fabrication, and assembly that can be fed into production systems.

Industry relevance centers on the promise of engineering efficiency and productivity gains from automating the design-to-assembly pipeline, and on the practical value of knowing which model to trust for which class of construction problem. The paper explicitly frames BuildArena as a controlled testbed rather than a direct proxy for real manufacturing pipelines.

Future Directions

1. Closing the simulation-to-reality gap. Besiege offers realistic physics but differs from real robotics and manufacturing constraints; extending evaluation toward those constraints is an open problem.
2. Robust iterative refinement. The main experiments are single-shot. The closed-loop studies show post-failure reflection can help, but the authors call robust iterative refinement an open challenge.
3. Broader task and module coverage. The suite currently covers three task categories, and the limited diversity of basic units in the module library constrains what can be built.
4. Diagnosing and improving the weak dimensions. Because all models show consistent weaknesses in Quantification, Robustness, Compositionality, and Precision, the paper points to these as concrete targets for future model improvement, alongside investigating latent engineering knowledge in pretraining.

Target Audience

Researchers in LLM evaluation, embodied AI, and AI for science or engineering who need a physics-grounded benchmark; robotics and simulation researchers interested in language-to-structure pipelines; and engineering practitioners exploring automated design, fabrication, and assembly planning. Readers with a machine learning background will get the most out of the benchmark design and comparative results, while those focused on construction will find the task taxonomy and simulation protocols most useful.

Authors’ abstract

Engineering construction automation aims to transform natural language specifications into physically viable structures, requiring complex integrated reasoning under strict physical constraints. While modern LLMs possess broad knowledge and strong reasoning capabilities that make them promising candidates for this domain, their construction competencies remain largely unevaluated. To address this gap, we introduce BuildArena, the first physics-aligned interactive benchmark designed for language-driven engineering construction. Technically, it contributes to the community in two aspects: (1) an extendable task design strategy spanning static and dynamic mechanics across multiple difficulty tiers; (2) a 3D Spatial Geometric Computation Library for supporting construction based on language instructions. On nine frontier LLMs and three additional open-weight models, BuildArena comprehensively evaluates their capabilities for language-driven and physics-grounded construction automation. We release the code at https://github.com/AI4Science-WestlakeU/BuildArena to benefit construction automation in engineering applications.