BrickBench: Evaluating Agentic Brick Design

AI summary

Overview

• Research area: Artificial intelligence / agentic coding benchmarks, with a focus on text-conditioned 3D assembly design using LEGO parts.
• Technical level: Advanced. The paper assumes familiarity with coding-agent evaluation, benchmark design, LDraw/part-connector representations, rigid-body simulation, and ELO-style pairwise preference modeling, though its central question is stated plainly.
• Scope (one sentence): The paper introduces BrickBench, a 300-prompt benchmark in three settings that scores coding agents on producing physically valid, prompt-aligned, and well-designed LEGO assemblies inside a provided environment called BrickAgent.

What This Paper Is About

Existing text-to-LEGO systems have been evaluated on small objects of at most 100 parts, and they only measure whether an assembly reproduces a description. That leaves untested both physical complexity (hundreds or thousands of interconnected parts, or a restricted inventory) and design quality — whether a build is worth building, as opposed to merely valid and faithful. The paper asks whether general coding agents, given tools to construct, inspect, and validate assemblies, can design LEGO sets from text prompts, and it measures how far they still fall short of human designers.

Key Contributions

1. BrickBench, a benchmark of 300 prompts in three settings — Model (at most 400 parts), Set (400–4000 parts), and Alt-Build (a fixed inventory) — with metrics for physical validity (connections, collisions, and stability under simulation), semantic alignment (VQA over yes/no questions decomposed from each prompt), and design quality (ELO from pairwise judgments by a VLM validated against human raters).
2. BrickAgent, an environment letting coding agents programmatically construct, inspect, and validate complex LEGO assemblies, with part search, placement by connectors, subassemblies, rendering, and a validator that names the parts at fault. Without it, the share of valid assemblies falls from 100% to 40% for GPT-6 Astra and to under 1% for GPT-5.6 Luna.
3. A systematic evaluation of eleven frontier agents and two data-driven baselines, including human studies that validate the design metric and measure the gap to human design.
4. Validation that a general agent subsumes task-specific models, by running GPT-5.6 Luna on the BrickNet text-to-assembly validation set and finding it approaches the alignment of the held-out ground-truth assemblies.

Main Findings

• Validity is nearly solved with the environment. Five agents deliver a valid assembly for every prompt, and none fall below a Valid proportion of 0.80 overall. Of the 137 invalid assemblies in the reference set, 44 were prompts with no delivered assembly; the rest fail on stability (41), collisions (31), and the part requirement (21).
• GPT-6 Astra leads overall among the reference agents. Overall, Astra scores Valid 1.00, VQA 0.954, ELO 1297 ± 22, Align ELO 1295 ± 27, Design ELO 1299 ± 27 at a cost of $4.06 per assembly. The weakest reference agent, GLM 5.3 Flash, scores Valid 0.92, VQA 0.590, ELO 752 ± 23 at $0.54.
• Design quality separates agents far more than alignment does. VQA spans roughly 0.59 to 0.95, while Design ELO shows a 545-point gap between GLM and Astra — a gap the paper interprets as a judge preferring Astra's assembly in around 96% of pairs. GPT-5.6 Sol, Gemini 3.8 Flash, Qwen 3.8 Flash, and DeepSeek V4.1 Flash form a middle group within 41 ELO points of one another.
• Newer agents improved the cost-performance frontier. GPT-6.1 Sol matches Astra on ELO (1293 against 1297, with overlapping intervals) at a fifth of its cost ($0.87), and Claude Opus 5.5 rates 148 points above Claude Opus 5 at 40% of its cost.
• Cost does not track rank. The reference set spans a factor of 30 in cost; Muse Spark 1.3 costs nine times what DeepSeek V4.1 Flash costs and rates 138 points lower. The conclusion states that agents displace task-specific data-driven models at an increase of 900–6,800 times the cost.
• General agents beat the task-specific baselines. Evaluated on the BrickNet validation set (512 held-out captions, single objects of at most 100 parts), GPT-5.6 Luna scores PE 0.313, SigLIP 2 0.818, and VQAScore 0.732, within 0.02 of the ground-truth assemblies (0.315, 0.826, 0.748) and above BrickNet-14B (0.283, 0.625, 0.602) and BrickGPT (0.157, 0.052, 0.050). On BrickBench's Model setting, BrickNet-14B is collision-free on only 26% of prompts, and BrickGPT, though valid on each prompt, satisfies 2.4% of the questions (reported as 2%).
• The environment is essential to physical validity. Without BrickAgent, GPT-6 Astra's Valid proportion drops from 1.00 to 0.40 and GPT-5.6 Luna's falls below 0.01; their assemblies average 7.19 and 155.51 colliding pairs and 6.12 and 83.00 connected components, against 0.00 collisions and 2.21 and 4.60 components with the environment. Alignment and design do not collapse: Astra's VQA falls by 0.01 and its ELO moves within the interval, while Luna's VQA rises by 0.02 and its ELO by 62 points — suggesting buildability constrains expressivity.
• The automated design judge tracks human judgment. In a Prolific study with 102 raters providing 4,080 judgments, human rankings agree with Design ELO on 34 of 36 agent pairs (Kendall tau = 0.89).
• A clear gap to human designs remains. In a second study, five raters gave 360 ratings and selected the human model in 323 of them. No agent was taken for a human designer more than one time in five; GPT-6 Astra, most often taken for a person, was chosen in 4 of its 21 pairs.
• The Set setting is hardest on validity. Validity drops to 0.68 for Muse Spark 1.3, 0.79 for Qwen 3.8 Flash, and 0.87 for DeepSeek V4.1 Flash, and part counts crowd the 400-part floor so many agents build the smallest scene the setting allows. Alt-Build costs the least and maintains the highest validity, and alignment is highest in Model, the only setting where the baselines apply.
• Failure is driven by question type, not subject. Every agent satisfies at least 91% of questions about color, but texture is satisfied only 49–83% of the time, with object type and state the next hardest. No prompt category is consistently harder, and the two top agents score at least 0.90 in every one.
• Stronger agents build bigger and more varied assemblies. In Model, Astra averages 217 parts of 35 types in 9 colors, while GLM averages 27 parts of 10 types in 5 colors. In Set, Qwen, DeepSeek, Muse, and GLM average 23 to 40 connected components, against 2.2 to 4.2 for Astra, both Sol models, and both Opus models.

Methodology in Plain English

The researchers turn LEGO design into a benchmark for coding agents. Each agent receives a text prompt and a part budget dictated by one of three settings: Model (at most 400 parts), Set (400–4000 parts), or Alt-Build, which restricts the agent to the 783 pieces of 104 part types in retail set 10698. The Model and Set settings draw on the full LDraw library as supported by BrickNet, 14,441 part types, and each setting contains 100 prompts — ten in each of ten topic categories derived from the BrickLink Designer Program, with the Floral/Nature/Animal category split into Floral/Nature and Animal. Prompts average 31 words in Alt-Build, 45 in Model, and 57 in Set, and were LLM-generated and manually validated.

Rather than telling agents how to design, the authors give them BrickAgent: the LDraw library, BrickNet's typed connector system (stud, hinge, axle, ball, and fixed connectors), and tools to search the library, place parts by position or connector, mirror and measure, define and combine subassemblies, render from any viewpoint, and validate. The validator reports connectivity, collision, and stability failures with the parts involved. Stability is tested in PyBullet by treating connected components as rigid bodies of uniform density, with high friction and no restitution; an assembly passes if no point of any component moves more than 3 LDraw Units, described as three quarters the height of the stud of a brick. An assembly is valid only if it meets its setting's part requirement and is collision-free and stable.

Scoring uses three views of quality. Valid is the proportion of prompts producing a buildable assembly. VQA renders each assembly from eight views and asks Gemma 4 31B every question of a question graph built in the style of DSG — atomic yes/no questions about entities, attributes, and relations, organized by dependency so that a missing entity fails the questions hanging from it. ELO comes from pairwise VLM judgments: every pairing of assemblies from different systems is shown four views each, asked once which better matches the prompt and once, with the prompt withheld, which is better designed by the standard of an official set, with each pair asked in both orderings. Wins are fit with a Bradley–Terry model and reported as ELO, with Align ELO and Design ELO averaged after rescaling to the anchors.

Eleven frontier agents are evaluated (GPT-6 Astra, GPT-6.1 Sol, GPT-5.6 Sol, GPT-5.6 Luna, Claude Opus 5.5, Claude Opus 5, Gemini 3.8 Flash, Qwen 3.8 Flash, DeepSeek V4.1 Flash, GLM 5.3 Flash, and Muse Spark 1.3), with Opus 5 and Opus 5.5 in Claude Code and the others in Codex CLI, each given 300 turns per prompt. BrickNet-14B and BrickGPT are evaluated as baselines, restricted to the Model setting. Nine agents form the ELO reference scale; Opus 5.5 and GPT-6.1 Sol were released after the human studies and were scored on automated metrics only.

Why This Matters

The paper reframes what is hard about generative design: physical validity and prompt adherence are becoming checkable and largely checkable-solved, while design quality — clever part usage, proportion, and surface detail — remains the open problem and has no pass/fail test. It also shows empirically that a general coding agent plus a well-designed tool environment can beat models purpose-built for the task, and that removing those tools causes physical validity to collapse even when alignment holds up.

Real-world applications implied by the framing:

• Generative design tools for physical assembly, where both semantic matching and structural constraints matter: CAD (BenchCAD-style CadQuery programs), graphics scripting (Blender scripts, SceneCraft), and machine assembly from modular parts (BuildArena, PhyBlock).
• Evaluation of agents that produce executable artifacts, extending execution-based benchmarks like SWE-bench and OSWorld to domains where a result must be geo­metrically and physically consistent rather than merely passing tests.
• Preference-based quality metrics for creative outputs, using pairwise VLM judgments validated against human raters where no definitive correctness test exists.
• Content generation for creative industries, since the Set setting is sized to mirror the BrickLink Designer Program, the standard for published fan-designed LEGO sets.

Industry relevance: the results give a concrete picture of cost-performance tradeoffs across eleven frontier agents, note that BrickNet-14B produces an assembly for six hundredths of a cent while GPT-6 Astra does so in $4.06, and identify the environment as the binding constraint on reliability — all directly relevant to anyone deploying agents that write and verify code against physical or geometric constraints.

Future Directions

• Closing the gap to human design. Participants familiar with LEGO design filtered agent-designed assemblies from human ones nine out of ten times; the paper leaves prompt-matched human comparisons to future work, since no human designs exist for the BrickBench prompts and a matched study would require commissioning designs.
• Stronger physical simulation. The current simulator checks that an assembly balances as a rigid body but does not model the holding force of stud connections or the play of axles, so a physical build might sag or come apart where the simulation passes. The authors note that a general force model or finite-element approach would currently limit the available part catalog.
• Reducing cost. Displacing task-specific models costs 900–6,800 times more, which the authors name as an axis for future work.
• Improving fine-grained prompt adherence. Texture, object type, and state questions are the hardest for every agent, and the paper notes that without the buildability constraint, agents sometimes depict prompts as well or better — raising the question of how to keep expressivity without sacrificing validity.

Target Audience

Researchers and engineers working on agentic benchmarks, code-generating agents, and generative 3D or physical design; practitioners evaluating the cost-performance tradeoffs of frontier models on long-horizon construction tasks; and anyone interested in automated evaluation of creative quality where correctness alone is insufficient. The paper is most useful to readers comfortable with benchmark construction and preference-based evaluation rather than to newcomers to LLM agents.

Authors’ abstract

We propose BrickBench, a benchmark for agentic text-conditioned LEGO-set design. Given a prompt, an agent is tasked with producing an assembly that not only satisfies semantic and design criteria, but that can also be physically built. To do so, it must select parts from a discrete library and reason jointly about local and global constraints. We score validity, alignment, and design across three settings that vary in scale and part availability. We provide BrickAgent, an environment for coding agents to construct, inspect, and validate their designs. We find that leading agents largely satisfy verifiable physical and semantic requirements, but fall short of human designs. We release our benchmark and environment at http://www.brickben.ch