BrickWright: Agentic Creation of Physically Feasible Brick Builds

AI summary

Overview

Research area: Computer vision and 3D content generation, specifically agentic (LLM-driven) generation of multi-part physical assemblies, evaluated on LEGO® brick builds.

Technical level: Intermediate. The core ideas are accessible (write code, check it, fix it), but the paper assumes familiarity with LLM coding agents, 3D part placement, and the LDraw brick format.

Scope: The paper introduces BrickLayer, a Python toolkit for assembling and physically validating brick builds, and BrickWright, a plan-execute-validate harness that guides pre-trained LLM agents to use it, evaluated on the 512-prompt BrickNet-512 benchmark.

What This Paper Is About

Frontier coding agents are good at 3D modeling, but they struggle when the object must actually be assembled in the real world: they produce builds with floating, colliding, or disconnected parts. The paper asks whether giving agents a well-designed assembly API plus a physical validator is enough for them to produce valid builds without any fine-tuning, using LEGO® brick builds as the test domain. A build counts as "physically feasible" if it uses only real catalog parts, has no colliding or floating parts, and forms a connected structure if possible.

Key Contributions

1. BrickLayer, a Python package for brick construction and validation with three components: keyword search over the LDraw catalog of 17k+ parts (ranked by BM25 score times a logarithmically scaled popularity factor), an ergonomic assembly API built on typed connectors, and a validator that reports collisions, floating elements, and disconnected components.

2. BrickWright, a lightweight agent harness that prompts a single pre-trained LLM agent to write and revise a build.py file containing create_build() and run_tests(), iterating through the tools validate, render, and finalize (plus file-editing and explore tools) in a plan-execute-validate loop.

3. An empirical demonstration that this toolkit alone raises physical validity to 100% for all four tested agents on 512 prompts, including lightweight models, without a significant cost increase or loss of prompt fidelity.

4. Ablations isolating each component, showing that the assembly API roughly halves inference cost relative to working in raw LDraw, and that removing the validator drops every agent below 79% validity.

Main Findings

• Validity jumps to 100%: Baseline agents range from 9% validity (GPT-6 Luna) to 69% (GPT-6 Astra); with BrickWright, all agents reach 1.000 valid, 1.000 collision-free, and 1.000 floater-free on BrickNet-512, with mean component counts of 1.00.

• Lightweight models benefit most: Claude Haiku 5.5 goes from 0.328 to 1.000 validity; GPT-6 Luna goes from 0.086 to 1.000. Luna's prompt adherence improves while costing $0.03 per prompt versus $1.28 for the full BrickWright Astra configuration.

• Prompt adherence is preserved or improved: VQAScore rises for Luna (0.6504 to 0.6945) and Haiku (0.6032 to 0.6523) and holds steady for Sol (0.7610) and Astra (0.7695). BrickWright agents also use more bricks and more unique part IDs than vanilla baselines (e.g., Astra: 89.3 parts, 18.9 unique).

• It beats specialized text-to-brick models: BrickGPT reaches 0.990 validity but only 0.0658 VQAScore and 144.8 parts on average; BrickNet reaches 0.318 validity; Articraft reaches 0.684 validity at $3.00 per sample versus $1.28 for BrickWright with Astra.

• The assembly API is the biggest cost lever: Without it, agents spend roughly twice the tokens and about three times as many validate calls. GPT-6.1 Sol passes validation within two calls 80.7% of the time with the full system versus 36.9% without the API. Luna's average part count collapses from 27.4 to 7.2, producing overly simple structures.

• The validator is essential for validity: Using BrickLayer's API without the provided validator boosts validity by up to 44 percentage points over the Codex/Claude Code baselines, but all agents then score below 79%.

• Few iterations are needed: Most BrickWright agents use 2–3 validate calls; Luna uses 4–5 (2.89 for Haiku, 2.40 for Astra, 4.62 for Luna).

• Failure modes differ by agent strength: Physical checks account for 63–85% of failed validation calls, followed by runtime errors at 11–28%. GPT agents mostly fail on collisions; Haiku's main failure mode is connectivity.

• Vanilla agents do try to self-validate: Of 512 builds by GPT-6 Astra with Codex, 471 (92%) used a generator program and 424 (83%) used a validator program, yet still produced invalid builds.

• Extensions come for free: BrickWright handles image conditioning, scenes, builds over 29,000 parts, articulated objects, build editing, and generation constrained to a given set of bricks.

Methodology in Plain English

The researchers first built a better programming interface. The standard LDraw format only lets you place parts by absolute 3D position, which makes it awkward to attach one part to another and easy to get wrong. BrickLayer instead models each part as having typed connectors (studs, antistuds, pins, axles, ball joints, and so on) and lets you say "join this connector to that one." It also offers convenience functions for common shapes like bricks, plates, slopes, minifigures, and filling a box or wedge. Builds can be nested, so complex models are assembled from smaller components.

Validation is geometric. To detect collisions, each mesh is inset by 0.1 mm before checking intersections, because snap-fit connections are tight enough that a naive check gives false positives. To detect floating parts, the build is dropped to the ground plane and each part must reach grounded material through mated connectors or upward-facing surface contact. Connector alignment depends on joint type, and for sliding joints the system rotates the joint 15° about its axis to decide whether it is prismatic or cylindrical.

BrickWright then wraps a single pre-trained agent around this toolkit. The agent writes a Python file defining a build and its own tests, calls validate (which runs physical checks plus the agent's tests), calls render to look at the result, and calls finalize when satisfied. The system prompt contains only the BrickLayer documentation and the feasibility requirements, no worked examples or planning advice. Baselines include specialized trained models and the same agents run with Codex or Claude Code, which can read and execute arbitrary files (including the LDraw docs and part library) but have no BrickLayer. Validity numbers were spot-checked by a human LEGO designer for potential false positives, which shifted collision-free rates by 0.2 to 1.4 percentage points and component counts by up to 0.13.

Why This Matters

The paper's central claim is that for physically constrained generation, the tooling matters more than model scale: a good API and a deterministic verifier let even a lightweight agent match or beat a frontier agent working unaided. That reframes agent capability as partly an interface-design problem.

Real-world applications:

• Automated design tools for construction toys: generating instruction-ready models from text or reference images, including large builds and editable drafts.
• General CAD and assembly design: the same connector-and-validator pattern applies to furniture, machinery, and modular hardware where parts must mate correctly.
• Robotics and manufacturing planning: verifying that a set of physical parts can be joined without interpenetration before committing to a physical build.
• Interactive creative software: agents that revise an existing model based on natural-language edits while keeping it structurally sound.

Industry relevance: the cost results matter commercially. Luna with BrickWright reaches 100% validity at $0.03 per prompt versus $1.28 for Astra with BrickWright and $3.00 for Articraft, suggesting that feasibility tooling can substitute for expensive frontier inference in constrained domains.

Future Directions

• Assembly sequences and stability: the current feasibility definition ignores whether a build can actually be put together in order and whether it would stay standing; the authors name force-based stability analysis as a natural extension.

• Distilling agent capability into smaller models: agentic methods beat specialized models on quality but are slower and more costly (e.g., 267s for BrickWright Astra vs 5s for BrickNet), and closing that gap is left open.

• Generalizing the toolkit pattern beyond bricks: whether the same API-plus-validator recipe transfers to other snap-fit or connector-based physical systems is untested here.

• Benchmarking against concurrent work: the paper notes BrickBench as concurrent work introducing a toolkit for agentic brick creation focused on comparing agents, leaving direct comparison as an open question.

Target Audience

Researchers and practitioners working on LLM agents for 3D content creation, code-as-representation methods, and physically grounded generation; developers building tool APIs for AI agents; and LEGO® or hobbyist CAD tool builders interested in automated, physically valid model generation. Readers with basic familiarity with LLM agents and 3D geometry will get the most from it.

Authors’ abstract

Frontier coding agents are becoming increasingly adept at 3D modeling. Yet, they still face challenges when building objects meant to be assembled in the physical world, often creating invalid assemblies with floating, colliding, or disconnected parts. We argue that providing agents with tools for assembling builds and verifying their physical feasibility is sufficient for them to produce physically valid assemblies that still adhere to input prompts. Focusing on the domain of LEGO brick builds, we present BrickWright, a framework for generating physically feasible brick builds from text or image prompts using pre-trained agents, where a build is physically feasible if it uses only real parts, has no colliding or floating parts, and forms a connected structure if possible. BrickWright guides agents to create builds by writing code with BrickLayer, a Python toolkit with two main components: an ergonomic brick assembly API that halves inference cost compared to working in raw LDraw, and a validator that provides feedback on physical feasibility. Whereas baseline agents achieve physical validity rates ranging from only 9% (GPT-6 Luna) to 69% (GPT-6 Astra), BrickWright-guided agents achieve 100% validity on a sample of 512 prompts, while improving prompt adherence for lightweight agents and maintaining it for stronger agents. Additionally, they outperform specialized models on the text-to-brick task. Code is available on our project page: https://avalovelace1.github.io/brickwright/