Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models

AI summary

Overview

Research area: Robotics, specifically text-driven robotic assembly, 3D generative AI, and vision-language model (VLM) reasoning for design-to-fabrication pipelines.

Technical level: Intermediate — the paper is a systems-integration paper rather than a deep-learning-methods paper. It combines existing generative and multimodal models with an industrial robot arm, and the core ideas are understandable without specialized math.

Scope in one sentence: The paper presents an end-to-end pipeline that converts a natural language prompt into a multi-component 3D model and then physically assembles it with a robot, using a VLM to decide which parts of the AI-generated mesh need panel components based on the object's function and geometry.

What This Paper Is About

3D generative AI can create shapes from text prompts, but its output is typically a single monolithic mesh with no notion of parts — which makes it unusable for robotic assembly, where the robot needs to know what each piece is and where it goes. The authors tackle the harder case of objects made from multiple component types with different functions (here: structural frames and functional panels), where deciding which regions need panels depends on what the object is for (a chair seat needs a flat panel; a lamp shade needs a light-diffusing panel). Their goal is a pipeline where a user types a prompt, a VLM reasons about the object's geometry and function to decompose the mesh into assemblable components, and a robot builds the result — with the user able to correct the VLM through conversation.

Key Contributions

1. A function- and geometry-aware decomposition method. A VLM performs zero-shot multimodal reasoning over the user's text prompt, an axonometric image of the AI-generated mesh, and the component type, to decide which mesh regions receive panel components. No task-specific training is used.

2. A two-stage VLM task chain. The first VLM task returns the parts that need panels (e.g., "Parts = seat, backrest"); the second maps those named parts onto labeled faces in the discretized mesh (e.g., "Labels = 4, 6") so the robot knows physical locations.

3. A conversational human-in-the-loop workflow. Users can issue natural-language feedback (e.g., "I want panels on the seat") and the VLM regenerates component assignments, letting users refine or override the AI without task-specific training.

4. An end-to-end demonstration. Natural language → 3D generative AI → VLM assignment → robotic assembly using a UR20 arm with Robotiq grippers, evaluated in a 32-participant user study.

Main Findings

• VLM assignments were strongly preferred. Across five objects and 32 participants, the VLM approach had a mean selection rate of 90.6%, compared to 59.4% for the rule-based approach and 2.5% for random assignment.

• Per-object performance (participants selecting / 32). VLM: Chair 96.9% (31), Table 100.0% (32), Lamp 81.3% (26), Shelf 100.0% (32), Trash Can 75.0% (24). Rule-based: Chair 18.8% (6), Table 100.0% (32), Lamp 34.4% (11), Shelf 100.0% (32), Trash Can 43.8% (14). Random: Chair 0.0% (0), Table 0.0% (0), Lamp 0.0% (0), Shelf 6.3% (2), Trash Can 6.3% (2).

• Where the rule-based baseline broke down. The rule-based method (assign panels to all upward-facing surfaces) matched the VLM on objects dominated by horizontal surfaces — the table and shelf, both 100.0% — but failed on the chair (18.8%), lamp (34.4%), and trash can (43.8%).

• Statistical significance. Pairwise McNemar tests gave χ² = 38.11 for VLM vs. rule-based, χ² = 137.11 for VLM vs. random, and χ² = 88.17 for rule-based vs. random, all with p < 0.001 and all remaining significant after Bonferroni correction (α = 0.05/3 ≈ 0.0167). Discordant counts were b = 56 / c = 7 (VLM vs. rule), b = 143 / c = 2 (VLM vs. random), and b = 94 / c = 2 (rule vs. random), over 160 trials per comparison.

• Users wanted to diverge from the AI. Participants proposed alternative panel placements outside the three given methods — panels on the chair seat but not the backrest, on the lampshade but not the lamp base, and only on the bottom two tiers of the shelf — indicating multiple valid solutions rather than a single ground truth. The authors attribute this to design problems having preference-dependent answers.

• The robot respected fabrication constraints. The robot successfully executed grasp configurations and assembly sequences from VLM outputs and user-refined outputs, and did not place any inward- or downward-facing panels. This worked because labels for inward-facing vertical and downward-facing horizontal faces were removed from the input image beforehand, since the arm could not reach them.

• Note on a reported inconsistency. The main text and Table 1 describe 32 participants and 480 judgments, while checklist item 14 states "25 participants." The counts across the five objects in Table 1 are consistent with 32 participants.

Methodology in Plain English

Starting point. A user types a prompt. Autodesk's 3D generative AI model turns it into a mesh.

Two component types. Everything is built from a fixed library: structural components (load-bearing volumetric cubes that form the frame) and panel components (flat planes that attach to the frame and provide functional surfaces). The structural cube has magnets arranged in a polarity pattern on all six sides so neighboring pieces self-align; the panel has the matching magnet pattern; both have features the gripper can grab.

Discretization. The generated mesh is voxelized at a fixed grid resolution sized to the structural components, producing the load-bearing frame. Panels are not attached everywhere — the authors note that indiscriminate panels add unnecessary weight and waste components.

First VLM task (function-aware part selection). Given (1) the user's object description, (2) an axonometric image of the mesh, and (3) the component type (panel), the VLM returns the minimal set of part names that need panels. Example: prompt "I want a chair" returns "Parts = seat, backrest." The model used is Google's Gemini 2.5 Pro.

Second VLM task (geometry-aware label mapping). Coplanar faces in the discretized mesh are merged and each face gets a unique integer label. Given (1) the description, (2) a labeled axonometric rendering showing both sides, and (3) the part list, the VLM maps parts to face labels, e.g., "Labels = 4, 6." Those labels become the panel placements in the 3D assembly.

Third VLM task (conversational alignment). The same inputs plus free-form user feedback; the VLM outputs updated labels, which regenerate the multi-component assembly. This is how a user can override the AI's initial choices.

Robotic assembly. The assembly is exported as a coordinate list C of (x, y, z, rx, ry, rz) tuples and a component type list T = {t₀, t₁}, where tᵢ = 0 means structural (picked from source sᵢ = 0, a conveyor belt) and tᵢ = 1 means panel (picked from source sᵢ = 1, a stack). The coordinate list is sorted bottom-to-top while preserving connectivity so physically connected pieces are placed consecutively. The robot moves to the correct source, picks, moves to the component coordinate, and places. Hardware: a UR20 robotic arm with Robotiq grippers.

Evaluation. Thirty-two participants each judged component assignments for five objects across three methods (VLM, rule-based, random), yielding 480 judgments. The rule-based baseline placed panels on all upward-facing surfaces; the random baseline picked random face labels. Participants could select all acceptable options or none, to avoid forced-choice bias. Selection rate was computed as the number of participants selecting a method on an object divided by 32, times 100%. The study received an IRB exemption determination (Exemption Category 3) from MIT COUHES; participants were adults who gave informed consent and received no compensation.

Why This Matters

Research impact. The paper shows that a general-purpose VLM, used zero-shot, can serve as the "reasoning layer" between generative 3D models and physical fabrication — bridging two literatures (part-aware 3D generation and VLM-guided robotic assembly) that have largely developed separately. It also argues that the hard part of assembly decomposition is not purely geometric but functional and preference-dependent, which reframes the problem as one of human-AI co-creation rather than pure automation.

Real-world applications (as motivated or demonstrated in the paper):

• Accessible fabrication for non-experts — letting people without 3D CAD or manufacturing expertise get a physical object from a text prompt.
• Modular construction with component reuse — because the output is part-based, parts can be reused and edited at the part level, unlike a monolithic 3D print.
• Furniture and lighting fabrication — the demonstrated objects are a chair, table, lamp, shelf, trash can, wide bowl, and round coffee table, with functional panel logic such as flat seat surfaces and light-diffusing lampshades.
• Robotic assembly planning — the pipeline produces coordinate/type lists and bottom-to-top connectivity-preserving sequences that an industrial arm can execute directly, which is reusable for other pick-and-place assembly tasks.

Industry relevance. The work is a collaboration including Google DeepMind, Google Paradigms of Intelligence, and Autodesk Research, and the pipeline is built on Autodesk's 3D generative model and its internal robotics research platform integrated with Fusion 360. That points to near-term relevance for CAD/design software vendors and for the robotics-integrated construction and furniture industries. The authors are transparent that the code is closed-source and under active research, while the VLM prompts and assembly algorithm are published.

Future Directions

1. Expand the component library. The current implementation is restricted to two component types. The authors propose adding functional components such as hinges and handles, and material-specific components such as wood, plastic, or metal.

2. Test with harder prompts and objects. Evaluation is limited to common objects and simple prompts; the authors call for exploring complex user prompts or unconventional objects, which may require multi-turn human-AI interaction with larger edit distances.

3. Formalize multi-turn interaction. Conversational feedback is currently a single refinement step. The paper raises the open question of how to support iterative co-creation where users diverge substantially from the initial AI output.

4. Generalize the VLM role. The VLM is used only for component assignment; the authors leave open whether the same zero-shot reasoning can be extended to other parts of the assembly pipeline, and whether the approach transfers to robot platforms or component systems beyond the UR20 setup used here.

Target Audience

This paper is best suited to robotics and fabrication researchers working on assembly planning and human-robot collaboration; researchers in generative AI and HCI interested in AI-assisted physical making and co-creation workflows; and practitioners in CAD, construction, and industrial automation who want to see a working example of a text-to-physical-object pipeline. Designers and architects curious about where generative AI and robotic fabrication intersect would also find the demonstration useful, and the human-subjects methodology (non-exclusive selection, McNemar testing with Bonferroni correction) is directly reusable by anyone running comparative design studies.

Authors’ abstract

Advances in 3D generative AI have enabled the creation of physical objects from text prompts, but challenges remain in creating objects involving multiple component types. We present a pipeline that integrates 3D generative AI with vision-language models (VLMs) to enable the robotic assembly of multi-component objects from natural language. Our method leverages VLMs for zero-shot, multi-modal reasoning about geometry and functionality to decompose AI-generated meshes into multi-component 3D models using predefined structural and panel components. We demonstrate that a VLM is capable of determining which mesh regions need panel components in addition to structural components, based on the object's geometry and functionality. Evaluation across test objects shows that users preferred the VLM-generated assignments 90.6% of the time, compared to 59.4% for rule-based and 2.5% for random assignment. Lastly, the system allows users to refine component assignments through conversational feedback, enabling greater human control and agency in making physical objects with generative AI and robotics.