Apples on the Table? Evaluating Text-Guided 3D Scene Synthesis via Fine-Grained Constraint Verification

AI summary

Overview

Research area: Natural Language Processing / embodied AI — specifically the evaluation of text-guided 3D scene synthesis (turning a written room description into a 3D indoor environment).

Technical level: Intermediate. The paper assumes familiarity with VLMs, 3D simulators, CLIP-style similarity metrics, and precision/recall/Cohen's κ, but its central idea (check every requirement one at a time with tools rather than eyeballing a picture) is easy to follow.

Scope: The paper introduces LEGO, a human-curated benchmark of 130 scene descriptions with 1,250 annotated constraints and matching reference scenes, plus LEGO-Eval, a tool-augmented VLM evaluation framework, and uses both to benchmark existing scene synthesis methods.

What This Paper Is About

Text-guided 3D scene synthesis systems are supposed to build a 3D room that matches a user's written description — how many apples, where the door is, what color the walls are. Current evaluation methods either compare the description to a picture of the scene with a coarse similarity score (CLIPScore) or ask a vision-language model to judge it, and neither can reliably check fine-grained requirements like "the cup is on the red table." The paper's goal is to create a dataset of descriptions paired with explicitly annotated constraints and reference scenes, and an evaluation framework that decomposes a description into individual constraints and verifies each one using tools that ground text to 3D objects and reason about spatial relations.

Key Contributions

1. LEGO dataset. A fully human-curated dataset pairing each natural-language scene description with fine-grained annotated constraints and a manually constructed reference scene, containing 130 descriptions with 1,250 constraints total (averaging 9.6 per description), plus rich scene metadata including positions, orientations, scales, and material properties.

2. LEGO-Eval framework. A tool-augmented evaluation framework that decomposes a description into atomic constraints, plans and executes tool calls to ground each constraint in the 3D scene, judges each constraint, and aggregates per-constraint results into a final judgment and report.

3. Empirical validation against humans. A comparison showing LEGO-Eval agrees with human judgments far better than CLIPScore, VLM-as-a-judge, and SceneEval, including an ablation over tool types and an error-type analysis.

4. Benchmarking of synthesis methods. An evaluation of five LLM-based 3D scene synthesis methods on LEGO revealing that none exceeds a 10% holistic success rate, with a human evaluation corroborating the results.

Main Findings

• LEGO-Eval beats all baselines on human agreement. With GPT-4.1 as backbone, LEGO-Eval reaches holistic F1 0.81, precision 0.84, recall 0.82, Cohen's κ 0.63, and partial F1 0.83, precision 0.86, recall 0.81, Cohen's κ 0.66. For comparison, CLIPScore scores holistic F1 0.49 and κ 0.02; VLM-as-a-judge with Gemini 2.5 Pro, GPT-4.1, and GPT-5.1 scores holistic F1 0.38, 0.40, and 0.38 respectively; SceneEval with GPT-4o scores holistic F1 0.47 and κ 0.15. The paper notes SceneEval cannot assess 41% of LEGO's constraints due to its fixed evaluation criteria, and was therefore evaluated only on the constraints it can handle.

• Even a lightweight backbone works. LEGO-Eval with Qwen3-VL-8B achieves holistic F1 0.67 and partial F1 0.71, still outperforming every baseline; Qwen3-VL-32B reaches holistic F1 0.67 and partial F1 0.80.

• Current synthesis methods fail almost completely. On LEGO, holistic success rates are 10.0% (LayoutVLM), 8.4% (Holodeck), 6.9% (LayoutGPT), 5.4% (MANSION), and 3.8% (I-Design). No method exceeds 10%. Average partial success rates are higher — 60.6% (LayoutVLM), 58.5% (Holodeck), 58.2% (MANSION), 55.2% (LayoutGPT), 34.2% (I-Design) — but object selection and object placement sub-scores fall below 50% for Holodeck (46.3 and 43.7), LayoutGPT (40.9 and 37.3), and LayoutVLM (49.8 and 46.0), with I-Design very low at 11.0 and 4.1.

• Methods collapse on complex descriptions. Categorizing descriptions by constraint count into simple (2–7 constraints), moderate (8–12), and complex (more than 12), the paper reports that existing methods consistently fail on complex descriptions. User-written room descriptions contain an average of 18.2 constraints per room.

• Human evaluation confirms the low scores are real, not artifacts of a strict evaluator. Across 20 randomly sampled scenes per synthesis method judged by three undergraduate evaluators (inter-evaluator agreement 80.3%), LEGO-Eval's agreement with human labels ranges from 72.6% to 89.8%. Human holistic success rates (5.0% for iDesign, 15.0% for LayoutVLM, 5.0% for LayoutGPT, 10.0% for Holodeck, 0.0% for Mansion) track LEGO-Eval's closely.

• More scene information does not fix VLM judges. Giving VLM-as-a-judge the full text scene graph (holistic F1 0.37), or scene graph plus images (0.37), performs no better than images only (0.40) — and all remain far below LEGO-Eval (0.81). SceneEval, which also accesses a structured scene representation, stays at 0.47.

• Adding identification modules makes things worse. Augmenting VLM-as-a-judge with Grounding DINO, LASER, or Point-LLM increased total errors from 376 to 451, 497, and 447 respectively, with Identification errors rising from 150 to 288, 233, and 203. LEGO-Eval produced 119 total errors (53 Identification, 20 Spatial, 46 Attribute).

• Textual tools matter most in the ablation. Disabling multimodal reasoning drops holistic F1 by 0.04% and partial F1 by 1.02%; disabling textual reasoning drops them by 5.05% and 2.65%; disabling textual plus multimodal reasoning drops them by 6.46% and 2.81%; disabling environment interaction plus multimodal reasoning drops them by 24.90% and 5.34%.

• LEGO-Eval works as a refinement signal. Feeding LEGO-Eval's rationale back to Holodeck over 3 refinement iterations produced greater improvements than using VLM-as-a-judge as the feedback signal, with GPT-4.1 as the baseline VLM judge.

Methodology in Plain English

The authors first built the dataset by hand. Four annotators with a two-hour training session each wrote 30 to 50 fine-grained room descriptions based on their own experience and real-world indoor images, with explicit viewpoints specified so that "left" and "right" are unambiguous. They then marked every requirement implied by each description using a custom annotation tool plus Label Studio, restructuring dependent requirements into conditional statements (for example, "the bed in the room with the red floor is white") and merging multiple attributes of the same entity into one constraint. Each constraint was labeled as one of four types: floor layout, material selection, object selection, or object placement. To build the reference scene, annotators started from Holodeck's JSON scene output, sourced object instances from Objaverse and structural elements from ProcTHOR and AI2-THOR asset libraries, then manually edited the scene until every constraint was satisfied. Cross-validation between annotators ran for two iterations.

The evaluation framework, LEGO-Eval, works in five steps. First, a VLM judge splits the description into atomic constraints and assigns each a type. Second, it builds a graph-structured plan of which tools to call for each constraint, reusing results from earlier constraints when possible. Third, it selects arguments for each tool — grounding the specific objects the constraint refers to — and executes them. Fourth, it issues a binary judgment and an explanation for that constraint. Fifth, it aggregates all per-constraint judgments into a final judgment, which is valid only if every individual constraint passed, plus a per-constraint report. The framework is built on the AI2-THOR simulator and equipped with 21 tools in three groups: environment interaction (Unity-level access, e.g. top-down scene rendering, spatial relations), textual reasoning (attribute lookup from the structured scene representation), and multimodal reasoning (VLM-based visual-to-text conversion).

To test LEGO-Eval, the authors expanded the 130 descriptions into 260 pairs by manually crafting a misaligned scene for each description, then compared LEGO-Eval's judgments against human labels using F1, precision, recall, and Cohen's κ at both holistic (whole description) and partial (per constraint) levels. For the synthesis benchmark, they ran five methods — LayoutGPT, Holodeck, I-Design, LayoutVLM, and MANSION — augmenting the three that only place objects (I-Design, LayoutGPT, LayoutVLM) with Holodeck so they produce complete scenes.

Why This Matters

Impact on research. The paper argues that misalignment between a synthesized scene and its description injects misleading supervision into embodied-agent training, shifting the training distribution away from the user's intent and confounding agent failures with environment errors. By producing constraint-level diagnoses rather than a single similarity number, LEGO and LEGO-Eval give the field a way to say where and why a scene is wrong. The paper also shows that the conventional fix — giving a judge more scene information or stronger identification modules — does not help, which reframes the problem as one of deciding what information each constraint needs.

Real-world applications:

• Training embodied agents and robot policies in simulators where the scene must match the task description, reducing the sim-to-real gap.
• Large-scale generation of scene assets for robotics and navigation research, where manual scene authoring is infeasible at the required scale.
• Quality control for interior design and architectural visualization tools that turn a written brief into a 3D layout.
• Automated verification that a generated or edited 3D environment matches a customer's or designer's stated specification before it ships.

Industry relevance. Companies building simulation environments, synthetic data pipelines, or 3D content generation tools need automatic, trustworthy checks that a generated scene matches the brief. The paper's finding that even a lightweight model (Qwen3-VL-8B) exceeds all baselines when given the tool-augmented framework suggests the approach is deployable without frontier-scale models — though the authors note that evaluating the full LEGO benchmark takes approximately two hours.

Future Directions

• Richer scene geometry. LEGO currently focuses on scenes composed of rectangular or square rooms; extending the benchmark to more diverse spatial layouts and scene descriptions is flagged as future work.

• Simulator-independent evaluation. The environment interaction and multimodal reasoning tools are simulator-dependent, so when the simulator is unavailable, evaluation must rely only on textual tools. Making LEGO-Eval work without simulator-specific tools is called out as an important direction.

• Efficiency. Evaluating the LEGO benchmark with LEGO-Eval takes roughly two hours; faster model inference and more efficient tool execution are identified as needed improvements.

• Closing the capability gap. Since no current synthesis method exceeds a 10% holistic success rate, an open question is whether the fine-grained feedback LEGO-Eval produces — which improved Holodeck over 3 refinement iterations — can be scaled into a training signal that meaningfully raises holistic success rates on complex descriptions.

Target Audience

Researchers and engineers working on 3D scene generation, embodied AI simulation, and vision-language model evaluation. It is most useful for those who need to measure whether a generated 3D environment matches a text description, and for those building training data pipelines where scene-description fidelity directly affects downstream agent learning. Practitioners evaluating VLM judges or designing tool-augmented evaluation frameworks will also find the negative results (more information and better identification modules do not help) directly relevant.

Authors’ abstract

Accurately synthesizing 3D scenes from user-provided text descriptions is crucial for developing embodied agents. Despite the importance of scene-description alignment, existing evaluation methods for such text-guided 3D scene synthesis either capture only coarse similarity between the synthesized scene and the user description, or ignore the spatial reasoning for verifying object placement. None of them addressed the fine-grained constraints (e.g., X needs to be in the scene in a Y manner) implied by the description from users. To address this, we introduce LEGO, a benchmark dataset that pairs each user description with human-annotated constraints and a reference scene, and LEGO-Eval, an evaluation framework that decomposes a description into atomic constraints and verifies each one using tools that ground textual references to 3D objects and reason about their spatial relationships. We show that (i) LEGO-Eval evaluates misalignment far more accurately than existing methods and (ii) current scene synthesis approaches achieve at most 10% success rate in LEGO-Eval.