Scene-Designer: Learning Design Intelligence for Agentic 3D Scene Generation

AI summary

Overview

• Research area: 3D indoor scene generation from natural-language instructions, combining vision-language models, agentic tool use, and reinforcement learning (GRPO).
• Technical level: Advanced. It assumes familiarity with vision-language models, supervised fine-tuning, GRPO/preference optimization, and 3D scene representations (object poses, collisions, out-of-bound checks).
• Scope: The paper proposes Scene-Designer, an agentic framework that learns both to design 3D indoor scenes and to critique their quality, trained on two new datasets (Scene-OTA-35K and Scene-OTC-15K) and optimized with a critic-guided GRPO variant called Scene-GRPO.

What This Paper Is About

Generating a complete, usable 3D room from a text prompt is not just object placement — it is a design problem. Current systems either prompt general-purpose models (like GPT-5.2) with scene-construction tools, or learn from static scene datasets that record only final object layouts, never the decisions and corrections that produced them. Scene-Designer addresses this by teaching a model two complementary skills — designing scenes and judging scene quality — from construction trajectories and automatically generated preference comparisons, then using the learned critic to improve design decisions.

Key Contributions

1. Two complementary datasets for design intelligence. Scene-OTA-35K provides 1K scenes and 35K Observation–Thought–Action steps for learning scene construction and efficient error correction; Scene-OTC-15K provides roughly 15K Observation–Thought–Critic preference examples for learning scene critique along physical validity, functional correctness, and layout coherence.
2. Causal Trajectory Consolidation. A method that strips ineffective trial-and-error from raw GPT-5.2 construction trajectories while preserving the final scene state, reducing average trajectory length from about 110 to 30 steps.
3. Scene-GRPO. A critic-guided GRPO framework in which a frozen Critic supplies a branch-ranking reward (comparing sibling candidate scenes) and an improvement reward (comparing each child scene to its parent), plus format, length, and object-completeness auxiliary rewards.
4. A trained critic that also helps at inference. A joint SFT stage produces Scene-Builder; the learned Scene-Critic is used both as a training reward and as a test-time branch selector, yielding further gains (Scene-Designer w/ Critic).

Main Findings

• General models correct inefficiently. In an analysis of 100 scenes generated by GPT-5.2, many errors that could be fixed in one or two operations required long corrective sequences — for example, roughly 18% of window-accessibility blockages still needed more than five corrective moves.
• General models critique weakly on harder rankings. In the diagnostic study, GPT-5.2 reached around 72% accuracy in two-scene pairwise comparison but around 50% in four-scene preference ranking. In the later critic evaluation (Fig. 10a), GPT-5.2 is reported at 72% for n=2 and 15% for n=4; the base Qwen3-VL-8B-Instruct model is reported at 72% and 2%.
• Fine-tuning sharply improves critique. OTC-only fine-tuning reaches 86% (n=2) and 51% (n=4); joint Scene-OTA+OTC training reaches 89% (n=2) and 76% (n=4), with 82% Top-1 and 76% Top-4 accuracy on the four-scene set.
• Critic rankings are stable to input order. Over five independent image-label shuffles, the critic scores 90.00 ± 2.07% on n=2 and 75.64 ± 6.88% on n=4, close to the unshuffled results.
• Correction efficiency improves substantially. From Scene-Builder to Scene-Designer, the share of cases needing five or more corrective moves drops from 15% to 13% for door clearance occupation, 21% to 5% for window accessibility blockage, 40% to 7% for interaction orientation error, and 50% to 23% for scale mismatch.
• Scene quality improves across room types. On Bedroom and Living Room (Table 1), Scene-Designer reaches 20.0 and 20.3 objects with zero out-of-bound objects and zero collisions, and GPT-4 visual-semantic scores of 9.5/9.5/8.8/9.4 (Real./Func./Lay./Comp.) and 9.6/9.0/9.3/9.0; Scene-Designer w/ Critic reaches 20.8 and 21.1 objects with 9.8/9.8/9.2/9.7 and 10.0/9.4/9.6/9.5.
• Gains hold on other room types. On Bathroom and Office (Table 2), Scene-Designer w/ Critic reaches 22.0 and 24.2 objects, zero out-of-bound objects and collisions, and 9.7/9.5/9.1/9.7 and 9.8/9.4/9.2/9.6 on the four GPT-4 dimensions, improving on Scene-Designer's 21.0 and 23.0 objects and 9.4/9.2/8.7/9.4 and 9.5/9.0/8.7/9.3.
• Every reward component matters. The reward ablation averaged over Bedroom, Living Room, Bathroom, and Office shows that removing any reward degrades performance, with the branch-ranking reward causing the largest drop and the improvement reward also contributing.
• Design and critique reinforce each other. Adding OTC supervision to OTA training produces scenes that better capture design intent in qualitative examples, and the critic is far stronger when trained jointly than with OTC alone — evidence of mutual enhancement between designing and critiquing.

Methodology in Plain English

The work proceeds in three layers.

Building design experience. The authors use the SceneSmith construction scaffold driven by GPT-5.2 to produce scene-building trajectories, then manually select high-quality scenes. Because raw trajectories are full of dead ends, they consolidate them: each operation is checked for whether it moved the scene closer to the final accepted state or reduced the number of remaining errors, and a GPT-5.2 judge decides whether flagged operations were genuinely useless or necessary intermediate steps. Only useless operations are deleted, and the shortened trajectories are replayed to confirm the final scene is identical. Step-level reasoning traces are then generated for the retained actions.

Building critique experience. Starting from those high-quality scenes, they apply three severity levels of perturbation — mild layout deviations that hurt coherence, functional degradations that break functional relationships, and severe violations such as collisions and out-of-bound placements. Rankings are produced automatically when perturbations accumulate or vary in strength on the same object, and manually when the same perturbation hits different objects whose functional roles differ. GPT-5.2 writes reasoning traces from renderings, giving Observation–Thought–Critic data. Test sets of 94 two-scene comparisons and 55 four-scene rankings are held out.

Training. A joint supervised fine-tune of Qwen3-VL-8B-Instruct on both datasets produces Scene-Builder. Then Scene-GRPO initializes a frozen Critic and a trainable policy from that model; at each editing step the policy samples four candidate branches, the Critic ranks sibling branches and compares each child to its parent, and these signals plus format, length, and completeness rewards drive GRPO updates. Evaluation uses the prompt "Design me a {room_type}", 10 scenes per room type, objective physical metrics (#Obj, #OB, #CN), and GPT-4 scores from 0 to 10 on realism, functionality, layout quality, and completeness, compared against ATISS, DiffuScene, PhyScene, LayoutGPT, Holodeck, I-Design, and SceneWeaver.

Why This Matters

Impact on research. The paper reframes 3D scene generation as a design problem with a learnable critique component, and shows that process data (construction trajectories, corrections) carries supervision that static layout datasets such as 3D-FRONT and 3D-SynthPlace do not. It also demonstrates that a small open 8B model, once trained, can outperform API-based general models on scene critique, and that critique knowledge transfers back into generation.

Real-world applications:

• Automated interior/spatial design tools that propose layouts and explain why one layout beats another.
• Virtual content and game/simulation asset creation, where rooms must be physically valid and functionally sensible.
• Embodied AI and robotics simulation, which need coherent, collision-free indoor environments for training and testing.
• AR/VR and digital-twin authoring from short natural-language briefs.

Industry relevance. The approach reduces reliance on expensive general-model API calls at inference and cuts the interaction cost of agentic construction loops — both direct cost drivers for any product that iteratively builds 3D environments. A trained critic that can filter candidate edits before committing them is also a practical quality-control mechanism for production pipelines.

Future Directions

• Scaling beyond single rooms. Evaluation covers Bedroom, Living Room, Bathroom, and Office with 10 scenes each; extending to multi-room homes and more categories is an open question.
• Reducing dependence on GPT-5.2 for data creation. Both the reasoning traces and the correction-effect judgments use GPT-5.2, so cheaper or self-generated supervision is a natural next step.
• Closing the design–critique loop further. The paper reports a trend of mutual enhancement but trains in two stages; co-training or iterated critic-policy improvement is unexplored.
• Richer notions of quality. The critic currently scores physical validity, functional correctness, and layout coherence; style, aesthetics, and human-centered preferences are not modeled, and the robustness study covers only five shuffles on the reported test sets.

Target Audience

Researchers and engineers working on 3D scene synthesis, agentic LLM/VLM systems, and reinforcement learning from preference or critic feedback. It is most useful to readers who already understand vision-language model fine-tuning and GRPO, and who want a concrete recipe for turning messy agent trajectories and automatically generated preferences into training signal. Practitioners building spatial-design or simulation-content products will find the evaluation protocol and correction-efficiency analysis directly applicable.

Authors’ abstract

Generating complete, coherent, and usable 3D indoor scenes from natural language requires strong \textbf{design intelligence}: the ability to design high-quality scenes and critique their quality. However, existing approaches either rely on general-purpose models lacking domain-specific design experience or learn from static scenes. In this work, we introduce \textbf{Scene-Designer}, an agentic framework that learns design intelligence through complementary design and critique experiences. We construct two datasets: \textbf{Scene-OTA-35K}, which provides Observation--Thought--Action supervision for learning scene design and efficient error correction, and \textbf{Scene-OTC-15K}, which provides Observation--Thought--Critic supervision for learning scene quality critique across physical validity, functional correctness, and layout coherence. Based on these experiences, we propose \textbf{Scene-GRPO}, a Critic-guided GRPO framework that improves design decisions and reduces unnecessary trial-and-error corrections. Extensive experiments demonstrate that Scene-Designer achieves superior performance in physical validity, functional correctness, and layout coherence, while the learned Scene-Critic significantly outperforms general-purpose models in scene critique. Moreover, test-time Critic guidance further improves scene generation, showing that the Critic can provide useful guidance beyond training. Additional analysis suggests a trend of mutual enhancement between scene design and critique abilities.