SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation

AI summary

Overview

• Research area: Computer vision and graphics — text-to-3D scene synthesis, agentic AI, and Vision-Language Model (VLM) based spatial planning.
• Technical level: Advanced. The paper assumes familiarity with 3D scene representations, VLM agents, the ReAct paradigm, and 3D asset generators.
• Scope: The paper proposes SceneAssistant, a VLM-driven agent that builds open-vocabulary 3D scenes through a closed loop of action execution and rendered visual feedback, evaluated with human studies against Holodeck and SceneWeaver.

What This Paper Is About

Existing text-to-3D scene generators are mostly limited to specific domains (such as indoor rooms) or depend on a fixed vocabulary of predefined spatial relations like "on" or "in front of," so they break down when a user describes unusual objects or layouts. SceneAssistant instead treats a VLM as an autonomous agent that creates objects, manipulates them in 6-DoF, moves the camera, and checks its own work by looking at rendered images of the scene at each step. The goal is unconstrained, open-vocabulary 3D scene synthesis that follows arbitrary natural-language descriptions, plus the ability for users to issue follow-up edits in plain language.

Key Contributions

1. A pure visual-feedback agentic framework for open-vocabulary 3D scene generation, operating in a closed loop that bridges high-level text concepts to concrete 3D scenes.
2. A complete suite of scene-editing and viewpoint-control Action APIs (Create, Duplicate, Delete, Translate, Place, Rotate, Scale, ViewScene, FocusOn, RotateCamera, MoveCamera, GenerateFloorTexture, Finish) that elicits the latent spatial reasoning of modern VLMs without predefined spatial relationships or external layout solvers.
3. Extensive experiments on diverse and unconventional descriptions showing consistent improvements over existing state-of-the-art methods in layout coherence, asset quality, and alignment with complex user intent.
4. Human-agent collaboration and natural-language scene editing, where user instructions are injected via system messages to refine or expand an existing scene.

Main Findings

• Outperforms indoor-specific baselines despite not being designed for indoor scenes: On 8 curated indoor scenes rated by human participants on a 1–10 scale, SceneAssistant scored 6.888 for Layout Correctness and 6.950 for Object Quality, versus SceneWeaver at 5.800 and 6.150, and Holodeck at 4.475 and 4.763. Human Preference Rate was 61.25% for SceneAssistant, 36.25% for SceneWeaver, and 6.25% for Holodeck.
• Wins in open-vocabulary settings as well: Across 22 unconstrained open-vocabulary scenes, SceneAssistant scored 7.600 (Layout Correctness), 7.277 (Object Quality), and 65.00% preference, compared with NoActionAPI at 7.005, 6.591, and 35.91%, and NoVisFeedback at 6.255, 5.673, and 26.82%.
• Direct JSON manipulation hurts more than it helps: The NoActionAPI variant has the same degrees of freedom and the same visual feedback as SceneAssistant, but the cognitive load of generating whole JSON strings caused incorrect object orientation and unaddressed 3D generation failures.
• Visual feedback is essential: Without any feedback loop (NoVisFeedback), the agent struggled with relative scaling and object orientation because it could not perceive or correct spatial misalignments.
• Absolute coordinates beat predefined relations: Holodeck and SceneWeaver place objects flush against walls due to their spatial-relation primitives, while SceneAssistant predicts absolute coordinates, yielding more flexible arrangements and avoiding extraneous default objects such as cabinets or windows.
• Visual prompting and collision checking are both necessary: Removing object label overlays and the coordinate axis HUD (the No Visual Prompting ablation) caused disorganized layouts and missing components, while removing collision-detection feedback (No Collision Check) produced physical interpenetrations the agent failed to resolve. These ablations are reported qualitatively in Figure 6, with no numeric scores given.
• The agent adapts its own termination: In the supplementary material, one trajectory terminates by reaching the maximum step limit (T = 20) and another by explicitly calling Finish.

Methodology in Plain English

SceneAssistant hands a VLM a fixed toolbox rather than asking it to write rendering code. At every step, the VLM receives one rendered image of the current scene, a list of the objects present, and the sequence of actions taken so far — but not previous reasoning or older images, to avoid overloading it. The VLM reasons, then emits a batch of actions: creating objects (powered by a 3D asset generator), duplicating or deleting them, positioning and rotating them, scaling them, generating a floor texture, or moving the camera for a better view. To help it connect what it sees to what it should do, every object is drawn with a text label and a coordinate axis HUD is overlaid on the viewport. Behind the scenes, Blender performs the rendering and assembly. The system automatically lifts objects that sink below the floor at Z = 0 and runs a BVH tree-based collision check, reporting any mesh collisions back to the agent as a system message. Creation and manipulation actions cannot be mixed in the same batch, so the agent always sees a newly generated asset before trying to place it, and it may manipulate at most 3 objects per batch. The loop ends when the agent calls Finish or hits the maximum of 20 steps. Users can also interrupt with natural-language edit requests, which are delivered through the same system-message channel.

Why This Matters

• Research impact: The paper argues against the prevailing assumption that VLMs cannot directly infer spatial arrangements and must rely on external optimizers or predefined spatial primitives, demonstrating that a well-designed action interface plus visual feedback is sufficient for open-vocabulary layout control.
• Real-world applications:
  • Filmmaking and previsualization, where sets must be assembled from arbitrary descriptions.
  • Game development, where open-domain environments are needed on demand.
  • Robotics simulation and embodied AI research, where training environments must be generated quickly.
  • Digital content creation and design prototyping, where non-experts need to build and then edit 3D scenes using natural language.
• Industry relevance: The framework is modular and extensible — the authors note that capabilities such as GenerateFloorTexture can be added seamlessly — and it swaps in commercial-grade components (a Gemini-3.0-Flash backbone and Hunyuan3D asset generation), making it a practical recipe for tooling vendors rather than a standalone research artifact. Code is released at https://github.com/ROUJINN/SceneAssistant.

Future Directions

1. Close the fine-grained perception gap: The authors observe a performance discrepancy between VLMs' strong instruction-following and their occasionally limited fine-grained visual perception, and use human collaboration to compensate — automating that correction remains open.
2. Reduce reliance on explicit collision feedback: The No Collision Check ablation shows the agent cannot reliably avoid interpenetrations from raw visuals alone, pointing to a need for agents with stronger implicit physical reasoning.
3. Scale evaluation beyond human ratings: The current quantitative results rest on human scores over 30 test cases (8 indoor, 22 open-vocabulary) with 10 participants each; automated metrics that account for semantic and aesthetic alignment are not established in this work.
4. Handle 3D asset generator instability more directly: The paper works around stochastic, occasionally poor generations by letting the agent delete and regenerate, but notes that rerunning Create is expensive and unpredictable.

Target Audience

Researchers and practitioners in 3D content generation, computer graphics, and multimodal/VLM agents who are interested in closed-loop, tool-using agents; also useful for technical artists and engineers building natural-language-driven scene authoring tools, and for embodied AI researchers who need to generate diverse environments.

Authors’ abstract

Text-to-3D scene generation from natural language is highly desirable for digital content creation. However, existing methods are largely domain-restricted or reliant on predefined spatial relationships, limiting their capacity for unconstrained, open-vocabulary 3D scene synthesis. In this paper, we introduce SceneAssistant, a visual-feedback-driven agent designed for open-vocabulary 3D scene generation. Our framework leverages modern 3D object generation model along with the spatial reasoning and planning capabilities of Vision-Language Models (VLMs). To enable open-vocabulary scene composition, we provide the VLMs with a comprehensive set of atomic operations (e.g., Scale, Rotate, FocusOn). At each interaction step, the VLM receives rendered visual feedback and takes actions accordingly, iteratively refining the scene to achieve more coherent spatial arrangements and better alignment with the input text. Experimental results demonstrate that our method can generate diverse, open-vocabulary, and high-quality 3D scenes. Both qualitative analysis and quantitative human evaluations demonstrate the superiority of our approach over existing methods. Furthermore, our method allows users to instruct the agent to edit existing scenes based on natural language commands. Our code is available at https://github.com/ROUJINN/SceneAssistant