Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansion

AI summary

Overview

Research area: Computer Vision / 3D scene generation, specifically text-driven generation of city-scale 3D environments using agentic (multi-agent) pipelines built on large language models and vision-language models.

Technical level: Advanced. The paper is written for readers familiar with diffusion models, image-to-3D generation, LLM/VLM agent design, and 3D scene assembly.

Scope: The paper proposes Yo'City, a multi-agent framework that turns a single user text prompt into a personalized, hierarchical 3D city scene and then expands that city indefinitely through user interaction.

What This Paper Is About

Most existing 3D city generation methods train or deploy a single diffusion model, which makes it hard to honor personalized user requests and hard to scale a scene beyond a fixed, bounded area. Yo'City instead treats city generation as a "planning-generation-expansion" problem solved by off-the-shelf large models: an LLM plans the city, a VLM-guided loop synthesizes each grid, and a scene-graph optimization decides where new districts should be added as the city grows.

Key Contributions

1. Yo'City framework. A multi-agent framework for boundless and realistic 3D city generation guided by user-customized textual instructions.
2. Hierarchical grid-based planning. The city is modeled with a "City-District-Grid" structure, and a top-down coarse-to-fine planning strategy (Global Planner followed by Local Designer) produces spatially coherent urban layouts.
3. Self-critic expansion mechanism. A scene-graph-based module performs distance- and semantics-aware location optimization, enabling automated and plausible city expansion without user-specified coordinates.
4. Multi-dimensional benchmark and metrics. A dataset of 100 city textual descriptions plus evaluation that assesses semantic consistency and visual quality across five visual aspects: geometric fidelity, texture clarity, layout coherence, scene coverage, and overall realism.

Main Findings

• Highest semantic consistency by VQAScore. Yo'City scores 0.7151, compared with 0.6975 for SynCity, 0.6198 for Hunyuan3D (API), 0.6189 for Trellis, and 0.5639 for CityCraft.
• Consistent win rates in pairwise visual comparisons. Against each of the four baselines, Yo'City's win rates span 75.00% to 96.50% across the five visual dimensions, judged by GPT-5 and by ten human judges with each comparison repeated twice. Against SynCity, for example, GPT-5/human win rates were 85.00%/88.00% (Geometric Fidelity), 78.50%/81.50% (Texture Clarity), 86.00%/89.50% (Layout Coherence), 82.00%/84.50% (Scene Coverage), and 84.50%/88.00% (Overall Realism).
• Stronger grid-level quality. In a grid-level experiment against SynCity, Yo'City reached an Alignment Score of 0.6927 versus 0.6572, and an Aesthetic Score of 5.52 versus 4.95 (gain of +0.0355 and +0.57 respectively).
• Coarse-to-fine planning helps. Removing the planning strategy changes VQAScore from 0.7034 to 0.7151, Layout Coherence win rate from 27.00% to 73.00%, and Overall Realism from 24.50% to 75.50% when the planning strategy is included.
• Expansion is stable. Across four expansion steps for each of five cities, VQAScore stayed stable with a Coefficient of Variation of 3.34%. Measured across expansion strategies, CV was 8.76% for random placement, 5.16% for semantics-only placement, and 3.34% for the proposed method.
• Qualitative advantages. Yo'City produces well-proportioned buildings with clear textures and high-fidelity details (e.g., windows), while CityCraft generalizes poorly beyond modern-city prompts and SynCity shows imbalanced spatial distribution and coarse textures. Yo'City also captures fine-grained stylistic cues such as "sharp-edged", "Silk Road", and "stacked structures".

Methodology in Plain English

The system splits the job into four cooperating modules.

Global Planner. An LLM reads the user prompt and estimates the city's size as an H×W grid. It then identifies N functional districts, writes short blueprints for each, and places those districts onto the grid while respecting proximity and adjacency constraints. A Retrieval-Augmented Generation step pulls information about a referenced real city (for example a "New York-like" request) from a Wikipedia corpus, distills it with GPT-4o-mini, and feeds those structural and functional traits into the plan.

Local Designer. For each district, the LLM takes the district blueprint plus the global prompt and writes a detailed description for each grid, including spatial organization and visual character. All grids inside a multi-grid district are planned jointly so they stay stylistically continuous. This coarse-to-fine split gives the model an implicit reasoning process from high-level organization down to fine detail.

3D Generator. Each grid description becomes a 2D isometric image through a produce-refine-evaluate loop: generate the image on a fixed ground platform to anchor scale and alignment, let an image editing model remove the platform and clean up surfaces, then have an evaluator score text-image alignment, realism, and layout rationality and send feedback back for regeneration. The loop runs up to three iterations, with scores from 0 to 10 and a threshold of at least 6 for acceptance; below 6, a VLM rewrites the instruction using the negative feedback. The accepted image is lifted to 3D by a pretrained image-to-3D model. Because the pipeline is grid-aligned and parallel (not tile-by-tile autoregressive), models can be placed directly according to the planner's layout without complex 3D blending. Roads, ground, and other elements are then added and scaled, with customizable colors and roughness.

Relationship-guided Expansion. A VLM inspects the rendered city and writes a description for the new grid plus a scene graph whose central node is the new grid and whose edges encode qualitative distances ("near", "relatively near", "slightly near", "no special constraint", "far"). Breadth-first search enumerates feasible candidate locations. A distance-driven objective weights Euclidean distances by signed relationship weights, so the new grid is pulled toward regions it should be near and pushed away from those it should be separated from. A semantic regularization term uses Sentence-Bert embedding similarity between the new description and neighboring grids. The two terms are combined with a balance parameter, and the location minimizing the total objective is chosen.

Why This Matters

The work shows that city-scale 3D content can be produced and then extended by composing off-the-shelf large models rather than training a new generative model, and that a hierarchy mirroring real urban structure ("City-District-Grid") mitigates the global inconsistency and error accumulation that plague flat tile-by-tile pipelines. It also provides a benchmark with a defined dataset and metric set for a task that previously had no standard evaluation.

Real-world applications the paper points to:

• Virtual reality and simulation games.
• Digital twins of urban environments.
• Urban planning.
• Robotics.

Industry relevance: the framework needs no map or satellite training data, accepts flexible natural-language input, supports iterative expansion, and allows appearance customization (road and ground colors, road connectivity, "ancient" versus "modern" themes), which suits content pipelines, simulation platforms, and interactive design tools that need large, on-demand 3D environments.

Future Directions

• Explicit cross-grid 3D interaction. The paper deliberately skips explicit cross-grid interaction in 3D generation to avoid computational overhead, relying instead on consistent per-grid descriptions and evaluator-triggered regeneration. Stronger inter-tile 3D coherence remains an open problem.
• Scale of evaluation. The benchmark covers 100 city descriptions; expanding to larger and broader prompt distributions would test generalization further. The paper notes that in practice the method can accommodate paragraph-level inputs beyond what the baseline-constrained dataset contains.
• Robustness beyond curated references. RAG currently draws on a curated Wikipedia corpus; how the planner behaves for fictional or hybrid city references is not reported.
• Human-in-the-loop expansion control. Expansion decisions are currently made by a VLM plus a distance and semantics objective; allowing users finer control over which relationships are prioritized is a natural extension.

Target Audience

Researchers and practitioners in 3D scene generation, AIGC, and city-scale simulation; engineers building agentic pipelines that combine LLMs, VLMs, image editing, and image-to-3D models; and developers in VR, gaming, digital twins, and urban planning who need personalized, expandable 3D city content without map or satellite data. Readers should be comfortable with diffusion-based generation, LLM/VLM agent design, and 3D asset assembly.

Authors’ abstract

Realistic 3D city generation is fundamental to a wide range of applications, including virtual reality and digital twins. However, most existing methods rely on training a single diffusion model, which limits their ability to generate personalized and boundless city-scale scenes. In this paper, we present Yo'City, a novel agentic framework that enables user-customized and infinitely expandable 3D city generation by leveraging the reasoning and compositional capabilities of off-the-shelf large models. Specifically, Yo'City first conceptualizes the city through a top-down planning strategy that defines a hierarchical "City-District-Grid" structure. The Global Planner determines the overall layout and potential functional districts, while the Local Designer further refines each district with detailed grid-level descriptions. Subsequently, the grid-level 3D generation is achieved through a "produce-refine-evaluate" isometric image synthesis loop, followed by image-to-3D generation. To simulate continuous city evolution, Yo'City further introduces a user-interactive, relationship-guided expansion mechanism, which performs scene graph-based distance- and semantics-aware layout optimization, ensuring spatially coherent city growth. To comprehensively evaluate our method, we construct a diverse benchmark dataset and design six multi-dimensional metrics that assess generation quality from the perspectives of semantics, geometry, texture, and layout. Extensive experiments demonstrate that Yo'City consistently outperforms existing state-of-the-art methods across all evaluation aspects.