CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation

AI summary

Overview

• Research area: Computer vision / generative 3D modeling — sketch-guided generation of 3D outdoor semantic urban scenes, with a new benchmark (SketchSem3D) and a new diffusion model (CymbaDiff).
• Technical level: Advanced. The paper assumes familiarity with diffusion models, latent VAEs, state space models (Mamba), and 3D semantic scene completion.
• Scope: It defines a new task (sketch-based 3D outdoor scene generation), builds the first large-scale benchmark for it, and proposes a Mamba-based latent diffusion denoiser that improves spatial coherence.

What This Paper Is About

Generating large-scale 3D outdoor scenes is hard, and progress has been limited by the lack of a public, well-annotated benchmark — existing work tends to use self-curated, heavily preprocessed data. This paper introduces SketchSem3D, a benchmark of LiDAR voxel scenes paired with freehand-like sketches and pseudo-labeled satellite image annotations, and proposes CymbaDiff, a model that generates 3D semantic voxel scenes from those two condition inputs.

Key Contributions

1. A new task: "sketch-based 3D outdoor scene generation," where users control scene layout with freehand sketches plus pseudo-labeled satellite image annotations, reducing reliance on manual semantic annotation. The authors frame this as an efficient way to generate training data for urban-scale simulation and autonomous driving.
2. SketchSem3D benchmark: described as the first public large-scale sketch-based benchmark for 3D outdoor semantic scene generation, with two subsets — Sketch-based SemanticKITTI and Sketch-based KITTI-360 — for standardized benchmarking.
3. CymbaDiff model: a generative model built on a latent diffusion denoising network that incorporates proposed cylinder mamba blocks to enhance spatial coherence during generation.
4. Experiments: extensive evaluation on both subsets showing state-of-the-art performance in 3D semantic scene generation and completion, including cross-dataset generalization without fine-tuning.

Main Findings

• Benchmark scale and format: SketchSem3D contains 95,044 3D scenes in total, compared to 34,149 in the BEV-based NuScenes setup used by UrbanDiff. Table 1 lists 58,987 pairs for Sketch-based SemanticKITTI (20 classes) and 36,057 for Sketch-based KITTI-360 (19 classes), both with 3D GT of 256 × 256 × 32 voxels; BEV-based NuScenes has 17 classes and 192 × 192 × 16 voxels.
• Split sizes: Sketch-based SemanticKITTI has 58,172 training and 815 validation frames; Sketch-based KITTI-360 has 33,892 training and 2,165 validation frames.
• Generation quality on Sketch-based SemanticKITTI: CymbaDiff reaches 3D FID 40.67 and MMD 0.04, versus 165.65 / 0.09 for a 3D Latent Diffusion baseline and 138.86 / 0.08 for 3D DiT. Reference numbers reported for other methods are 2D FID: SSD 112.82, Semcity 56.55. The paper states CymbaDiff improves FID by approximately 16 points compared to Semcity.
• Cross-dataset generalization: trained only on Sketch-based SemanticKITTI and applied to Sketch-based KITTI-360 with no fine-tuning and only the 16 overlapping class labels, CymbaDiff reaches 3D FID 107.53 and MMD 0.08, versus 330.86 / 0.12 (3D Latent Diffusion) and 272.83 / 0.11 (3D DiT).
• Comparison to UrbanDiff: UrbanDiff reports a 3D FID of 291.4 and 3D MMD of 0.11 on NuScenes, but at lower voxel resolution (192 × 192 × 16) and with only 17 semantic classes; its code and preprocessed data are not released, preventing direct comparison.
• Ablation study (Sketch-based SemanticKITTI test set): removing CSCB gives FID 90.53 / MMD 0.06; removing DDCB gives 76.57 / 0.06; retaining only the triple Mamba layers ("w/o C-Mamba") gives 74.09 / 0.05; the full CymbaDiff gives 40.67 / 0.04.
• 3D semantic scene completion (SemanticKITTI validation, Table 4): using only sketch and pseudo-labeled satellite annotations, CymbaDiff attains 43.2% IoU and 14.6% mIoU. The paper states this outperforms the leading monocular baseline by 1.1% mIoU and the best stereo-based method by 0.9% mIoU. Per-class mIoU includes sidewalk 33.3, building 32.4, car 32.1, vegetation 28.0, terrain 22.2, road 52.4, fence 11.2 and pole 12.7.
• Known limitations: the model occasionally fails to reconstruct small or occluded objects that are underrepresented in training data or sparsely encoded in the sketch and PSA inputs. Differences between sketch and PSA pairs can arise because SemanticKITTI ground truth was collected around 2013 while the satellite images used for PSA were captured around 2025.
• Reporting caveat: the text in Section 5.1 states a benchmark resolution of 256 × 256 × 256 and "16 classes for the Sketch-based KITTI-360 subset," which differs from Table 1 (256 × 256 × 32 voxels; 19 classes for Sketch-based KITTI-360). The 16-class figure also appears as the number of overlapping classes used in cross-dataset evaluation.

Methodology in Plain English

• Building the dataset. Sketches are produced by running the Canny edge detector on bird's-eye-view projections of existing 3D ground truth from SemanticKITTI and SSCBench-KITTI-360. Semantic categories are turned into descriptive texts using GPT-4; CLIP encodes those texts and SAM produces mask-level embeddings from GPS-matched satellite images; cosine similarity between text and image embeddings infers the semantic composition, yielding pseudo-labeled satellite annotations (PSA). A manual review of class distributions checks for labeling errors. Each sample bundles a sketch, a satellite image, pseudo-labels, label keywords in a .txt file, and 3D ground truth stored as a volumetric label map with 16-bit unsigned integers following the SemanticKITTI format.
• Overall architecture. A Scene Structure Estimation Network (SSEN) extracts coarse structural information from the sketch and PSA, providing a prior that guides early diffusion steps. A VAE (also used as the Latent Mapping Network) compresses scenes into a latent representation at one quarter of the input spatial resolution (downsampling factor 4), and the CymbaDiff denoiser works in that latent space. The VAE is trained with cross-entropy plus Lovász-Softmax loss.
• Efficiency tricks. Multi-scale modules use parallel 3 × 3 × 3 convolutions instead of larger 5 × 5 × 5 and 7 × 7 × 7 kernels, and Dimensional Decomposition Residual (DDR) blocks split a k × k × k 3D convolution into 1 × 1 × k, 1 × k × 1 and k × 1 × 1 layers, reducing parameters by a factor of three. The Dilated Decomposed Convolution Block uses dilation rates 1, 2 and 3, and the Cross-Scale Contextual Block captures local-to-global context.
• The cylinder mamba idea. The denoiser uses state space (Mamba) sequence modeling. A Triple Mamba module operates on Cartesian grids to preserve precise geometric distances, but adjacent voxels in a Cartesian scan order can misrepresent true spatial relationships in outdoor scenes. The cylinder mamba layer re-orders tokens by angular, radial and vertical indices (θ, r, z), giving a vehicle-centric view that captures cylindrical continuity and vertical hierarchy; outputs are mapped back to Cartesian ordering and fused with the Triple Mamba outputs. Each Mamba layer runs three modules with forward, backward and random inter-slice orderings.
• Evaluation protocol. Following UrbanDiff, a dedicated network extracts latent features encoding geometry and semantics, and these are used to compute 3D FID and MMD against ground-truth scenes; the authors argue these 3D metrics assess voxel-space geometry and semantics better than the 2D FID used by SSD and Semcity.

Why This Matters

• Research impact: the paper defines a new task and supplies the first public large-scale benchmark for it, addressing what the authors identify as a key blocker — the absence of standardized data leading to heterogeneous, non-comparable evaluations.
• Real-world applications:
  • City-scale urban simulation.
  • Autonomous driving, where generated scene data can supplement scarce annotated LiDAR.
  • Remote sensing and other settings where RGB imagery is unavailable or impractical, since sketch plus PSA are easier to obtain at test time than BEV images or sensor data.
  • Synthetic training-data generation that reduces manual semantic annotation effort.
• Industry relevance: the sketch plus satellite-annotation interface is a cheaper, more accessible control modality than BEV maps, and the benchmark's reported cross-dataset generalization (no fine-tuning between KITTI subsets) is directly relevant to deploying models across cities and sensor configurations. The paper states the code and dataset will be released at a public GitHub repository.

Future Directions

• Increasing the representation of small and occluded objects in the dataset, which the authors identify as the main route to improving performance on underrepresented classes.
• Improving annotation quality and addressing the temporal domain gap between 2013-era 3D ground truth and 2025 satellite imagery used for pseudo-labels, since automatic PSA generation is prone to errors.
• Releasing the code, dataset and preprocessed data so that UrbanDiff and other BEV-based or multi-scale methods can be compared under identical conditions.
• Extending standardized benchmarking to more outdoor datasets and conditions beyond the two KITTI-derived subsets, and further exploring the fusion of Cartesian and cylindrical spatial representations.

Target Audience

Researchers and graduate students in 3D generative modeling, autonomous driving perception, and remote-sensing scene synthesis; practitioners who need synthetic 3D urban training data; and engineers interested in sketch- or satellite-conditioned generative interfaces. The paper is best suited to readers already comfortable with diffusion models, latent autoencoders and state space sequence models.

Authors’ abstract

Outdoor 3D semantic scene generation produces realistic and semantically rich environments for applications such as urban simulation and autonomous driving. However, advances in this direction are constrained by the absence of publicly available, well-annotated datasets. We introduce SketchSem3D, the first large-scale benchmark for generating 3D outdoor semantic scenes from abstract freehand sketches and pseudo-labeled annotations of satellite images. SketchSem3D includes two subsets, Sketch-based SemanticKITTI and Sketch-based KITTI-360 (containing LiDAR voxels along with their corresponding sketches and annotated satellite images), to enable standardized, rigorous, and diverse evaluations. We also propose Cylinder Mamba Diffusion (CymbaDiff) that significantly enhances spatial coherence in outdoor 3D scene generation. CymbaDiff imposes structured spatial ordering, explicitly captures cylindrical continuity and vertical hierarchy, and preserves both physical neighborhood relationships and global context within the generated scenes. Extensive experiments on SketchSem3D demonstrate that CymbaDiff achieves superior semantic consistency, spatial realism, and cross-dataset generalization. The code and dataset will be available at https://github.com/Lillian-research-hub/CymbaDiff