AssetFormer: Modular 3D Assets Generation with Autoregressive Transformer

AI summary

Overview

Research area: Computer vision / 3D content generation, specifically text-conditioned generation of modular 3D assets built from discrete building primitives.

Technical level: Intermediate. Familiarity with autoregressive Transformers, tokenization, and 3D representations helps, but the core design ideas (sequencing primitives, decoding strategies) are explained in accessible terms.

Scope: The paper introduces AssetFormer, an autoregressive Transformer that generates modular 3D assets (buildings) from text, along with a dataset of modular assets and an analysis of token ordering and decoding choices.

What This Paper Is About

Most 3D generation methods produce dense meshes, voxels, point clouds, or neural fields. These are often too heavy or too imprecise for real production pipelines such as games and user-generated content (UGC) platforms, where artists build assets from reusable primitives under constrained design rules. The paper's goal is to generate such modular assets automatically from text, framing an asset as an ordered sequence of primitives (each with a class, a rotation, and a 3D position) so that a language-model-style autoregressive Transformer can generate it.

Key Contributions

1. An autoregressive generation framework for modular 3D assets. AssetFormer models an asset as a sequence of primitive token tuples and predicts them with a decoder-only Transformer using standard next-token cross-entropy loss, showing promising results relative to other 3D modalities.

2. A large-scale modular 3D dataset. The authors collected and cleaned real user-created modular assets from an online UGC platform and combined them with procedurally generated data, producing a high-quality dataset of 16,000 real samples and 4,000 synthesized samples (average token length larger than 4,000). They describe it as the only real-world modular 3D dataset of high quality.

3. Analysis of tokenization order and decoding strategies. The paper studies how primitive re-ordering (raw order vs. breadth-first search vs. depth-first search, plus a RAR-style randomized baseline) and sampling/decoding choices (greedy, beam, top-k, SlowFast speculative decoding) affect generated asset quality and diversity.

4. A practical, deployable representation. Because assets are generated as primitives rather than dense meshes, the outputs need no post-processing such as vertex merging and can be integrated directly into game engines and rendering pipelines, benefiting from established primitive-to-texture mapping.

Main Findings

• Top-k sampling beats greedy and beam search. On 500 generated assets rendered from a fixed viewpoint and evaluated with clean-FID against the full training set, top-k sampling reached FID 55.186 and CLIP 0.320, compared with greedy search at 63.351 / 0.319 and beam search at 63.333 / 0.321. True data scores FID undefined ("/") and CLIP 0.322.

• The PCG baseline scores far worse on FID. The procedural generation baseline used for data synthesis achieved FID 108.476 and CLIP 0.319 — it produces compact but structurally simple assets and offers no text control.

• Depth-first search ordering works best. Among ordering techniques, DFS achieved FID 55.186 / CLIP 0.320, versus BFS 61.620 / 0.319, raw order 65.215 / 0.318, and the RAR-style annealing randomization at 83.561 / 0.313. The authors hypothesize RAR's token disturbance, helpful for images, hinders learning of local 3D structure.

• Combining synthesized and collected data is better than either alone. Synthesized data only: FID 113.560 / CLIP 0.320. Collected data only: FID 63.381 / CLIP 0.321. Combined: FID 55.186 / CLIP 0.320. The structured synthetic data acts as scaffolding, while user-created data adds diversity.

• SlowFast decoding trades a small amount of quality for speed. AssetFormer-S (87M, 12 layers) reached FID 60.420 at 151.31 tokens/s; AssetFormer-B (312M, 24 layers) reached FID 55.186 at 80.62 tokens/s; SlowFast decoding reached FID 55.831 at 119.02 tokens/s.

• Native 3D representations lose modular information. When the authors exported their modular assets to geometry and applied watertight preprocessing for native generative models, primitives were merged into a single unstructured mesh and fine details were distorted. Fine-tuning Hunyuan3D 2.1 on a subset of only 10 modular samples, even under overfitting conditions, still failed to reconstruct training-sample details.

• Language-model fine-tuning on raw JSON performed poorly. LongLoRA fine-tuning of Llama-2 on building JSON data (which can require more than 3K tokens for up to 1,000 primitives) yielded results far worse than AssetFormer, with a qualitative winrate below 5%.

• User study results are mixed but favor AssetFormer on richness. Six participants aged 22–28 rated batches on a 1–5 scale. AssetFormer scored Compactness 3.42, Diversity 3.50, Aesthetic 3.50, Complexity 3.92. Ground truth scored 3.83, 4.00, 3.67, 4.42. PCG scored the highest compactness (4.47) but the lowest diversity (2.42) and complexity (2.08).

• CLIP scores were unreliable for prompt alignment. Scores between images and generation prompts fluctuated near 0.29 across settings and poorly aligned with human validation, so the authors used CLIP between rendered images and a fixed prompt, "A high-quality 3D model of a building".

Methodology in Plain English

The researchers start from a representation in which every 3D asset is a collection of primitives. Each primitive has five discrete attributes: a class (one of 25), a rotation (one of 4), and a 3D position along three axes (59, 44, and 81 discrete values respectively). These discrete spaces let the authors build a single joint vocabulary of size 214 (including an <EOS> token) instead of relying on a learned codebook or graph encoder.

An asset becomes a flat token sequence of length 5 times the number of primitives, with assets padded by <EOS>. Crucially, the order in which primitives are written matters. The authors re-order primitives by traversing the connectivity graph of the asset — either breadth-first or depth-first — starting from a lower corner, so that spatially adjacent modules appear close together in the sequence. Empirically DFS wins.

The model itself is a decoder-only Transformer with a Llama-style backbone (no pre-trained weights), 24 layers for the 312M-parameter AssetFormer-B and 12 layers for the 87M AssetFormer-S draft model. Text is encoded by FLAN-T5 XL and projected through an MLP, then pre-filled into the token sequence as a prefix. Training uses next-token cross-entropy, with a condition dropout ratio of 0.1 to support classifier-free guidance (CFG scale 2.0 at inference). Because different attribute types have separate vocabularies, the model filters and re-normalizes logits at each step so it only samples valid tokens for the current attribute slot.

For speed, the authors adapt speculative decoding into "SlowFast decoding": the small draft model quickly proposes tokens for easy, pattern-like parts, and the larger target model handles harder, context-dependent primitives, with the same filtering applied during rejection sampling. For text data, GPT-4o generates phrase bundles such as "(apartment, multi-story, flat roof, few windows)" from rendered images of the assets.

Why This Matters

Impact on research: The paper argues that modular, primitive-based representations are a practical alternative to dense meshes and native 3D representations for domains with constrained design spaces. It also shows that design decisions from language modeling — token ordering, vocabulary construction, filtered decoding, speculative decoding — transfer meaningfully to 3D sequential generation, and it opens a data-poor area (modular assets) by releasing both a dataset and code at https://github.com/Advocate99/AssetFormer.

Real-world applications:

• Game development, where artists can generate reusable building blocks that drop directly into engines without mesh clean-up.
• UGC platforms and online games, where small primitive-based assets ease storage and transmission pressures compared with large mesh files.
• Rapid prototyping and level design, letting designers iterate on structures from text descriptions rather than hand-placing every module.
• Accessible creation tools for non-professional users, lowering the barrier to producing consistent, engine-ready content.

Industry relevance: Modular and CSG-style workflows are already standard in CAD and game production. By generating within that same constrained design space, AssetFormer targets pipelines where asset consistency, file size, and integration cost matter more than raw geometric freedom — and where primitive-to-texture mapping is already mature, giving cleaner texturing than current dense-mesh texturing approaches.

Future Directions

1. Image-based conditioning. The authors state explicitly that AssetFormer currently accepts only text input, and that image conditioning remains uncertain and unexplored.

2. Beyond fixed vocabularies. The model relies on fixed discrete vocabularies, so adapting to varying or larger design spaces would require additional design considerations.

3. Stronger captioning for the domain gap. The paper notes a significant domain gap between rendered modular assets and natural images, which makes it hard for multimodal language models to produce discriminative global type captions.

4. Better native 3D data curation. The authors argue that native 3D generation needs more advanced data curation and improved preprocessing — particularly around the watertight step that destroys modular detail — to reach fine-grained control.

Target Audience

Researchers and practitioners in 3D content generation, computer graphics, and game or UGC pipeline engineering who are interested in structured, deployable asset synthesis. It is also relevant to those studying how autoregressive and language-model techniques transfer to non-text modalities, and to anyone needing a real-world modular 3D dataset for benchmarking.

Authors’ abstract

The digital industry demands high-quality, diverse modular 3D assets, especially for user-generated content~(UGC). In this work, we introduce AssetFormer, an autoregressive Transformer-based model designed to generate modular 3D assets from textual descriptions. Our pilot study leverages real-world modular assets collected from online platforms. AssetFormer tackles the challenge of creating assets composed of primitives that adhere to constrained design parameters for various applications. By innovatively adapting module sequencing and decoding techniques inspired by language models, our approach enhances asset generation quality through autoregressive modeling. Initial results indicate the effectiveness of AssetFormer in streamlining asset creation for professional development and UGC scenarios. This work presents a flexible framework extendable to various types of modular 3D assets, contributing to the broader field of 3D content generation. The code is available at https://github.com/Advocate99/AssetFormer.