CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models

AI summary

Overview

Research area: Computer vision and multimodal generative modeling, specifically 3D content generation and 3D understanding powered by large language models.

Technical level: Advanced. The paper assumes familiarity with autoregressive transformers, mixture-of-transformers architectures, VAE latent spaces, and rectified flow / diffusion sampling.

Scope: This paper introduces CG-MLLM, a single multimodal large language model that performs both 3D captioning and end-to-end high-resolution 3D object generation by combining a Qwen3-VL backbone with the Hunyuan3D-2.1 VAE latent space under a Mixture-of-Transformer design.

What This Paper Is About

Existing attempts to make multimodal large language models produce 3D content either emit meshes as text-like tokens, which caps mesh complexity and resolution, or build coarse low-resolution voxel or lego-style proxies that still need an external 3D diffusion model for fine detail. The authors' goal is a single language-image-3D model that natively generates detailed, high-fidelity 3D geometry inside the LLM framework itself, while also improving the model's ability to reason about 3D structure from images.

Key Contributions

1. A new end-to-end 3D generative method natively integrated with a large vision-language model, removing the need for a separate external 3D generator.
2. The CG-MLLM architecture, which is reported to outperform Qwen3-VL on 3D understanding tasks and to achieve state-of-the-art performance among LLM-based 3D generation methods.
3. A Mixture-of-Transformer design that binds transformers to generation modes rather than tasks: a Token-level Autoregressive (TokenAR) transformer for sequential token modeling and a Block-level Autoregressive (BlockAR) transformer for parallel block prediction, initialized from pretrained Qwen3-VL weights.
4. A bridge between multimodal LLMs and high-precision 3D synthesis at lower computational cost, positioned as an efficient and scalable architecture for future generative 3D modeling research.

Main Findings

• Generation quality among MLLM-based methods: On the paper's table of generation metrics, CG-MLLM records p-FID 12.55, p-KID 0.27, CLIP-IQA+ 0.45, MUSIQ 71.65, CLIP 84.47, and a user study score of 3.32. This beats the compared MLLM-based models SAR3D (p-FID 30.07, p-KID 1.00, CLIP-IQA+ 0.42, MUSIQ 66.01, CLIP 82.86, user study 2.93) and ShapeLLM-Omni (p-FID 13.11, p-KID 0.29, CLIP-IQA+ 0.37, MUSIQ 55.71, CLIP 84.18, user study 2.30) across all metrics.
• Still short of non-MLLM methods: The authors state that CG-MLLM does not comprehensively surpass state-of-the-art non-MLLM-based methods. For reference, TRELLIS reaches p-FID 7.36 and p-KID 0.12, and SAM3D reaches the highest user study score at 3.45.
• 3D generation transfers back to perception: Training for 3D generation improved image-based 3D understanding. The captioning results for Ours-2B MOT are BLEU-1 13.51, ROUGE-L 19.13, and METEOR 14.28, better than InstructBLIP-13B (4.65 / 8.85 / 13.23), LLaVA-13B (4.02 / 8.15 / 12.58), and Qwen3-VL-2B (3.13 / 7.21 / 11.92). It still trails the 3D-latent-input models 3D-LLM (16.91 / 19.48 / 19.73), PointLLM-13B (17.09 / 20.99 / 16.45), and ShapeLLM-Omni-7B (18.51 / 21.37 / 19.89), which the authors attribute to those models' larger 3D captioning datasets and greater model scale.
• Parallel block modeling is faster: At a spatial latent resolution of 4096 tokens, the block-level approach achieves a threefold speedup compared with token-level processing.
• Ablation trends favor larger components: In the ablation table, removing the Hunyuan3D-2.1 VAE and MOT with a Qwen2.5-0.5B backbone at 512 tokens gives p-FID 53.66 and p-KID 1.76; adding the stronger VAE gives 44.91 / 1.42; adding MOT gives 30.60 / 0.77; switching to a Qwen3VL-2B backbone gives 15.61 / 0.43; and scaling tokens to 4096 with Qwen3VL-2B reaches 12.55 / 0.27.
• Training and inference cost scale predictably: Under a fixed budget of 20,480 tokens per GPU on 8 GPUs, memory drops from 75 GB at 512 3D tokens to 40 GB at 5120 tokens, with total samples per batch falling from 300 to 90. Inference time rises from 12 s at 512 tokens (9 GB) to 45 s at 5120 tokens (12 GB).
• AdaLN hurt training in this setting: Adding Adaptive Layer Normalization to the multimodal branches produced substantially higher training loss than the baseline, which the authors hypothesize compromises the stability of the shared causal-parallel self-attention mechanism.
• Importance points degraded VAE reconstruction: Following the Hunyuan3D-2.1 point sampling strategy (uniform plus importance points) caused distorted reconstructions with holes in some cases, so the authors used only uniformly sampled points.
• Observed failure modes: Ambiguous hints commonly produce suboptimal results, and in a rare case a hint image of a sheep produced a rabbit, which the authors attribute to hallucination because the vision encoder operates at the semantic level.

Methodology in Plain English

The model is built on a pretrained vision-language backbone, Qwen3-VL. Text goes through Qwen's byte-pair-encoding tokenizer with a vocabulary of 151,669, images go through a SigLIP-2 encoder with 2D-RoPE and a two-layer MLP that compresses each 2 by 2 patch of visual features into one visual token, and 3D shapes are turned into points on their surfaces and encoded by a frozen spatial VAE adapted from Hunyuan3D-2.1 with a downsampling factor of 20 and a latent dimension of 64.

The architecture splits generation into two cooperating transformers inside one decoder-only model. The TokenAR transformer predicts tokens one at a time for language and other sequential content, while the BlockAR transformer predicts whole blocks of 3D tokens in parallel. Both are initialized from Qwen3-VL weights and share attention, with a hybrid masking scheme that visualizes which tokens may attend to which others depending on modality and task. Because each transformer is matched to a generation mode rather than to a task, encoders can be plugged into whichever transformer matches their original pre-training scheme, which keeps the perturbation to the pretrained model small.

Training happens in two stages. In the alignment stage, 90 percent of conditional inputs are discarded and the model trains at a 3D resolution of 512 tokens to build unconditional generation and initial understanding. In the progressive resolution stage, the resolution rises from 512 to 4096 tokens while the discard probability falls from 90 percent to 10 percent. Optimization uses AdamW with the learning rate annealed from 1e-4 to 5e-5, a logit-normal timestep sampler at scale 1.0, and 16 NVIDIA H20 GPUs with maximum sequence length growing from 36,864 to 51,200. At inference, the classifier-free guidance scale is 7.5 with 50 sampling steps. Generated 3D tokens are decoded by the Hunyuan3D-2.1 VAE, followed by its material generator.

Data comes from LLaVA-OneVision, Trellis-500K, and Objaverse++, with the toys4k subset of Trellis-500K held out for testing. From Objaverse++, only samples with the highest aesthetic rating (score=3) were selected, using rendered images from Objaverse-MIX. For Trellis-500K the Hunyuan3D2.1 pipeline was used for watertight processing and surface sampling; the authors pre-sampled 124,928 uniform points and importance points but ended up using only the uniform points.

Why This Matters

The work pushes 3D content creation into the same paradigm that has driven rapid progress in language and 2D vision, so 3D generation can potentially benefit from the scaling behavior of large language models instead of relying on separate, task-specific pipelines. It also reports a transfer effect in the other direction: learning to generate 3D geometry improved the model's ability to interpret 3D structure from images alone.

Real-world applications implied by the paper's framing:

• Text- or image-driven 3D asset creation, where a single model produces detailed geometry and materials without a separate generator.
• 3D content understanding and captioning for catalogs or datasets of objects, using rendered images as input.
• Reducing computational cost in 3D generation pipelines, since the design reports lower training and inference overhead than running decoupled language and diffusion stages.
• Scaling toward a unified model that handles multimodal understanding and generation, which the authors name as future work.

Industry relevance centers on the cost of 3D pipelines: the paper reports a threefold speedup at 4096 tokens from parallel block modeling, training costs that fit on 16 NVIDIA H20 GPUs, and inference running between 12 and 45 seconds across 512 to 5120 3D token lengths. The authors are affiliated with Zhejiang University and LIGHTSPEED.

Future Directions

• Closing the remaining quality gap to non-MLLM-based 3D generation methods, through stronger multi-encoder image conditioning, scaling up model parameters, and increasing the 3D token budget.
• Addressing the limited 3D captioning supervision, since most captions in the dataset used are usually fewer than 20 words and were generated by an MLLM.
• Developing more lightweight and efficient 3D VAEs, which the authors expect would substantially improve their model's capabilities given that current high-resolution 3D methods often need up to ten times more tokens than the fewer than 4k tokens used here.
• Improving robustness to ambiguous or confusing input hints, and reducing semantic hallucination from the vision encoder.
• Exploring scalability toward a fully unified model for multimodal generation and understanding.

Target Audience

Researchers and engineers working on multimodal large language models, 3D generative modeling, and vision-language architectures. The paper is most useful to readers who already understand autoregressive transformers and diffusion or rectified-flow sampling and want to see how those pieces can be fused into a single backbone for 3D output. Practitioners interested in reducing the cost of 3D asset pipelines will also find the training and inference cost tables and the speedup claims relevant.

Authors’ abstract

Large Language Models(LLMs) have revolutionized text generation and multimodal perception,but their capabilities in 3D content generation remain underexplored. Existing methods compromise by producing either low-resolution meshes or coarse structural proxies, failing to capture finegrained geometry natively. In this paper, we propose CG-MLLM, a novel Multi-modal Large Language Model (MLLM) capable of 3D captioning and high-resolution 3D generation in a single framework. Leveraging the Mixture-ofTransformer architecture, CG-MLLM decouples disparate modeling needs, where the Token-level Autoregressive (TokenAR) Transformer handles token-level content, and the Block-level Autoregressive (BlockAR) Transformer handles blocklevel content. By integrating a pre-trained visionlanguage backbone with a specialized 3D VAE latent space, CG-MLLM facilitates long-context interactions between standard tokens and spatial blocks within a single integrated architecture. Experimental results show that CG-MLLM significantly outperforms existing MLLMs in generating high-fidelity 3D objects, effectively bringing high-resolution 3D content creation into the mainstream LLM paradigm. Beyond generation, we further observe that learning to produce 3D content transfers back to perception, strengthening the model's image-based 3D understanding.