Octrees as an Explicit 3D Language

AI summary

Overview

Research area: 3D-native multimodal large language models that unify 3D shape generation and 3D shape understanding in a single LLM backbone.

Technical level: Advanced. The paper assumes familiarity with transformer LLMs, rotary position embeddings, LoRA versus full fine-tuning, octree/voxel representations, and flow-based mesh decoders.

Scope: The paper introduces OctLLM, a Qwen2.5-VL-7B-Instruct backbone in which 3D geometry enters and leaves the model as sequences of octree occupancy tokens routed through trainable 3D branches, while the pretrained text and image pathway stays frozen. (arXiv:2610.02388v1 [cs.CV], 01 Oct 2026; project page: https://plurato.github.io/OctLLM-page/)

What This Paper Is About

Existing 3D large language models compromise on two fronts. First, they compress shapes into VQVAE codebook indices or serialize meshes as OBJ coordinate text, which removes spatial structure from what the model observes. Second, they acquire the 3D modality by fine-tuning the backbone, which overwrites its general language ability. OctLLM's goal is to satisfy both requirements at once: let geometry enter as an explicit, hierarchical 3D structure the LLM can directly read and write, and add 3D capacity in parameters separate from the pretrained ones so language ability survives.

Key Contributions

1. S-Octrees as an explicit 3D language. OctLLM reads and writes a Sparse Octree (S-Octree) — a coordinate- and depth-anchored occupancy-token sequence — using position-aware mask queries for generation and direct S-Octree input for understanding. To control sequence growth with depth, the S-Octree randomly empties a fraction of second-to-last-level nodes and omits their finest-level descendants while preserving overall shape.
2. A token-routed dual-stream transformer. Mesh and position-query tokens are routed through full-rank trainable 3D branches (attention projections and FFNs) in a subset of blocks, with their own mesh embeddings and mesh head, while text and image tokens stay on the frozen vision-language pathway. Both streams interact through shared causal self-attention.
3. A complete generation pipeline. An S-Octree predicted by the LLM is decoded into a sparse occupancy grid, completed by a 3D U-Net, and passed as structural condition to TRELLIS, used off the shelf as the flow-based mesh decoder.
4. State-of-the-art results among unified multimodal LLMs on 3D generation and 3D understanding, while matching the backbone exactly on general language benchmarks — reported as lowering image-to-3D FID by 17.4% and raising render-grounded captioning by 28.7 points over ShapeLLM-Omni.

Main Findings

• Image-to-3D generation. On 1K held-out Toys4K assets, OctLLM reaches FID (Inception) 31.58, KID (Inception) 0.56, FID (DINOv2) 135.58, KID (DINOv2) 7.94, and CLIP 87.31, versus ShapeLLM-Omni at 38.21 / 1.02 / 145.14 / 9.98 / 86.75. The paper reports this as reducing Inception FID by 17.4% and KID by 45% relative to ShapeLLM-Omni. Task-specific TRELLIS scores 18.15 / 0.13 / 48.47 / 2.06 / 92.86, and SAR3D scores 67.42 / 3.16 / 206.79 / 20.56 / 84.46.
• Text-to-3D generation. OctLLM scores FID (Inception) 45.61, KID (Inception) 0.98, FID (DINOv2) 175.58, KID (DINOv2) 9.01, and CLIP 22.30. It leads the multimodal-LLM family on Inception FID and both KID scores, surpasses the task-specific octree generator OctGPT (46.40 / 1.13 / 228.81 / 15.92 / 19.70), and performs comparably to ShapeLLM-Omni (52.36 / 1.38 / 169.28 / 10.34 / 22.46) on DINOv2 FID and CLIP. TRELLIS attains the best scores overall (30.90 / 0.34 / 113.16 / 3.54 / 24.71), with OctLLM ranking second on most metrics. The paper notes ShapeLLM-Omni and AR3D-R1 also use TRELLIS as their flow-based mesh decoder, so OctLLM's gain over them comes from its 3D representation and pipeline.
• 3D understanding on PointLLM-200. OctLLM scores Sentence-BERT 38.80, SimCSE 38.48, GPT-ref 26.61, and GPT-img 46.88. It leads every metric among multimodal LLMs; its GPT-img score is best across all methods, exceeding ShapeLLM-Omni's 18.14 by more than 28 points. It trails only PointLLM-7B (48.05 / 48.63 / 46.06 / 45.63) on the embedding metrics and GPT-ref. The paper attributes the remaining gap to input content: PointLLM reads colored point clouds and can report material and color, whereas the S-Octree encodes geometry alone, so OctLLM describes shape and part structure and says nothing about appearance.
• Language ability preservation. OctLLM records MMLU 64.65, HellaSwag 67.44, GSM8K 83.01, and IFEval 69.87, against the backbone Qwen2.5-VL-7B at 64.65 / 67.44 / 83.54 / 75.41 — reproducing the backbone exactly on the two multiple-choice benchmarks and staying within one point on GSM8K. ShapeLLM-Omni scores 30.17 / 26.74 / 27.45 / 58.04 and LLaMA-Mesh 51.84 / 34.03 / 52.69 / 66.54, which the paper describes as baselines losing between a third and a half of the backbone's accuracy. The residual IFEval drop is attributed to the mesh head occasionally opening a 3D sequence mid-answer and is reported as eliminated by masking the mesh vocabulary at inference time.
• Parameter efficiency. Routing 3D branches through 12 of the backbone's 28 decoder blocks, at 0-indexed layers {0, 4, 8, 12, 14, 16, 18, 20, 22, 24, 26, 27}, adds about 2.80B trainable parameters — roughly a quarter of the total and about a third of what full fine-tuning updates.
• Tokenization efficiency. Packing every 8 consecutive binary occupancy bits into one byte-level mesh token with value in [0, 255] reduces sequence length by a factor of 8 compared with a raw binary sequence. Sparsification is reported to reduce training memory and time without sacrificing generation quality (Appendix D).
• Qualitative behavior. In image-to-3D comparisons against ShapeLLM-Omni, TRELLIS, and SAR3D on four held-out assets, SAR3D recovers only a coarse silhouette with noisy surfaces, ShapeLLM-Omni produces cleaner surfaces but does not preserve part-level structure, and OctLLM reproduces both global layout and individual parts, visually closest to TRELLIS. In text-to-3D comparisons against ShapeLLM-Omni, AR3D-R1, and LLaMA-Mesh, LLaMA-Mesh returned no valid mesh for a stringed-instrument prompt; OctLLM produced two stacked wings joined by struts, four legs with a long tail, and a rectangular basin on a cylindrical column.

Methodology in Plain English

Turning a mesh into language tokens. The input mesh is encoded as an octree whose finest level matches the target voxel resolution, and the

Authors’ abstract

Existing 3D large language models (LLMs) compromise on two fronts: they compress shapes into latent codebook indices or coordinate text, which removes spatial structure from what the model observes, and they acquire the 3D modality by fine-tuning the backbone, which overwrites its general language ability. We present OctLLM, which addresses both limitations. Geometry enters as an explicit 3D sequence of octree occupancy tokens. However, full octree sequences grow rapidly with depth; OctLLM therefore randomly empties penultimate-level nodes and omits descendants while preserving shape, yielding a shorter coordinate- and depth-anchored Sparse Octree (S-Octree) for position-aware mask-modeling generation and 3D understanding. On the other front, existing methods introduce a new modality with full fine-tuning or LoRA, but full fine-tuning is costly, LoRA limits 3D capacity, and both modify the language pathway. OctLLM instead adds 3D capacity in parameters separate from the pretrained ones: mesh tokens are routed through independent trainable branches in a subset of blocks while text and image tokens retain the frozen vision-language pathway, and the two streams interact through shared self-attention. It trains far fewer parameters than full fine-tuning, yet sets a new state of the art among unified multimodal LLMs, lowering image-to-3D FID by $17.4\%$ and raising render-grounded captioning by $28.7$ points over ShapeLLM-Omni, while matching the backbone on general language benchmarks.