ZipTok3D: High-Fidelity 3D Tokenization with Compact Token Prefixes

AI summary

Overview

Research area: 3D computer vision and 3D representation learning, specifically learned 3D tokenizers (autoencoders that map geometry into latent token sequences) and their use in downstream 3D generative modeling.

Technical level: Advanced. The paper assumes familiarity with neural fields, triplane representations, cross-attention tokenizers, nested dropout, and shared-weight Transformer decoding.

Scope: The paper introduces ZipTok3D, a 3D tokenizer that combines nested prefix training with parameter-shared iterative refinement to reconstruct detailed 3D geometry from extremely short latent token sequences.

Affiliations: Zhejiang University, Monash University, University of Adelaide. Posted as arXiv:2609.01740v1 [cs.CV] on 01 Sep 2026.

What This Paper Is About

Existing 3D tokenizers represent an object either as spatially organized tokens or as a fixed-size set of global tokens. Both degrade sharply when the token budget is pushed to only a handful of tokens, because each token must absorb a much larger share of the object's geometry. ZipTok3D targets this bottleneck: it reorganizes the latent sequence so that the leading tokens carry the object-wide structure needed for faithful reconstruction, and then unfolds that compact information through repeated, weight-shared decoding rather than a single feed-forward pass.

Key Contributions

1. ZipTok3D, a 3D tokenizer for extremely short token sequences. The encoder emits a maximum of 128 latent tokens of width 512, and the model is designed so that a single token can already carry object-wide geometry.

2. Nested prefix tokenization. During training, nested dropout retains a prefix of length K sampled uniformly from the exponentially spaced budgets {1, 2, 4, 8, 16, 32, 64, 128} and masks the suffix from both triplane selection and refinement, forcing each retained prefix to reconstruct the complete object.

3. Parameter-shared iterative refinement. A single six-layer Transformer block is applied repeatedly (five passes in the main results) with the retained prefix held fixed, increasing effective decoding depth without adding step-specific parameters, and without a separate generative sampling stage.

4. Intermediate reconstruction supervision and intra-loop self-distillation. The final output plus one randomly sampled intermediate step are supervised, using the detached final prediction as a soft target for the intermediate prediction.

Main Findings

• One token on ShapeNet matches 32-token COD-VAE on rounded CD and F1. ZipTok3D at K = 1 reaches IoU 96.8, CD 0.012, F1 97.8, versus COD-VAE-32 at IoU 97.1, CD 0.012, F1 97.8 — within 0.3 IoU points — using a 32× shorter latent sequence.

• Four tokens on TRELLIS improves on COD-VAE-32 across reported metrics. ZipTok3D at K = 4 reaches IoU 75.31, CD 0.0166, F1 95.92, versus COD-VAE-32 at IoU 75.25, CD 0.0172, F1 95.67, with an 8× shorter latent sequence.

• The gain does not come from merely shrinking a fixed latent set. COD-VAE drops to IoU 77.7, CD 0.032, F1 80.9 at two ShapeNet tokens, showing that fixed-budget objectives break down in the few-token regime.

• Paired bootstrap on the TRELLIS K = 4 comparison. Using 20,000 resamples and seed 123456, query IoU changes by +0.06 points (95% CI [-0.08, +0.20]), mesh CD decreases by 0.00058 (95% CI [0.00039, 0.00080]), and mesh F1 increases by 0.25 points (95% CI [0.16, 0.34]). The IoU interval includes zero; the CD and F1 intervals exclude zero and favor ZipTok3D.

• Intermediate token budgets degrade gracefully. On TRELLIS, ZipTok3D at K = 1 reaches IoU 75.18, CD 0.0168, F1 95.81; K = 2 reaches IoU 75.22, CD 0.0167, F1 95.86.

• Ablations isolate the two mechanisms (ShapeNet, K = 2). "Prefix only" keeps COD-VAE's 12-layer single-pass decoder and only adds nested dropout: IoU rises from 77.7 to 92.3, CD falls from 0.032 to 0.015, F1 rises from 80.9 to 95.8 at identical decoder cost (39.3M parameters, 1.47 ms latency). Removing iterative refinement (6×1) gives IoU 91.9, CD 0.017, F1 94.2 at 23.5M parameters and 0.98 ms. Five passes without intermediate supervision give IoU 96.6, CD 0.013, F1 97.6 at 23.5M parameters and 2.65 ms. The full model gives IoU 96.9, CD 0.012, F1 97.8 at the same 23.5M parameters and 2.65 ms.

• Prefix length and refinement depth are complementary. Longer prefixes help most under a single decoding pass; additional refinement helps most at K = 1 and K = 4. ShapeNet and TRELLIS both approach saturation by L = 5. Sweeps cover K ∈ {1, 2, 4, 8, 16, 32, 64, 128} and L ∈ {1, 2, 3, 4, 5, 6}.

• Per-object improvement rates. At K = 1, moving from L = 1 to L = 3 improves IoU for 100.0% of ShapeNet and 98.6% of TRELLIS objects, CD for 99.3% and 99.8%, and F1 for 97.6% and 99.8%. Moving from L = 3 to L = 5 improves IoU for 88.2% and 91.2%, CD for 81.1% and 91.8%, and F1 for 58.1% and 84.7%. At K = 4, the L = 1 to L = 3 rates are 100.0% and 98.5% (IoU), 98.7% and 99.7% (CD), 94.8% and 99.4% (F1); the L = 3 to L = 5 rates are 89.5% and 90.5% (IoU), 81.5% and 90.7% (CD), 55.7% and 83.4% (F1).

• Compact prefixes support class-conditioned generation. A causal second-stage VAE maps each K × 512 tokenizer prefix to a K × 32 latent sequence without changing length. ZipTok3D-2 uses a 2 × 32 stage-2 sequence versus COD-VAE-32's 32 × 32 (16× shorter). Distribution metrics: MMD 5.142 vs 5.020, COV 84.91 vs 84.96, 1-NNA 54.21 vs 53.11. Efficiency: sampling throughput 50.62 vs 46.02 samples/s, full throughput 36.64 vs 36.14, peak memory 2.20 GiB vs 2.26 GiB. The VecSet baselines are substantially slower on the full pipeline (VecSet-512: 2.14 samples/s full, 33.24 GiB; VecSet-32: 6.59 samples/s full, 25.68 GiB).

• Datasets and evaluation. ShapeNet-v2 evaluation split has 1,283 objects from 55 categories. TRELLIS-500K evaluation split has 2,613 assets, built with watertight preprocessing and seed 42. Reconstruction uses query IoU over 500,000 volume queries plus mesh CD and F1 from 128³ marching-cubes reconstructions. Generation uses 2,000 generated shapes per category for airplane, car, chair, table, and rifle, with MMD-CD, COV-CD, and 1-NNA-CD. Diffusion models use 18 sampling steps; 3DILG uses its native 512-step autoregressive sampler.

• Unavailable results. TRELLIS reconstruction numbers for 3DILG and VecSet-32/64 are marked as dashes, which the paper defines as unavailable because the corresponding public checkpoint was not released.

Methodology in Plain English

The approach splits a hard problem into two questions that a conventional autoencoder does not separate: what geometry survives after the latent sequence is shortened, and how much computation is spent interpreting that compact code.

On the representation side, the encoder takes 2,048 surface points and produces up to 128 latent vectors of width 512. During training, a random prefix length is drawn from a fixed set of budgets, and everything after that prefix is masked away from the decoder. Because the masked model must still reproduce the full object, the earliest tokens are pushed to carry the information needed for a complete reconstruction, while later tokens hold residual detail. Every shorter prefix is contained inside every longer one, so the same checkpoint works at any trained budget and the sequence can be shortened simply by truncation.

On the decoding side, a small set of learnable triplane tokens first interacts with the retained prefix in a selection block, which divides them into a selected state and a set of redundant tokens. A single six-layer Transformer block then updates that selected state repeatedly, conditioned on the same fixed prefix every time. Each update can be restored into a full triplane representation and queried by a shared occupancy MLP, producing a valid occupancy prediction at each step. Because the parameters never change between steps, adding refinement passes adds depth without adding parameters.

Training. The final occupancy prediction is supervised with a weighted binary cross-entropy over uniformly sampled volume queries and near-surface queries (λ_near = 0.1). One intermediate step is sampled uniformly from {1, ..., L−1} and receives both direct reconstruction supervision and self-distillation toward the stop-gradient final prediction, with the balance between the two annealed over training (α_t = max(0, 1 − t/T_α)). Two auxiliary losses inherited from COD-VAE stabilize training (λ_init = 0.5, λ_unc = 0.001), and β = 0.5 weights the intermediate-supervision term.

Configuration. Training is FP16 with AdamW, learning rate 10⁻⁴, weight decay 0.01, effective batch size 672 on four A800 GPUs, for 1,000 epochs on ShapeNet and 300 on TRELLIS, with seed 123456. Checkpoints are selected by highest validation query IoU. During training, independent axis scales are sampled uniformly from [0.75, 1.25], shapes are renormalized to the unit cube, and Gaussian noise with standard deviation 0.005 is added to surface points before clipping to [−1, 1]; held-out splits are not augmented.

Why This Matters

Impact on research. The paper reframes short-token reconstruction as a joint problem of information ordering and decoder computation rather than compression alone. It provides a deterministic alternative to approaches that use generative completion for short prefixes, since ZipTok3D reconstructs the encoded instance directly from each retained prefix. It also shows that a fixed-budget objective and a nested-prefix objective produce very different behavior at the same token count, which has implications for how 3D tokenizers are trained and evaluated.

Real-world applications:

• Text- or class-conditioned 3D asset generation, where shorter latent sequences directly reduce second-stage generative modeling cost.
• Interactive 3D content creation pipelines, where fast reconstruction from compact codes lowers per-asset compute.
• Large-scale 3D asset libraries in gaming and simulation, where storing objects as short latent prefixes reduces memory footprint relative to spatially organized representations.
• Robotics and embodied simulation, where compact, truncated latent codes can serve as shape descriptors under tight latency budgets.

Industry relevance. Generation throughput is a practical constraint in 3D content pipelines. The paper reports ZipTok3D-2 at 50.62 samples/s sampling throughput and 36.64 samples/s full throughput with 2.20 GiB peak memory, compared with 0.24/0.22 samples/s and 26.18 GiB for 3DILG and 4.27/2.14 samples/s and 33.24 GiB for VecSet-512 under the same protocol, measured on one NVIDIA H20 GPU with batch size 16 and FP32 inference.

Future Directions

1. Generative models that better exploit short latent sequences. The authors state that future work will investigate generative models designed around compact prefixes rather than adapting existing ones.

2. Adaptive allocation of prefix length and refinement depth per input. The paper leaves open the question of choosing K and L per object rather than at a fixed operating point.

3. Extending the analysis beyond the evaluated budgets. The sweeps cover K ∈ {1, 2, 4, 8, 16, 32, 64, 128} and L ∈ {1, 2, 3, 4, 5, 6}; behavior outside this range, including with other token widths, is not reported.

4. Whether the later refinement passes remain worthwhile more broadly. The paper shows that the L = 3 to L = 5 transition improves a smaller and more metric-dependent fraction of objects than the L = 1 to L = 3 transition, with the weakest consistency on ShapeNet mesh F1 (55.7% at K = 4), which raises the question of how to schedule refinement per object.

Target Audience

Researchers and engineers working on 3D generative modeling, neural field representations, and learned compression for 3D assets. The paper is most useful to readers already comfortable with triplane decoders, cross-attention tokenizers, and training objectives for occupancy prediction, since it builds directly on the VecSet and COD-VAE line of work and evaluates against them under a shared protocol. Practitioners focused on inference cost in 3D content pipelines will also find the efficiency protocol and the two-token stage-2 generation results relevant.

Authors’ abstract

Compact token sequences are essential for efficient 3D generation. However, existing 3D tokenizers typically organize latent representations either over spatial regions or as fixed-size sets of global tokens, both suffering sharp reconstruction degradation when compressed to extremely low token budgets. In this paper, we present ZipTok3D, a 3D tokenizer designed for high-fidelity reconstruction from extremely short token sequences. Its key idea is to organize object geometry into progressively informative global-token prefixes and unfold these compact representations through iterative decoding. Specifically, nested dropout randomly truncates the latent sequence after encoding during training and requires each retained prefix to reconstruct the complete object, thereby prioritizing essential geometric information in the leading tokens. The decoder then repeatedly applies a parameter-shared Transformer block to recover fine-grained geometry from each prefix without a separate generative sampling stage. With the same token dimension, ZipTok3D achieves reconstruction quality comparable to the 32-token COD-VAE baseline using only one token on ShapeNet and four on TRELLIS, yielding $32\times$ and $8\times$ shorter token sequences, respectively.