Patch-Derived or Register Tokens? Comparing Variable-Length Image Tokenizers

AI summary

Overview

Research area: Computer vision — image tokenization for generative modeling (vector-quantized autoencoders, variable-length latent representations, autoregressive image generation).

Technical level: Intermediate. The paper assumes familiarity with Vision Transformers, vector quantization and autoregressive image models, but its central comparison is conceptual and its results are readable by anyone who knows what a tokenizer does.

Scope (one sentence): A controlled, same-everything-but-the-bottleneck comparison of patch-derived tokens and learned register tokens for variable-length image tokenization, measuring reconstruction quality, generation quality, inference speed and the spatial structure that emerges in each latent space.

What This Paper Is About

Most image tokenizers place latents on a downsampled 2D grid, so the number of tokens grows with image resolution — a serious cost for high-resolution image generation, video and world modeling. A newer line of work decouples sequence length from resolution, but two rival designs exist: patch-derived tokens (latents derived from spatially arranged encoder features) and register tokens (extra learned tokens appended to the patch sequence and kept as the representation). Prior studies compared them across different backbones, quantizers, objectives and downstream models, so nobody could say which observed differences came from the bottleneck itself. This paper isolates that one design choice.

Key Contributions

1. A controlled comparison. The authors present what they describe as the first controlled comparison of patch-derived and register bottlenecks for flexible-length image tokenization, holding the backbone, quantizer, training objective and training regime fixed, so architecture and training differences no longer confound the result.
2. A three-way trade-off characterization. They map reconstruction quality, downstream generation quality and computational cost against token budgets spanning 1 to 256 tokens (reconstruction) and 16 to 256 tokens (generation).
3. A spatial-influence analysis of latent tokens. Using a one-token-at-a-time corruption procedure plus a spatial-entropy measure, they quantify how localized or diffuse each latent token's effect on the reconstruction is, and how representation alignment (REPA) changes it.
4. An empirical result on representation alignment. They show that aligning tokenizer decoder features with DINOv2-L features substantially improves the generative performance of patch-derived tokens, letting them match or beat register tokens depending on the budget.

Main Findings

• Extreme compression favors patch-derived tokens. Without representation alignment, reconstruction at high token counts is comparable between the two designs, and register tokens are slightly better at intermediate budgets. But at low token counts patch-derived tokens are clearly stronger — at 8 tokens, rFID is 40.90 for patch-derived versus 77.93 for register. At the very lowest budgets (1 and 4 tokens) both approaches fail; the reported rFID values there are 395.80 / 344.65 at 1 token and 238.72 / 223.42 at 4 tokens for patch-derived / register respectively.

• Representation alignment cuts both ways. Adding REPA (λ = 1, aligning the first decoder layer to DINOv2-L via a 3-layer MLP) drops patch-derived rFID at 8 tokens from 40.90 to 11.60 and at 4 tokens from 238.72 to 73.89. For register tokens it helps at 4 tokens (212.21 versus 223.42) but deteriorates rFID and LPIPS at high budgets — 1.67 versus 1.29 rFID, and LPIPS 0.308 versus 0.258, at 256 tokens.

• Register tokens win on generation without REPA. In the class-conditional LlamaGen evaluation, register tokens outperform patch-derived tokens: with LlamaGen-L and no REPA, gFID goes from 9.02 at 16 tokens down to 3.61 at 256 tokens for register, versus 9.00 down to 4.46 for patch-derived.

• REPA reverses that ordering. With REPA and LlamaGen-L, patch-derived tokens reach 6.37 gFID at 16 tokens (versus 7.22 for register) and land at 3.10 at 256 tokens versus 3.11 for register — a tie at the top, a clear patch-derived lead at small budgets.

• Capacity matters. The one clear failure case is LlamaGen-B with register tokens and REPA, whose gFID worsens with more tokens: 9.65 at 16 tokens rising to 15.30 at 256. The authors hypothesize that REPA restructures the latent space in a way a small generative model cannot exploit.

• Patch-derived tokens are cheaper to run. On 256×256 images, batch size 128, one NVIDIA A100: patch-derived encoder throughput is 1622.9 images/s and decoder 1756.8 images/s, constant across budgets, because both process 256 tokens regardless. The register encoder processes 256 patch embeddings plus 256 register tokens and drops to 901.3 images/s; its decoder ranges from 1750.9 images/s at 1 token down to 932.7 at 256 tokens.

• Both designs learn space, one is told about it. Patch-derived tokens largely retain their imposed spatial organization; register tokens develop an implicit 2D structure despite having no predefined layout. Patch-derived tokens show decreasing spatial entropy with increasing token index (progressively more concentrated influence) until REPA partially flattens the trend. Register tokens remain comparatively diffuse, and with REPA a peak at maximum entropy appears — attributable to inactive ("dead") tokens that contribute negligibly, consistent with the diminishing rFID gains at high budgets.

• Generation saturates around 128–192 tokens. Gains are largest up to 128 tokens and diminish thereafter across most configurations.

Methodology in Plain English

The authors build two autoencoders that are identical except for how information flows from encoder to decoder. Both use a Vision Transformer encoder and decoder (depth 12, dimension 768 each), patch size 16, 256×256 images, 256 image patches, and vector quantization into a codebook of 16,384 entries of dimension 8 — 195,962,460 parameters in total, of which 589,824 are the 256 register tokens used in the register variant.

In the patch-derived design, the encoder's kept tokens are copied straight into the decoder, which must turn them back into image patches. In the register design, the encoder sees the patch embeddings plus 256 learned register tokens, keeps only the register tokens, and the decoder uses positional query embeddings to attend into them.

To make one model work at many lengths, both are trained with nested dropout: for each example, a retained length r is sampled uniformly from 1 to n, only the first r tokens go to the decoder, and the rest are thrown away. The authors deliberately sample uniformly over all lengths rather than restricting to grid points or an exponentially spaced set, so that the contribution of each individual token can be evaluated cleanly. The training loss combines an ℓ2 reconstruction term, a perceptual loss, a PatchGAN adversarial loss, the VQ codebook loss, and optionally the REPA alignment loss, trained for 650K steps on ImageNet-1K at batch size 512.

Evaluation happens on three fronts: standard reconstruction metrics (rFID, LPIPS, MAE, PSNR, SSIM) on the ImageNet 256×256 validation split; gFID from 50,000 samples of class-conditional LlamaGen models trained for 1,500,000 steps at batch size 256 (with 2D RoPE swapped for RoPE, since no 2D layout is assumed); and a corruption probe where a single latent token is replaced by a random codebook entry and the per-pixel change in reconstruction is measured and turned into a spatial-entropy score.

Why This Matters

For research, the paper replaces a pile of incomparable anecdotes with a clean ablation: differences previously attributed to "register tokens versus patch tokens" may really have been differences in backbone, quantizer or loss. It also gives a mechanistic account — through the corruption analysis — of why register tokens work at all without a spatial prior, and shows that representation alignment is not a free win, since it can hurt small generative models and adds diffuse, sometimes dead tokens.

Real-world applications implied by the paper's motivation (the paper itself does not report deployed applications):

• High-resolution image generation, where grid tokenizers produce prohibitively long sequences as resolution grows.
• Video generation and world modeling, cited as settings where resolution-driven sequence growth is most painful.
• Compute-constrained deployment, where the patch-derived encoder's 1622.9 images/s versus the register encoder's 901.3 images/s on an A100 translates directly into serving cost.
• Bandwidth- or storage-limited image representation, since a single model can emit ordered prefixes of 16, 32, 64 or 128+ tokens depending on the budget available.

For industry, the practical takeaways are concrete: if you need extreme compression or a small generative model, patch-derived tokens with REPA are the strongest option here; if you have a large generative model and no alignment stage, register tokens are better; and if inference throughput matters, the register encoder costs roughly 1.8× more per image at the same nominal settings.

Future Directions

• A compute-matched comparison. The authors state explicitly that their comparison is not compute-matched: register tokens process more tokens per image, and disentangling bottleneck design from extra training compute is left open.
• Non-autoregressive decoders. The entire downstream evaluation uses autoregressive LlamaGen models, a natural fit for nested dropout's ordered representation; whether diffusion or other non-autoregressive generators can exploit that ordering is untested.
• Diagnosing dead tokens. The maximum-entropy peak for register tokens with REPA points to inactive latents; understanding and preventing them could recover the lost rFID gains at high budgets.
• Why REPA hurts small models. The LlamaGen-B plus register plus REPA degradation is explained only as a hypothesis about insufficient capacity, leaving the latent-space restructuring itself uncharacterized.

Target Audience

Researchers and engineers working on image tokenizers, vector-quantized autoencoders and autoregressive image generation will get the most from this, particularly anyone choosing a bottleneck design for a variable-length or flexible-budget tokenizer. It is also useful for practitioners who need to reason about the compute cost of a tokenizer separately from its quality, and for readers interested in how spatial structure emerges in models that are not given one. Beginners will follow the framing and headline results but will need background on VQ-VAEs and nested dropout to engage with the metrics tables.

Authors’ abstract

Image tokenizers convert pixels into compact latent sequences for efficient generative modeling. Most approaches arrange latents on a downsampled two-dimensional grid, tying sequence length to image resolution. We compare two variable-length strategies: computationally efficient patch-derived tokens, which inherit explicit spatial organization, and more demanding register tokens, which have no predefined spatial layout. Our reconstruction results reveal that patch-derived tokens perform substantially better under extreme compression, whereas register tokens achieve slightly better reconstruction quality at intermediate budgets. At high token budgets, both approaches achieve similar reconstruction quality. For downstream generative tasks, register tokens outperform patch-derived tokens. Adding representation alignment with external pretrained visual encoders substantially improves the generative performance of patch-derived tokens, allowing them to match or outperform register tokens depending on the token budget. We further find that both approaches learn spatially localized representations. Patch-derived tokens largely retain their explicitly imposed spatial organization, while register tokens develop an implicit two-dimensional spatial structure despite having no predefined layout. Representation alignment reduces this spatial organizational constraint, enabling individual tokens to exert a stronger non-local influence.