MoToRec: Sparse-Regularized Multimodal Tokenization for Cold-Start Recommendation

AI summary

Overview

Research area: Multimodal recommender systems, graph neural networks, vector quantization, and item cold-start recommendation.

Technical level: Advanced — the paper assumes familiarity with GNN-based collaborative filtering, vector quantization, residual quantization (RQ-VAE), and contrastive learning objectives.

Scope: This paper proposes MoToRec, a framework that reframes multimodal recommendation as a discrete semantic tokenization problem using a sparsely-regularized RQ-VAE, with the goal of producing robust and interpretable item representations for cold-start items.

What This Paper Is About

Graph neural network recommenders perform well when historical interaction data is dense, but their performance collapses for new items with few or no interactions — the item cold-start problem. Prior multimodal methods try to fix this by aligning continuous visual and textual embeddings with collaborative signals, but the authors argue that this continuous alignment is noisy and unreliable, a phenomenon they name "semantic fog," which produces suboptimal, out-of-distribution representations especially for cold-start items.

MoToRec instead converts raw multimodal features into a structured sequence of discrete tokens drawn from a learnable codebook, where each token represents a disentangled semantic concept, and then fuses those codes with pure collaborative signals through a hierarchical graph encoder.

Key Contributions

1. Reframing multimodal recommendation as discrete semantic tokenization. Instead of aligning continuous high-dimensional vectors, the paper learns compositional semantic codes to explicitly address the "semantic fog" and out-of-distribution issues in cold-start scenarios.

2. The MoToRec architecture. An end-to-end model that integrates three components: a sparsely-regularized Residual Quantized Variational Autoencoder (RQ-VAE) tokenizer, an adaptive rarity amplification mechanism, and a multi-source graph encoder for signal fusion.

3. A sparsity-inducing regularization for the codebook. A KL-divergence penalty drives the aggregate posterior distribution of codebook usage toward a sparse Bernoulli prior with small mean ρ, which the authors argue promotes disentangled representations by acting as a proxy for minimizing mutual information between codebook activations.

4. Comprehensive evaluation on three large-scale datasets (Baby, Sports, Clothing), showing improvements over state-of-the-art methods in both overall and cold-start settings.

Main Findings

• MoToRec outperforms all baselines on all three datasets. Reported N@20 scores are 0.0473 (Baby), 0.0529 (Sports), and 0.0456 (Clothing), with corresponding R@20 scores of 0.1077, 0.1163, and 0.1014.

• Reported improvement over ID-only models is up to 88%. The improvement rows show gains over the best traditional baseline (MF-BPR, LightGCN, SimGCL, LayerGCN) of 34.03%, 31.34%, 33.81%, 33.24% on Baby; 30.45%, 26.55%, 32.42%, 27.78% on Sports; and 85.44%, 79.15%, 88.00%, 84.62% on Clothing (R@10, R@20, N@10, N@20 respectively).

• Reported improvement over state-of-the-art multimodal methods is up to 11.57%. Against the best multimodal baseline (LGMRec, LPIC), improvements are 10.95%, 8.57%, 11.57%, 10.00% on Baby; 6.38%, 4.49%, 8.79%, 9.07% on Sports; and 9.55%, 7.76%, 10.26%, 8.57% on Clothing.

• The cold-start advantage is the largest effect. The paper reports an uplift of up to 12.58% in N@20 on items with the fewest interactions, attributing this to representing novel items as a composition of known concepts.

• Removing the RQ-VAE causes the largest ablation drop. In the ablation table, w/o RQ-VAE reduces Baby cold-start N@20 from 0.0147 to 0.0092, Sports cold-start N@20 from 0.0183 to 0.0115, and Clothing cold-start N@20 from 0.0170 to 0.0097 — the authors cite this as evidence for discrete tokenization over continuous feature mapping.

• All other components contribute. Removing Adaptive Rarity Amplification (w/o ARA), Sparsity regularization, Contrastive Loss (w/o CL), Hybrid Fusion (w/o HF), or Homogeneous Graph Enhancement (w/o HGE) each degrades both overall and cold-start performance versus full MoToRec.

• Optimal hyperparameters are dataset-dependent. The sparse Baby dataset favors γ = 0.05 and a compact codebook of K = 512, while the visually rich Clothing dataset favors γ = 0.01 and K = 1024. On Sports, cold-start results were found to be highly sensitive to sparsity deviations even when overall performance stayed robust.

• Discrete codes are human-interpretable and compositional. On the Clothing dataset, code <c_121> reliably activates for the color "red" and <a_34> for the category "T-shirt". A cold-start "red minimalist T-shirt" activates the combination <c_121> (color: red), <s_5> (style: minimalist), and <a_34> (category: T-shirt).

• Embeddings become better organized. A t-SNE projection of 500 sampled Sports items shows the full model learns a more organized semantic manifold than variants without RQ-VAE or without sparsity, with cold-start items integrated among semantic neighbors rather than isolated as outliers.

• Runtime is competitive. On Sports, MoToRec trains in 11.33s per epoch versus FREEDOM at 12.2s, LGMRec at 12.0s, and LightGCN at 6.5s — described as roughly 74% overhead relative to LightGCN, attributed solely to the tokenizer, since graph propagation preserves LightGCN efficiency. Test time is 3.8s per batch.

Methodology in Plain English

Setup. The model takes a sparse user-item interaction matrix plus two content features per item: a visual vector from a pre-trained vision transformer (BEiT) and a textual vector from a sentence embedding model (BGE). The goal is to produce user and item embeddings whose dot product ranks items.

Step 1 — Adaptive rarity amplification. Items are labeled cold if their interaction degree falls below a threshold τ. Rare-but-not-empty cold items (degree greater than zero) are assigned a weight of (log₂(dᵢ + 2))⁻¹; all other items get a weight of 1.0. This inverse logarithmic weighting amplifies the learning signal for the least frequent items.

Step 2 — Discrete tokenization. For each modality, an MLP encoder maps the raw feature into a latent vector, which is then quantized by a cascade of residual quantizers: each stage picks the nearest codebook prototype, subtracts it to form a residual, and passes that residual to the next stage. The final quantized representation is the sum of all selected prototypes. To prevent entangled codebook usage, a KL penalty pushes the average activation probability of each codeword toward a sparse Bernoulli prior with small mean ρ. The tokenizer is trained with reconstruction loss, a commitment loss, and this sparsity loss.

Step 3 — Hierarchical multi-source graph encoding. Three separate propagation channels run in parallel over the user-item graph, each using the LightGCN propagation rule with L = 2 layers: a visual channel initialized with tokenized visual embeddings, a textual channel initialized with tokenized textual embeddings, and a collaborative channel initialized with standard learnable ID embeddings. Keeping them separate preserves each signal's purity.

Step 4 — Fusion. The visual and textual representations are combined with a hybrid strategy that mixes concatenation with attention, balanced by a hyperparameter α. The combined multimodal embedding is then joined with the collaborative embedding through a gated residual connection.

Step 5 — Optimization. The full model is trained end-to-end on a composite loss: a BPR ranking loss, an InfoNCE contrastive loss, the RQ-VAE tokenization loss weighted by the rarity weight wᵢ, and L2 regularization. Embeddings are 64-dimensional, trained with Adam, and early stopping with patience of 20 epochs monitors R@20 on validation.

Why This Matters

Impact on research. The paper argues that continuous cross-modal alignment is a structural bottleneck rather than a tuning problem, and that converting features into discrete, compositional codes offers an alternative path to generalization for items with no interaction history. It positions vector quantization as a way to learn interpretable semantic vocabularies rather than only as a compression or sequence-generation tool, and it introduces vocabularies ("semantic fog", "alignment haze") for a failure mode that applies beyond recommendation.

Real-world applications (as implied by the domains studied):

• E-commerce product feeds, where newly listed items have no purchase or click history and must be recommended from images and descriptions alone.
• Content and media platforms that need to place newly uploaded titles into an existing semantic structure for immediate recommendations.
• Any catalog-facing system where categories such as color, style, and product type need to be represented in a form that humans can inspect and audit.
• Deployment-constrained settings, since the paper reports the tokenization overhead is modest and inference cost is on par with other high-performance models, avoiding the cost of running large language models as feature extractors.

Industry relevance. The paper explicitly frames its approach as an alternative to LLM-based recommendation, citing the high computational cost of LLMs as a practical hurdle; distilled discrete codes are presented as a scalable substitute. The reported training and test timings support the case that the accuracy gains do not come with prohibitive serving cost.

Future Directions

• Calibrating the tokenizer per domain. The hyperparameter study shows cold-start performance is highly sensitive to sparsity settings, and the paper's own results imply that optimal codebook size and sparsity strength vary with dataset characteristics. Automating that calibration is an open problem.

• Explaining the ablation gap for Homogeneous Graph Enhancement. HGE is listed among the ablated components and shown to matter, but the method section attributes higher-order graph structure capture to it only in passing; clarifying exactly what HGE contributes is a natural extension.

• Extending interpretability claims beyond one dataset. The case study demonstrating codes for "red", "minimalist", and "T-shirt" is performed on the Clothing dataset only; validating that the same disentanglement holds across Baby and Sports is not reported.

• Scaling and generalizing the tokenization. The paper validates on three Amazon review datasets with 64-dimensional embeddings and L = 2 graph layers; whether the discrete token approach holds for much larger catalogs, additional modalities, or other cold-start regimes such as cold-start users is not reported.

Target Audience

This paper is aimed at recommender-systems researchers and practitioners working on multimodal fusion, cold-start mitigation, or graph-based collaborative filtering. It is also relevant to engineers evaluating whether discrete representations can replace expensive LLM-based feature extraction in production, and to readers interested in interpretable or disentangled representation learning via vector quantization. Readers without a background in quantization-based generative models or GNN propagation will find the technical sections demanding.

Authors’ abstract

Graph neural networks (GNNs) have revolutionized recommender systems by effectively modeling complex user-item interactions, yet data sparsity and the item cold-start problem significantly impair performance, particularly for new items with limited or no interaction history. While multimodal content offers a promising solution, existing methods result in suboptimal representations for new items due to noise and entanglement in sparse data. To address this, we transform multimodal recommendation into discrete semantic tokenization. We present Sparse-Regularized Multimodal Tokenization for Cold-Start Recommendation (MoToRec), a framework centered on a sparsely-regularized Residual Quantized Variational Autoencoder (RQ-VAE) that generates a compositional semantic code of discrete, interpretable tokens, promoting disentangled representations. MoToRec's architecture is enhanced by three synergistic components: (1) a sparsely-regularized RQ-VAE that promotes disentangled representations, (2) a novel adaptive rarity amplification that promotes prioritized learning for cold-start items, and (3) a hierarchical multi-source graph encoder for robust signal fusion with collaborative signals. Extensive experiments on three large-scale datasets demonstrate MoToRec's superiority over state-of-the-art methods in both overall and cold-start scenarios. Our work validates that discrete tokenization provides an effective and scalable alternative for mitigating the long-standing cold-start challenge.