Hi-SAM: A Hierarchical Structure-Aware Multi-modal Framework for Large-Scale Recommendation

AI summary

Overview

• Research area: Large-scale multi-modal recommender systems, specifically semantic ID-based recommendation that combines multi-modal content encoding with Transformer-based sequential modeling.
• Technical level: Advanced. The paper assumes familiarity with vector quantization (RQ-VAE), contrastive/geometric alignment objectives, rotary position embeddings, mutual information estimation, and KV cache management in decoder-only Transformers.
• Scope: The paper presents Hi-SAM, a two-stage framework for converting item text and images into disentangled discrete semantic tokens and then modeling user behavior sequences with a hierarchy-aware Transformer, validated on two public Amazon 2023 subsets, one industrial dataset, and a live deployment.
• Venue and authorship: Presented at KDD 2026 (Jeju Island, Republic of Korea), arXiv:2602.11799v2. Authors Pingjun Pan, Tingting Zhou, Peiyao Lu, Tingting Fei, Hongxiang Chen (corresponding author), and Chuanjiang Luo are affiliated with Netease Cloud Music, Hangzhou, China; two authors are noted as contributing equally.

What This Paper Is About

Recommender systems increasingly convert items into "semantic IDs" — short sequences of learned discrete codes derived from item text and images — so that models can generalize beyond sparse interaction IDs. The paper argues that existing semantic ID systems fail in two ways: their tokenizers blur together cross-modal shared meaning with modality-specific detail (causing redundancy or "modality collapse"), and standard Transformers treat the resulting token streams as flat sequences, losing the natural hierarchy of user → item → attribute. Hi-SAM aims to fix both ends of this pipeline at once, producing cleaner discrete tokens and an architecture built specifically to consume them.

Key Contributions

1. Hi-SAM framework: A hierarchical structure-aware multi-modal framework for semantic ID-based recommendation, composed of a Disentangled Semantic Tokenizer (DST) and a Hierarchical Memory-Anchor Transformer (HMAT), explicitly targeting the tokenization–architecture gap.
2. Disentangled Semantic Tokenizer: Combines geometry-aware Cross-Modal Alignment (CGA, based on the Gramian Representation Alignment Measure / GRAM) with Disentangled Modal-Residual Quantization (DMRQ), which uses shared codebooks for cross-modal consensus and modality-specific codebooks (with a Parallel Semantically-Guided Recovery mechanism) for residual nuances, enforced by a mutual information minimization constraint using the vCLUB estimator.
3. Hierarchical Memory-Anchor Transformer: Introduces Hierarchical RoPE, which splits positional encoding into orthogonal inter-item and intra-item subspaces with different base frequencies, and Memory-Anchor Attention, which inserts an Anchor Token per item and masks attention so historical items are reachable only through those anchors.
4. Offline and online validation: The paper reports improvements over five baselines across three datasets, a 6.55% gain in the core online business metric, and 35% lower latency in production.

Main Findings

• Offline superiority across all metrics: Hi-SAM outperforms WuKong, HSTU, MTGR, QARM, and PSRQ+MCCA on AUC, GAUC, Cold AUC, and Cold GAUC across the Book, Movies & TV, and Industrial datasets. On the Industrial dataset, Hi-SAM-Small raises GAUC from 0.6068 (QARM) and 0.6131 (PSRQ+MCCA) to 0.6410, and Cold GAUC from 0.5477 and 0.5571 to 0.5835.
• Progressive training gives the strongest results: The w/ PT+SFT variant reaches AUC 0.7337, GAUC 0.6443, Cold AUC 0.8028, and Cold GAUC 0.5963 on the Industrial dataset, topping Hi-SAM-Large (0.7303 / 0.6432 / 0.7957 / 0.5913) and Hi-SAM-Small (0.7293 / 0.6410 / 0.7886 / 0.5835).
• Favorable scaling: Moving from the 4-layer Small configuration to the 12-layer Large configuration yields consistent gains, e.g., Cold GAUC from 0.5835 to 0.5913 on the Industrial dataset.
• Tokenizers matter more than backbones in decoupled ablations: Swapping in DST raises the HSTU base from 0.6640 AUC / 0.6067 GAUC / 0.6934 Cold AUC / 0.5304 Cold GAUC to 0.7049 / 0.6244 / 0.7798 / 0.5795, whereas substituting QARM or PSRQ tokenizers yields much smaller changes (0.6622/0.6049/0.7292/0.5481 and 0.6703/0.6084/0.7443/0.5446). Adding HMAT on top of QARM improves it from 0.6420 / 0.6068 / 0.7467 / 0.5477 to 0.7010 / 0.6270 / 0.7565 / 0.5573.
• Both alignment and quantization components are necessary: Removing CGA drops full Hi-SAM from 0.7293 / 0.6410 / 0.7886 / 0.5835 to 0.6813 / 0.6166 / 0.7327 / 0.5465; removing DMRQ gives a smaller drop to 0.7163 / 0.6347 / 0.7855 / 0.5822.
• Hierarchical positional encoding beats alternatives: Replacing H-RoPE with absolute positions yields 0.7241 / 0.6382 / 0.7824 / 0.5730, with relative absolute bias positions 0.7247 / 0.6399 / 0.7832 / 0.5807, and with 1D-RoPE 0.7260 / 0.6402 / 0.7850 / 0.5823 — all below the full model's 0.7293 / 0.6410 / 0.7886 / 0.5835.
• Semantic IDs help cold-start more than overall ranking in prior work: The paper notes PSRQ+MCCA surpasses HSTU in Cold GAUC by approximately 5.0% on the Industrial dataset (0.5571 vs. 0.5304) and 1.0% on Movies & TV (0.7434 vs. 0.7359), yet QARM yields only a 0.15% GAUC gain over HSTU on Book (0.6450 vs. 0.6440) and underperforms HSTU on Industrial (0.6068 vs. 0.6087).
• Production impact: Hi-SAM is deployed on a large-scale social platform serving millions of daily users, achieving a 6.55% gain in the core online business metric and 35% lower latency.
• Inference efficiency via anchor-based cache compression: Because historical items are only reachable through Anchor Tokens, the paper reports that fine-grained tokens can be losslessly evicted from the KV cache, reducing memory usage by roughly a factor of L_i and attention complexity from O((K · L_i)²) to approximately O(K²) for historical context.

Methodology in Plain English

The framework runs in two stages.

Stage 1 — Disentangled Semantic Tokenizer. Each item's raw modalities (for example image and text) pass through separate encoders and projection heads that map them into a common 256-dimensional space. Embeddings are normalized onto a unit hypersphere, and the Gramian Representation Alignment Measure (GRAM) aligns all modalities at once by minimizing the volume of the parallelotope their vectors span, using a symmetric contrastive loss (a data-to-anchor term and an anchor-to-data term). This replaces pairwise alignment such as CLIP. The aligned vectors are then quantized in a coarse-to-fine manner: a fused global representation is quantized with RQ-VAE using N_sh = 3 shared layers to capture cross-modal commonalities, and the remaining residual is unfolded into H = 8 latent subspaces, where each original modality vector acts as a query that selects the subspaces relevant to it (Parallel Semantically-Guided Recovery). Each recovered modality-specific feature is quantized into its own codebook entry. A mutual information penalty, estimated with the vCLUB estimator, keeps the shared and specific codes statistically independent, and the total objective combines reconstruction, codebook commitment loss, and this disentanglement term. Each item ends up as 6 discrete semantic tokens.

Stage 2 — Hierarchical Memory-Anchor Transformer. The token stream interleaves the user profile, each item's tokens, a special Anchor Token, and the action token. Every token receives a coordinate (m, n): m is the global item order (0 for user-profile tokens), and n is the position within the item. Hierarchical RoPE splits the embedding dimension in half — the first half rotates according to m using base frequency 10^4 for long-range stability, the second half rotates according to n using base frequency 100 for fine-grained local sensitivity. Because the two halves rotate independently, the attention score decomposes cleanly into an inter-item term depending on Δm and an intra-item term depending on Δn. Memory-Anchor Attention then applies a structural mask with three pathways: user-profile tokens (m_k = 0) are globally visible, tokens in the same item (m_q = m_k) see each other fully, and earlier items (m_k < m_q) are visible only through their Anchor Tokens; all other positions are masked out.

Training uses supervised fine-tuning on action tokens with the memory-anchor mask enabled, optionally preceded by a semantic pre-training stage of next-token prediction over the whole stream with the mask disabled. The tokenizer stays frozen throughout to keep the discrete space stable. For inference, candidates are scored in one forward pass with a block-diagonal mask, and their inter-item position coordinate is reset to the same value so batch order introduces no positional bias. A logical position mapping preserves the original coordinates for H-RoPE even after redundant keys and values are physically evicted.

Experimental setup. Three datasets are used: Amazon 2023 Movies & TV (657.2K users, 197.9K items, 7.4M interactions, average sequence length 11.25) and Books (0.78M users, 0.49M items, 9.5M interactions, average 12.66), plus an industrial dataset from a large online dating platform collected September–December 2025 (6.3M users, 1.38M items, 521M interactions, average 82.30). Ratings greater than 3 count as positive. Interactions are sorted chronologically with the first 90% for training and the final 10% for testing; users with fewer than 10 interactions are treated as cold-start. Metrics are AUC and GAUC, measured on the full test set and the cold-start subset. Online evaluation uses A/B testing on Response Rate and Response Depth. Encoders are BLIP-2 (2.7B) and a SASRec-based encoder; the base HMAT is 4 layers with hidden size 512, and the scaled Large variant is 12 layers. All models train on 8 NVIDIA A100 GPUs with Adam and a maximum sequence length of 300.

Why This Matters

• Research impact: The paper reframes semantic ID recommendation as a joint tokenizer-and-architecture design problem rather than a pure quantization problem, and it offers a concrete alternative to flat-sequence Transformers for hierarchical data. The GRAM-based alignment and the mutual-information disentanglement constraint give a template for avoiding modality collapse that does not depend on pairwise contrastive objectives.
• Real-world applications:
  • Music and content streaming platforms that must recommend newly released items with no interaction history, where the reported 6.55% core-metric gain and 35% latency reduction were observed.
  • Dating and social platforms, the source of the industrial dataset, where profile images and personal text are the primary signals and cold-start users are common.
  • E-commerce and video catalogs using cover images, titles, categories, and brand metadata to recommend long-tail inventory.
  • Ad or feed ranking systems where candidate scoring must happen in a single forward pass under strict latency budgets, benefiting from the one-pass parallel ranking and lossless KV cache eviction.
• Industry relevance: The work is evaluated at industrial scale (521M interactions, sequence lengths averaging 82.30, millions of daily users) and reports both offline metrics and a live A/B result, which is more operational evidence than most academic recommendation papers provide. The anchor-based cache compression addresses a real serving-cost bottleneck created by expanding one item into multiple tokens.

Future Directions

• The experiments section in the available content reports results for RQ1 (overall offline performance) and the two ablation tables; the reported alignment/disentanglement analysis (RQ3) and detailed scaling study (RQ4) are not present in the excerpt, so their specific findings remain to be seen in the full paper.
• The vCLUB mutual information estimator and the weighting hyperparameters β (quantization stability) and λ (disentanglement) are introduced but their sensitivity is only pointed to in the appendices, leaving open how robust the shared/specific split is under different modality counts.
• The tokenizer is frozen in both training stages to keep the discrete space stable, which raises the question of whether joint or periodically refreshed tokenizer training could improve quality further.
• The framework is instantiated with 6 tokens per item (3 shared, 1 per modality) and H = 8 subspaces; how these choices trade off against sequence length, cache savings, and accuracy is a natural next question, as is extending the disentanglement scheme beyond its current modality configuration.
• Historical items are summarized purely through Anchor Tokens, so the design open question is how much fine-grained historical detail can be discarded before long-range personalization degrades.

Target Audience

• Recommender systems researchers and engineers working on semantic ID generation, multi-modal fusion, or large-scale sequential ranking models.
• Practitioners responsible for production ranking systems who care about cold-start performance, serving latency, and KV cache memory costs.
• Machine learning engineers interested in position embedding design (multi-dimensional RoPE) and structured attention masking.
• Readers with foundational knowledge of vector quantization, contrastive learning, and Transformer internals will gain the most, since the methodology assumes familiarity with those building blocks.

Authors’ abstract

Multi-modal recommendation has gained traction as items possess rich attributes like text and images. Semantic ID-based approaches effectively discretize this information into compact tokens. However, two challenges persist: (1) Suboptimal Tokenization: existing methods (e.g., RQ-VAE) lack disentanglement between shared cross-modal semantics and modality-specific details, causing redundancy or collapse; (2) Architecture-Data Mismatch: vanilla Transformers treat semantic IDs as flat streams, ignoring the hierarchy of user interactions, items, and tokens. Expanding items into multiple tokens amplifies length and noise, biasing attention toward local details over holistic semantics. We propose Hi-SAM, a Hierarchical Structure-Aware Multi-modal framework with two designs: (1) Disentangled Semantic Tokenizer (DST): unifies modalities via geometry-aware alignment and quantizes them via a coarse-to-fine strategy. Shared codebooks distill consensus while modality-specific ones recover nuances from residuals, enforced by mutual information minimization; (2) Hierarchical Memory-Anchor Transformer (HMAT): splits positional encoding into inter- and intra-item subspaces via Hierarchical RoPE to restore hierarchy. It inserts Anchor Tokens to condense items into compact memory, retaining details for the current item while accessing history only through compressed summaries. Experiments on real-world datasets show consistent improvements over SOTA baselines, especially in cold-start scenarios. Deployed on a large-scale social platform serving millions of users, Hi-SAM achieved a 6.55% gain in the core online metric.