DuoLM: Fast and Effective Personalized Recommendation with Language Models

AI summary

Overview

Research area: Information retrieval / recommender systems, specifically LLM-based personalized recommendation combined with LLM serving-system engineering (cs.IR).

Technical level: Intermediate — the paper assumes familiarity with transformer language models, knowledge distillation, KV caching, and inference runtimes such as vLLM and SGLang.

Scope: DuoLM is a two-model framework plus custom serving runtime that replaces item text with precomputed single-vector item representations so that a personalized ranker LLM can score many candidates per request at high throughput, evaluated on Amazon Books and Video Games.

What This Paper Is About

Personalized recommenders built on language models must judge many candidate items against a user's history and feedback, but feeding full item text into a model for every candidate is expensive online, which caps serving throughput. The paper asks whether item text can be compressed once into a reusable vector per item, and whether the serving stack can be reorganized so that shared user history is computed only once per request instead of once per candidate. The goal is recommendation quality close to a full-text "teacher" model while serving far faster.

Key Contributions

1. A compact, reusable representation architecture. An encoder LLM compresses each item's text into a single hidden-state vector (the final-layer EOS state), which replaces the item's text tokens inside a personalized ranker LLM through a scale-calibrated continuous input interface. Item vectors are user-independent, so they are computed offline and reused across users and requests, while the ranker still conditions each score on the individual user's ordered history and feedback.

2. End-to-end training with labels plus teacher distillation. The encoder, interface, and ranker are trained jointly with recommendation cross-entropy on the observed target and a KL-divergence loss that matches the preference distribution of a frozen full-text teacher over the same candidate group, combined with a distillation weight that is increased linearly to a final value.

3. A serving runtime co-designed with the model. Virtual token IDs (VIDs) map item IDs into a GPU-resident table of precomputed embeddings so requests carry IDs rather than vectors; a multi-item scoring (MIS) path on SGLang prefills the user history once and shares its prefix KV cache across candidate branches using a tree-structured attention mask and corrected logical position IDs; segmented prefill CUDA Graphs are adapted to the MIS path to cut host dispatch and kernel launch overhead.

4. Quality and throughput evidence plus ablations. The paper reports parity-or-better quality versus the teacher on Books next-item recommendation and comparable quality on Video Games and Books rating prediction, together with ablations over readout mechanism, feedback placement, distillation settings, self-alignment, and each runtime component.

Main Findings

• Books recommendation beats the teacher: NDCG@10 of 0.5586 for DuoLM (FIR, m=1) versus 0.5440 for the full-text teacher, and 0.5508 for HLLM. HLLM does achieve a higher HR@1 on Books (0.4022 versus 0.3910).
• Video Games recommendation is comparable, not better: NDCG@10 of 0.5298 versus the teacher's 0.5313; HR@1 0.3240 versus 0.3237; HR@10 0.7650 versus 0.7697. HLLM reaches 0.5025 NDCG@10 there.
• Rating prediction is close to the teacher: average NLL across history lengths of 0.9263 versus the teacher's 0.9233. DuoLM has lower expected-rating MAE (0.6415 versus 0.6515) and RMSE (0.7725 versus 0.7795), while the teacher has lower median-based MAE (0.5308 versus 0.5362) and RPS (0.0962 versus 0.0968).
• Throughput on one H100: about 26,400 candidate scores per second at concurrency 64 (26,403.98 in the table), up from approximately 8,870 at concurrency 1. That is 22.7× to 60.1× the teacher served by vLLM with automatic prefix caching, and 37.6× to 109.9× the teacher on SGLang.
• Latency also improves: at concurrency 64, full-request p99 latency is 536.90 ms for DuoLM versus 17,093.82 ms for the teacher on vLLM and 28,189.04 ms on SGLang.
• MIS is the dominant runtime win: with the model fixed, multi-item scoring improves throughput by 4.81× at concurrency 1 and 29.3× at concurrency 64. CUDA Graph adds a further 3.54× at concurrency 1 but only 1.06× at concurrency 64, so graph replay helps most at low concurrency and history sharing helps most at high concurrency.
• VIDs shrink requests: the paper reports VIDs reduce average request payloads by over 90% versus full-vector transmission in the rating benchmark; in the separate FIR rating diagnostic (m=8) the reduction is 99.0% with a 66.5% throughput gain at concurrency 8. FP16 vector transfer reduced request size but did not consistently improve throughput.
• LastToken readout wins: a plain EOS hidden state gives the best ranking scores (HR@1 0.4024, HR@10 0.7452, MRR@10 0.5049, NDCG@10 0.5621) against MEM1 (0.3900 / 0.7374 / 0.4982 / 0.5555) and Perceiver1 (0.3588 / 0.7320 / 0.4714 / 0.5335), adds zero parameters, and has the shortest update time (7.39 h versus 7.54 h and 8.96 h over 7,424 updates). Perceiver1 adds 12,599,296 parameters; MEM1 adds 1,024.
• Feedback in the ranker is the default: putting feedback in the encoder (FIE) gives slightly better rating NLL (0.9236 versus 0.9249) but lower Books NDCG@10 (0.5538 versus 0.5586), so FIR is chosen because item vectors stay reusable independent of feedback.
• Distillation helps: CE-only training gives HR@1 0.3748 and NDCG@10 0.5502, versus 0.4024 and 0.5621 with the default CE+KD, at a small HR@10 cost (0.7474 versus 0.7452). Raising the distillation weight to β=0.9 drops NDCG@10 to 0.5428 and raising temperature to τ=2 gives 0.5568, so defaults are kept.
• Self-alignment is not worth it: adding full-text self-alignment lowers NDCG@10 from 0.5621 to 0.5554 and raises cumulative training update time from 7.39 to 16.95 hours (2.29×).
• More history helps: going from K=1 to K=10 lowers NLL from 0.9848 to 0.8806, median MAE from 0.5705 to 0.5056, and RPS from 0.1044 to 0.0911, even with one vector per item.
• CPU preparation can bottleneck: at 100 requests/s, going from one to two input-compilation processes brings the single-vector rating service close to the offered request rate.
• MIS is exact under stated conditions: for the same ranker with identical input embeddings and dropout disabled, MIS scores equal standalone scores in exact arithmetic, and MIS removes (K_q−1)·L_q(L_q+1)/2 repeated prefix query–key pairs per layer.

Methodology in Plain English

The system splits the job between two language models, both initialized from 0.6B Qwen3 backbones (28 layers, hidden dimension 1,024). The first, an encoder, reads one item's templated text (titles, categories, features, descriptions; capped at 512 tokens for recommendation and 256 for rating) and outputs a single vector from its last position. A small normalization-and-scaling step adjusts that vector so its magnitude matches the ranker's own token embeddings; the scale is estimated once from the ranker's embedding table and then fixed, with one learnable gain initialized to zero.

The second model, the ranker, receives a prompt in which each history item's text is replaced by its vector, immediately followed by the user's rating text, and the candidate item is likewise represented by its vector. A single linear head reads the final hidden state and produces a scalar score, so ranking needs no autoregressive generation. A separate full-text teacher with the same style of prompt scores candidate groups on text; its cached scores supply a soft target distribution. Training minimizes a weighted mix of cross-entropy on the observed target and KL divergence to the teacher's temperature-softened distribution over the same candidates, with item vectors recomputed in every forward pass so gradients flow back into the encoder.

Offline, item vectors are exported to an FP16 table on the GPU and addressed by virtual IDs offset past the native vocabulary, so an embedding lookup returns the item vector directly. Online, the runtime builds the history prefix once, packs all candidates after it with a mask that lets each candidate see the prefix and itself but no other candidate, and assigns all candidates the same logical position so their scores match what standalone scoring would produce. CUDA Graphs then replay the captured GPU work for the segmented prefill.

Evaluation used Amazon Reviews 2023. Rating prediction used 10,000 training and 1,000 development users with nested histories of length 1, 3, 5, and 10 (8,757 development targets, 35,028 predictions), scored by NLL, MAE_med, MAE_exp, RMSE, and RPS. Next-item recommendation used 50,000 training users per dataset (237,558 Books queries, 233,868 Video Games queries; catalogs of 495,063 and 25,611 items), with seven training negatives per target (four uniform, three retrieved by history content) and evaluation on one target plus 99 fixed randomly sampled unobserved items for 5,000 Books and 3,000 Video Games users. Serving was measured on a single H100 80GB with 100 candidates per request.

Why This Matters

The paper argues that the practical value of LLM recommenders depends on quality and serving efficiency, and it shows a route where the two are designed together rather than sequentially: representations are made reusable offline, and the runtime exploits the resulting structure (one shared prefix, many independent candidates) instead of treating each candidate as a separate LLM call. It also quantifies a point that is easy to miss — that saving model FLOPs does not automatically save wall-clock time, since scheduling, payload, and host-side preparation can dominate.

Real-world applications:

• E-commerce product ranking and "you may also like" modules, where one request must score many candidates against a long purchase history under tight latency budgets.
• Content and media discovery feeds (streaming, news, articles), where item text is rich but request volume is high.
• Cold-start and long-tail catalog serving, because item vectors derive from content and pretrained knowledge rather than only from interaction IDs.
• Rating or preference prediction surfaces (for example, predicting a star rating from prior ratings), which the paper evaluates directly on Books.

Industry relevance: the gains are reported on commodity serving stacks (vLLM, SGLang) and a single H100, and the techniques — embedding-table indirection via virtual IDs, shared-prefix multi-candidate scoring, CUDA Graph replay, and parallel CPU input preparation — are directly portable to production recommendation services. The reported p99 latency reduction from 17,093.82 ms to 536.90 ms at concurrency 64 is the kind of number that decides whether a model can be deployed at all.

Future Directions

• Beyond two datasets and 0.6B backbones. The authors state that results may not generalize to other domains, larger models, or settings with rapidly changing user preferences; scaling the encoder–ranker pair and testing other catalogs is the obvious next step.
• Harder candidate sets. Evaluation uses one target plus 99 randomly sampled unobserved items, which the authors note may be easier to distinguish than candidates from a strong retrieval system; testing against retrieved candidates is left open.
• Mitigating single-vector information loss. Compressing each item to one vector may discard details relevant to particular users, and offline vectors must be regenerated whenever item content or encoder parameters change — an open cost and quality question.
• Accounting for the full cost and broader hardware. Throughput numbers cover online serving only and exclude training and offline encoding, and were measured on an H100 under controlled loads; behavior under other hardware, traffic patterns, and latency constraints needs evaluation.

Target Audience

Machine learning engineers and applied researchers building LLM-based recommender or ranking systems who care about both model quality and serving cost; inference-runtime and platform engineers interested in shared-prefix KV reuse, CUDA Graph capture, and embedding-table tricks; and recommender-systems researchers who want a concrete, measured counterpoint to full-text cross-encoder scoring. Readers without background in KV caching or knowledge distillation will find the model sections accessible but the runtime sections demanding.

Authors’ abstract

Language models can make personalized recommendations from rich item content, user history, and feedback. However, evaluating many candidates with full-text inputs incurs substantial online computation and limits serving throughput. We introduce DuoLM, a framework for fast and effective personalized recommendation that integrates a compact recommendation model with an efficient serving runtime. An encoder LLM learns compact item representations for offline reuse, while a personalized ranker LLM combines them with user history and feedback for recommendation. We train both models end to end with recommendation supervision and teacher distillation. The runtime further accelerates inference through virtual token IDs for embedding access, shared history computation, and CUDA Graph replay. Benchmark results show that our framework achieves recommendation quality comparable to that of the expert teacher model. On a single H100, it delivers approximately 26,400 candidate scores per second, about $60\times$ the throughput of the teacher served with vLLM and automatic prefix caching.