Make It Long, Keep It Fast: End-to-End 10K Long User Behavior Sequence Modeling for Billion-Scale Douyin Recommendation

AI summary

Overview

Research area: Large-scale recommender systems — specifically long user-behavior sequence modeling for short-video ranking (Douyin / TikTok).

Technical level: Advanced. The paper assumes familiarity with attention mechanisms, distributed training, retrieval-augmented CTR stacks, and serving latency budgets.

Scope: The paper describes and evaluates a production deployed end-to-end system — Stacked Target Cross Attention (STCA), Request Level Batching (RLB), and length-extrapolative training — that scales history length to 10k tokens at billion-scale recommendation without exceeding production latency and cost budgets. (The body title reads "Make It Long, Keep It Fast: End-to-End 10k-Sequence Modeling at Billion Scale on Douyin Recommendation"; the work was published at WWW '26, DOI 10.1145/3774904.3792811, arXiv:2511.06077v3.)

What This Paper Is About

Short-video recommenders like Douyin have user histories thousands of videos long, but most industrial systems avoid modeling them end-to-end: they retrieve a small target-relevant slice of the history and feed only that truncated slice to the ranker. This retrieval step is efficient but breaks end-to-end gradient flow and discards information.

The goal of this paper is to keep the full long history, model it end-to-end, and still run within the same latency and hardware budgets — by changing the attention architecture, the batching layout, and the training-length curriculum rather than by truncating the data.

Key Contributions

1. Stacked Target Cross Attention (STCA) — a single-query, target-to-history cross-attention architecture that omits history self-attention entirely, reducing per-layer cost from O(L²) (as in Transformer-style self-attention over the target-plus-history) to O(L) in history length L, with stacked layers providing higher-order target-conditioned fusion. Because there is exactly one query, the computation can be reordered (Eq. 11) to avoid materializing the length-L projected key/value tensors.

2. Request Level Batching (RLB) — a user-centric batching scheme that aggregates multiple targets from the same user/request so the user-side encoding is computed once and reused. The paper reports up to 8× reduction under request-level sharing, and states that RLB is an unbiased estimator of empirical risk because it only regroupies samples and does not change the loss definition. It can also extend to multi-request sharing for the same user/session.

3. Length-extrapolative training ("Train Sparsely, Infer Densely") — a regimen that trains on sequences averaging roughly 2k tokens while serving on sequences up to 10k, using Stochastic Length training with a U-shaped Beta distribution for length sampling and a temporal-suffix element-selection policy. This decouples training cost from deployment-time context length.

4. A demonstrated production deployment — one month of full-traffic A/B testing on Douyin and Douyin Lite, with the STCA+RLB+Extrapolation stack replacing TWIN(10k)-augmented retrieval features.

Main Findings

• Offline gains despite a conservative setup: In Table 1, all baselines are augmented with TWIN(10k) while the proposed method drops TWIN(10k) entirely. The proposed method still achieves the strongest results: +0.49 ΔAUC / -1.16 ΔNLL on finish, +0.71 / -1.14 on skip, and +0.39 / -1.41 on head (all in %). For comparison, HSTU reached +0.31 / -0.86 (finish), Transformer +0.25 / -0.46, DIN +0.19 / -0.17.

• Ablation contributions (Table 2, 4L complex STCA at 512 tokens): Adding a token-wise FFN on the history path and doubling STCA depth from 2L to 4L gave the largest single boost (+0.18%), followed by upgrading FFNs to SwiGLU (+0.11%), enlarging sparse ID embeddings from 128 to 320 (+0.08%), time-delta side information (+0.08%), query fusion (+0.06%), and increasing attention heads from 8 to 16 (+0.05%).

• Linear versus quadratic scaling (Figure 3): From L=500 to L=10k, STCA's sequence-side FLOPs rise from 1.06 to 21.06 GFLOPs (about 19.9×), while Transformer rises from 2.08 to 236.26 GFLOPs (about 113.6×). At a similar NLL (≈0.396), STCA runs at L=10k with 21.06 GFLOPs, whereas Transformer requires L=8k and 156.24 GFLOPs — roughly 7.4× higher.

• Bandwidth and throughput savings from RLB: Measured reduction in inter-module traffic is 77% at L=512 and 84% at L=2k (including all non-sequence features). RLB delivers a 2.2× end-to-end training throughput gain, rising to 5.1× with additional kernel optimizations, and raises the maximum trainable sequence length by roughly 8×. It also cuts Parameter Server CPU usage by 50% during training and data-to-training communication bandwidth by 50%.

• Training length must approach inference length (Table 3): Raising the maximum training length from 2k to 4k to 10k lifts finish AUC by +0.03%, +0.09%, and +0.21% respectively.

• Sequence sparsity trade-off (Table 4): Going from L_train_avg = 1.0k to 2.0k to 2.5k raises AUC lift from +0.09% to +0.21% to +0.22%, while sequence sparsity rises from 10.0% to 20.0% to 25.0%. The paper concludes SS ≈ 20% is the optimal balance, and notes this compares favorably with HSTU's SL setting of SS=57.6%.

• Temporal suffix selection matters: Retaining the most recent interactions achieves +0.21% AUC improvement, while random sampling yields no gain.

• Beta distribution shape (Table 5): A U-shaped distribution (α=0.02) achieves +0.21%, versus +0.11% for a decreasing distribution (α=0.5) and +0.08% for a skewed one (α=10).

• Efficiency-accuracy frontier: The approach reaches +0.23% offline AUC improvement at 10k inference, capturing roughly 80% of the full 10k training gain (+0.30%) at one-third computational cost. Online A/B confirmed +0.17% finish AUC.

• One-month online A/B results (Table 6): Across all users on Douyin, lifts were 0.1161% (30-day Activeness), 0.9266% (Stay Time), 3.3454% (Finish), 1.5678% (Comment), and 1.8282% (Like). On Douyin Lite, all-user lifts were 0.1281%, 0.8467%, 4.2275%, 2.6167%, and 2.3828% respectively. Gains were strongest for low- and medium-activity users — for example, low-activity Douyin Finish was 5.4987% and low-activity Douyin Lite Finish was 6.2808%.

• Inference cost accounting: Removing TWIN increases GPU cost by +33% but reduces CPU cost by -16%, for an estimated net total cost change of about +17%, which the authors consider acceptable given the online gains.

• Scaling behavior (Figure 1): The paper reports monotonic Finish AUC lifts as sequence length scales toward 10k and as sequence-module capacity grows across Simple (6M), Medium (23M), and Complex (133M) parameter settings, which the authors describe as mirroring scaling-law behavior observed in large language models.

Methodology in Plain English

The authors start from the observation that in ranking, the most useful signal is the direct relationship between the candidate video and each item in the user's history; relationships among historical items themselves matter less. Standard Transformers pay a quadratic cost to model those secondary relationships. STCA drops them entirely: the candidate video is the only query, and each layer attends over the full history.

Because there is only one query per layer, the attention can be computed without ever building the full projected key and value matrices for the history — the computation is reordered so nothing length-L-by-dimension is materialized, cutting both memory and FLOPs.

The second problem is redundancy. In real logs a single user request usually contains several candidate videos that all share the same history. Without special handling, that same long history is serialized, shipped to the GPU, and re-encoded once per candidate. RLB groups those candidates into a user micro-batch, encodes the history once, and reuses the result. The authors show mathematically that averaging the loss within the user group leaves the expected objective unchanged.

The third piece is the training/inference gap. Training on 10k-token sequences is expensive, so they train on sequences averaging about 2k tokens (sampled with a U-shaped distribution that mixes very short and occasionally long windows) while serving on up to 10k. They keep the most recent interactions rather than sampling randomly, and apply a batch-level load-balancing operator so variable-length sequences do not make step time be dictated by the longest sequence in the batch.

Evaluation proceeds in layers: a matched-compute offline comparison on Douyin data against DIN, Transformer, and HSTU (each given TWIN(10k) retrieval features as a handicap in their favor); ablations at 512 tokens; measured FLOPs, bandwidth, and throughput; extrapolation studies varying training length and distribution shape; and finally a one-month full-traffic A/B test on two products.

Why This Matters

Impact on research. The paper argues that recommendation, unlike NLP or CV, cannot scale by simply adding data — it is bounded by user-generated interactions — so the sequence dimension is the natural axis for scaling. It provides evidence of smooth, predictable gains with sequence length and capacity, and shows a concrete architecture that makes those gains affordable at billion-scale. It also challenges the dominant two-stage retrieval paradigm by showing that a full end-to-end encoder can replace a heavy retrieval block (TWIN(10k)) at comparable end-to-end cost.

Real-world applications:

• Short-video feeds (Douyin, TikTok and similar platforms) where user histories run to thousands of videos.
• E-commerce and shopping recommenders, where long purchase and browsing histories carry preference signal.
• News and content feeds with long reading histories and multiple candidate items per page view.
• Any industrial ranking stack where a retrieval or truncation stage is currently used purely for latency reasons rather than for modeling reasons.

Industry relevance. The paper is explicitly framed around production constraints — latency, bandwidth, CPU and GPU cost, parameter-server contention, and memory-bound training. The reported figures (77–84% bandwidth reduction, 2.2×–5.1× training throughput gains, 8× longer trainable sequences, 50% PS CPU reduction) are systems metrics as much as modeling metrics, and the one-month A/B test on two live products addresses the deployment question directly. The +17% net cost estimate with the gains on engagement metrics is presented as the practical trade-off argument.

Future Directions

• Multi-request sharing at scale. The paper states RLB "can be extended to share across multiple requests for the same user/session," but the reported experiments use request-level sharing; how far that extension goes in production is not reported.
• Pushing beyond 10k. The extrapolation ratio studied is fixed at 5 (2k training average, 10k inference). Whether the length-extrapolative regimen continues to hold at larger ratios or longer targets is an open question the paper does not answer.
• Closing the training-inference gap. Table 3 shows that matching the maximum training length to inference length improves results, and the method captures only about 80% of the full 10k training gain at one-third the cost — leaving room to recover the remaining accuracy.
• Generalization to other objectives and domains. The paper focuses on the finish rate for clarity and reports three offline objectives (finish, skip, head), but how the architecture and length-extrapolation interact with other ranking objectives, and with domains where histories are less temporally concentrated, is not reported.

Target Audience

This paper is most valuable to industrial recommender-systems engineers and applied researchers working on ranking stacks at scale — particularly those who own the sequence encoder, the feature-serving pipeline, or the training infrastructure and are weighing long-sequence modeling against latency budgets. It will also interest ML systems researchers focused on the compute/memory/communication trade-offs of attention over very long contexts, and academic researchers studying scaling laws outside NLP, since the paper frames sequence length as recommendation's scaling axis. Readers without background in attention mechanisms, distributed training, and CTR ranking will find the systems details difficult, though the three core ideas (single-query cross-attention, per-request batching, and sparse training with dense inference) are stated clearly enough to follow at a high level.

Authors’ abstract

Short-video recommenders such as Douyin must exploit extremely long user behavior histories without breaking latency or cost budgets. We present an end-to-end industrial recommender system that scales long-sequence recommendation modeling to 10K-length histories in production. First, we introduce Stacked Target-to-History Cross Attention (STCA), which replaces history self-attention with stacked cross-attention from the target to the history, reducing complexity from quadratic to linear in sequence length and enabling efficient end-to-end training over long user behavior sequences. Second, we propose Request Level Batching (RLB), a user-centric batching scheme that aggregates multiple targets for the same user/request to share the user-side encoding, substantially lowering sequence-related storage, communication, and compute without changing the learning objective. Third, we design a length-extrapolative training strategy -- train on shorter windows, infer on much longer ones -- so the model generalizes to 10K-scale histories without additional training cost. Across offline and online experiments, we observe predictable, monotonic gains as we scale history length and model capacity, mirroring the scaling law behavior observed in large language models. Deployed at full traffic on Douyin, our system delivers significant improvements on key engagement metrics while meeting production latency, demonstrating a practical path to scaling end-to-end ultra-long sequence recommendation to the 10K regime.