Delayed Feedback Modeling for Post-Click Gross Merchandise Volume Prediction: Benchmark, Insights and Approaches

AI summary

Overview

Research area: Online advertising and recommender systems — specifically delayed feedback modeling for post-click Gross Merchandise Volume (GMV) prediction in ad ranking.

Technical level: Intermediate. The paper assumes familiarity with ad ranking objectives (CTR, CVR, ECPM), attribution windows, and streaming/online model training, but defines its core concepts clearly.

Scope: The paper builds the first public benchmark for post-click GMV prediction with full transaction sequences, reports two empirical insights from that benchmark, and proposes a repurchase-aware dual-branch model with debiasing modules to handle delayed labels.

What This Paper Is About

Ad ranking models are shifting from predicting probabilities like conversion rate (CVR) toward predicting numerical business metrics like post-click GMV — the total sales value attributed to an ad click. Unlike CVR, where a click leads to at most one conversion, a single click can trigger multiple purchases, and a click's true GMV label only becomes fully known at the end of the attribution window (7 days in this work), so labels arrive incomplete and keep changing. The paper's goal is to define this problem, release data for studying it, and build a model that predicts GMV accurately despite the delayed and partially observed labels.

Key Contributions

1. Data resources. The authors release TRACE (TRansaction sequenCes bEnchmark), described as the first benchmark for post-click GMV prediction with delayed feedback modeling, containing the complete purchase sequence (timestamps and transaction prices) for each ad click that leads to conversions.
2. Valuable insights. Exploratory analysis and experiments on TRACE produce two findings: online streaming training substantially beats offline daily-updated training, and the label distributions of single-purchase versus repurchase samples differ significantly, motivating separate modeling.
3. Effective approaches. The authors propose READER (RepurchasE-Aware Dual-branch prEditctoR), a dual-branch architecture with a sample router and two debiasing modules — a label calibrator and a partial label unlearning strategy — trained under a two-stage regime.
4. Empirical validation. READER outperforms all baselines on TRACE across AUC, ACC, and ALPR, and ablation studies isolate the contribution of each debiasing component, architecture choice, and routing strategy.

Main Findings

• TRACE dataset statistics: 7.16m clicks, of which 3.84m (53.55%) are repurchase samples, covering 3.60m users and 1.96m items, sampled from 82 days of Taobao display advertising logs with a 7-day attribution window and 15 randomly sampled advertising scenarios.
• Online training beats offline training: In an exploratory comparison, the online-trained streaming model reached an AUC of 0.8165 versus 0.8055 for the daily-updated offline model, which the authors attribute to model freshness.
• Labels accumulate slowly: Only 40% of the final GMV comes from immediate purchases after a click; the cumulative observed GMV rises to over 60% after one day and then grows more steadily. The authors state that using partially observed GMV directly as a training target causes severe underestimation.
• Single-purchase and repurchase distributions differ: Repurchase samples show substantially higher average GMV and a more pronounced right-skewed tail. A two-sample Kolmogorov–Smirnov test yields a p-value of 0.00.
• Repurchase prediction is learnable: A lightweight MLP-based router achieved an AUC of over 80% on held-out test data.
• READER's headline results: AUC 0.8235, ACC 0.2612, ALPR 0.7523 — a relative AUC gain of +0.86%, ACC gain of +2.19%, and ALPR reduction of 6.88% over the strongest baseline. READER exceeds Oracle-Single (AUC 0.8168, ACC 0.2580, ALPR 0.7823) and comes closest to Oracle-Dual (AUC 0.8486, ACC 0.4134, ALPR 0.6273).
• Dual-branch structure helps, but needs debiasing online: In the online setting, Online-Dual (AUC 0.8161, ACC 0.2562, ALPR 0.8506) improved ACC over Online-Single (AUC 0.8165, ACC 0.2495, ALPR 0.8079) but degraded AUC and ALPR, which the authors say motivates the debiasing strategies.
• Each debiasing module contributes: Adding Calib alone gave AUC 0.8180, ACC 0.2568, ALPR 0.7738; adding GRA raised AUC by +0.0055 to 0.8235 with ACC 0.2604 and ALPR 0.7534; adding PLU yielded the best ACC (0.2612) and lowest ALPR (0.7523) with no loss in AUC.
• Shared-bottom dual-branch is the strongest architecture under the oracle setting: Shared-Bottom Dual-Branch (AUC 0.8520, ACC 0.4124, ALPR 0.6199) beat Frozen-Bottom (0.8404, 0.3902, 0.6593), Independent Dual-Branch (0.8348, 0.3772, 0.6819), and Single-Branch (0.8168, 0.2580, 0.7823).
• Hybrid routing beats hard routing: Hybrid Routing (AUC 0.8161, ACC 0.2562, ALPR 0.8506) outperformed Hard Routing (AUC 0.8066, ACC 0.2538, ALPR 0.8566) on all metrics.

Methodology in Plain English

The researchers first built a dataset. From Alibaba's Taobao display advertising logs, they sampled 15 advertising scenarios and applied stratified user sampling (higher rates for highly active users, lower for less active ones) to get a representative subset. For every ad click that led to a purchase, they kept the full list of purchases with their timestamps and prices, so the cumulative GMV can be replayed over time as a stream.

They then ran diagnostics. They plotted how the average GMV label changes hour by hour, compared an offline daily-updated model against an online streaming model, traced how much of the final GMV has been observed at each point after a click, and compared the GMV distributions of single-purchase versus repurchase clicks. They also tested whether a small MLP could predict, at the moment of the first purchase, whether the click would lead to more than one purchase.

Building on those findings, they designed READER. Input features pass through a shared encoder that produces a feature vector. Two separate MLP towers sit on top: one for single-purchase samples and one for repurchase samples. A completely separate router network (its own embedding layer and MLP, with a temperature parameter) outputs a probability that the click leads to multiple purchases. Routing has three zones: below 0.1 goes to the single-purchase tower, above 0.9 goes to the repurchase tower, and in between the two predictions are blended with a weighted average, which keeps the routing differentiable.

Because early online labels are incomplete, they added two debiasing pieces. A label calibrator predicts the gap between the log of the partial observed GMV and the log of the final GMV, using the features, elapsed time since the click, and the number of purchases seen so far; this corrected value is used as a pseudo-label. At the end of the attribution window, ground-truth alignment trains the model on the real final label, and partial label unlearning deliberately pushes the model away from the inflated last pseudo-label to counteract its bias.

Training happens in two stages: pretraining offline on complete labels from the first part of the data, then online learning on the live stream. The predictor and router are updated with a log-MAE loss, combined with the alignment and unlearning terms weighted by hyperparameters.

Why This Matters

Impact on research: The paper claims to be the first to formally define and study delayed feedback modeling for post-click GMV prediction, a problem where the target is continuous and a single click can yield multiple purchases. Prior GMV work such as TransSun covered only single-purchase scenarios. By releasing TRACE with full transaction sequences, the authors give the community a way to study streaming GMV models and delayed feedback methods that were previously only explored for CVR.

Real-world applications:

• Value-based bidding modes such as Max Conversion Value and Target ROAS, where the bid depends on predicted post-click GMV.
• ECPM scoring in ad auctions, where the score is formalized as pCTR × λ × pCVR × pGMV, making pGMV a direct multiplier on auction outcomes.
• Automated bidding and traffic allocation in e-commerce advertising systems that need real-time value estimates.
• Repurchase-heavy retail settings where treating all clicks as single-purchase leads to systematically wrong value estimates.

Industry relevance: The dataset comes from the display advertising system of Taobao, Alibaba, and the code and dataset are released at a public repository. The authors note that the price of an advertised item is often known, but simply substituting item price for predicted GMV is sub-optimal because of repurchases and varying transaction prices from product variants and coupons. The paper also caveats that the sampled dataset does not represent real-world business metrics or operational conditions.

Future Directions

• Develop more advanced routing strategies for the dual-branch predictor; the authors explicitly call this a promising direction for future improvement, since hybrid routing already outperformed hard routing.
• Extend the study of online delayed feedback modeling for GMV more broadly, which the authors frame as a newly opened avenue for research.
• Explore the trade-off between model freshness and label correctness further, since the paper shows only 40% of final GMV arrives immediately and the remainder accrues gradually across the attribution window.
• Investigate whether the label calibrator and partial label unlearning strategies generalize beyond this setting; the authors froze the calibrator during online learning because they found online updating of it yielded only marginal gains, leaving open whether better calibration updates are possible.

Target Audience

Researchers and practitioners working on ad ranking, conversion and value prediction, and delayed feedback modeling in industrial recommendation and advertising systems. It is also relevant to machine learning engineers who deploy streaming models with continuously updating labels, and to anyone needing a public benchmark for post-click GMV prediction. Readers without background in online advertising metrics (CTR, CVR, ECPM, attribution windows) will need to acquire that context first.

Authors’ abstract

The prediction objectives of online advertisement ranking models are evolving from probabilistic metrics like conversion rate (CVR) to numerical business metrics like post-click gross merchandise volume (GMV). Unlike the well-studied delayed feedback problem in CVR prediction, delayed feedback modeling for GMV prediction remains unexplored and poses greater challenges, as GMV is a continuous target, and a single click can lead to multiple purchases that cumulatively form the label. To bridge the research gap, we establish TRACE, a GMV prediction benchmark containing complete transaction sequences rising from each user click, which supports delayed feedback modeling in an online streaming manner. Our analysis and exploratory experiments on TRACE reveal two key insights: (1) the rapid evolution of the GMV label distribution necessitates modeling delayed feedback under online streaming training; (2) the label distribution of repurchase samples substantially differs from that of single-purchase samples, highlighting the need for separate modeling. Motivated by these findings, we propose RepurchasE-Aware Dual-branch prEdictoR (READER), a novel GMV modeling paradigm that selectively activates expert parameters according to repurchase predictions produced by a router. Moreover, READER dynamically calibrates the regression target to mitigate under-estimation caused by incomplete labels. Experimental results show that READER yields superior performance on TRACE over baselines, achieving a 2.19% improvement in terms of accuracy. We believe that our study will open up a new avenue for studying online delayed feedback modeling for GMV prediction, and our TRACE benchmark with the gathered insights will facilitate future research and application in this promising direction. Our code and dataset are available at https://github.com/alimama-tech/OnlineGMV .