Modeling Cascaded Delay Feedback for Online Net Conversion Rate Prediction: Benchmark, Insights and Solutions

AI summary

Overview

• Research area: Machine learning for recommender systems and online advertising — specifically delayed-feedback modeling for conversion rate prediction.
• Technical level: Intermediate to Advanced. The paper assumes familiarity with CVR prediction, delayed feedback, importance weighting, and survival/delay distribution models.
• Scope: The paper introduces the first open benchmark for online continuous Net Conversion Rate (NetCVR) prediction and proposes TESLA, an online training method that models the cascaded delays of click→conversion and conversion→refund.

What This Paper Is About

Standard conversion rate (CVR) models predict whether a user will buy after clicking, but they ignore refunds, so they overstate true user satisfaction. NetCVR — the probability that a click leads to a purchase that is not refunded — is a better business signal, but it is much harder to predict because it involves two chained delays (click to conversion, then conversion to refund) whose effects pull in opposite directions. This paper builds the first public dataset for this problem and a purpose-built online training method that handles both delays simultaneously.

Key Contributions

1. CASCADE dataset. The first large-scale, open, timestamped dataset for online continuous NetCVR prediction, derived from Taobao display advertising. It captures the full user journey (click → conversion → refund) with per-event timestamps, plus a reproducible streaming evaluation environment.
2. Three empirical insights. Data analysis establishes that NetCVR has strong hourly temporal patterns, that cascaded CVR/RFR modeling beats direct NetCVR modeling, and that delay time itself is a predictive signal correlated with both conversion and refund behavior.
3. TESLA method. The first online continuous training framework for NetCVR, combining a two-stage observation-window data stream, a shared-private CVR-RFR cascade, stage-wise importance-weighting debiasing, and a delay-aware ranking loss.
4. Benchmark results. TESLA outperforms eight delayed-feedback baselines plus offline and oracle references on CASCADE, with absolute gains of 12.41% RI-AUC and 14.94% RI-PRAUC on NetCVR over the strongest baseline.

Main Findings

• Online continuous training is necessary. Both CVR and NetCVR fluctuate hour to hour, with a sharp drop during late-night hours (00:00–04:00). Daily offline updates (e.g., the prior ECAD approach) cannot track this, and every online model in the benchmark beat the daily-updated BDL model.
• Cascaded beats direct. Decomposing NetCVR as CVR × (1 − RFR) and modeling the two stages separately yields higher AUC and PR-AUC than predicting NetCVR directly. Direct modeling cannot distinguish non-conversions from refunds, overfits conversion intent, and produces poorly calibrated probabilities (PCOC above 1).
• Delay time is a behavioral signal, not just a nuisance. High-CVR users convert faster with lower delay variance. High-RFR users also refund faster, suggesting immediate post-purchase dissatisfaction. Delay therefore reflects user confidence and satisfaction, and can be exploited in training.
• Hybrid parameter sharing wins. CVR and RFR share some information but retain distinct characteristics. A shared-private (hybrid) architecture outperforms both fully shared and fully separate variants; fully shared beats fully separate given the scarcity of refund data.
• Both an observation window and explicit debiasing are required. Adding a refund observation window alone improves AUC and lowers NLL by filtering early refunds, but combining it with importance-weighting debiasing gives the best results.
• Uncertainty-guided negative sampling matters. Low-uncertainty negatives (those with low predicted CVR) beat random negatives, which beat high-uncertainty negatives. Adding delay-based positive weighting on top gives the largest gain.
• Gains are larger on NetCVR than CVR. RI-AUC improves 12.41% on NetCVR versus 6.01% on CVR, indicating TESLA's advantage comes specifically from handling multi-stage cascaded delays.
• Window length must be tuned asymmetrically. The conversion observation window performs best at roughly 0.05 days, while the refund window is best kept at or below 0.01 days — most refunds happen quickly, so a long window only adds label noise.

Methodology in Plain English

The authors start from a simple decomposition: a net conversion happens when a purchase occurs and no refund follows, so NetCVR = CVR × (1 − RFR). Rather than predicting NetCVR in one shot, TESLA predicts the two factors separately in a cascaded structure.

Streaming data setup. Instead of waiting weeks for labels to fully resolve, TESLA maintains two sequential observation windows: a conversion window triggered after each click, and a refund window triggered after each conversion. Interactions are sorted into seven trajectory types (immediate/delayed/no conversion, each optionally followed by immediate/delayed refund), which lets the model distinguish transient from sustained conversions and update labels as new events arrive.

Model architecture. A shared-private network (inspired by Progressive Layered Extraction) gives CVR and RFR their own embedding layers while also learning a common representation, then feeds the combined features into two task heads. This limits interference between tasks while still transferring knowledge — important because refund data is sparse.

Two-stage debiasing. Delayed labels create censored positives: a user who will convert or refund eventually is temporarily mislabeled as negative. The authors apply importance weighting at each stage, using a pre-trained delay distribution model to estimate the probability that a true event falls beyond the observation window. Positive and negative samples get separate weights, and the same treatment is applied first to the click→conversion stage and then to the conversion→refund stage, yielding an asymptotically unbiased NetCVR estimate.

Delay-aware ranking loss. Point-wise losses struggle with label uncertainty. TESLA adds a pairwise ranking loss with two refinements: positive samples with shorter delay times get higher weight (via a sigmoid centered on the batch median delay), and negative samples are drawn preferentially from those with low predicted CVR, i.e., likely true negatives. The final objective sums the debiased CVR loss, debiased RFR loss, and the two ranking losses.

Why This Matters

Research impact. This is the first work to formulate and benchmark online continuous NetCVR prediction under cascaded delayed feedback. It moves the field beyond offline, daily-updated batch models and provides a shared dataset and evaluation protocol so future methods can be compared fairly. It also extends single-stage delayed feedback theory (importance weighting, observation windows) to a two-stage cascade where the two delays have opposite effects.

Real-world applications.

• E-commerce ad ranking and traffic allocation, where optimizing for gross conversions without accounting for refunds wastes spend on items users return.
• Marketplace seller reputation and item quality scoring, since NetCVR directly reflects whether purchases were retained.
• Return-policy and logistics planning, using predicted refund propensity to anticipate reverse-logistics demand.
• Subscription or high-ticket retail recommendation, where post-purchase regret is a major source of revenue leakage.

Industry relevance. The work is a collaboration between Xiamen University and Alibaba's Taobao & Tmall Group, and the data comes from production display advertising on one of the world's largest e-commerce platforms. It was accepted at WWW 2026, and code and dataset are released at github.com/alimama-tech/NetCVR, making it directly deployable by industrial recommender teams.

Future Directions

• Generalizing to longer cascades. The framework handles two stages; real user journeys can include cancellation, exchange, or partial refunds. Extending the data stream and debiasing to three or more cascaded stages is an open problem.
• Adaptive window selection. The optimal conversion and refund windows were found empirically and asymmetrically. Automatically tuning windows per user segment, category, or time of day could improve both label freshness and purity.
• Better delay distribution estimation. Both debiasing towers rely on a pre-trained delay distribution model fit to historical logs. Online adaptation of that model, or joint end-to-end training with the prediction towers, is unexplored.
• Broader deployment validation. Results are reported on a single platform's dataset. Verifying whether the cascade-beats-direct finding and delay-time signal hold on other domains (app installs, subscriptions, other e-commerce platforms) would strengthen the conclusions.

Target Audience

Recommender systems and computational advertising researchers and engineers, particularly those working on delayed feedback and continuous/online learning. It is also useful for applied ML practitioners at e-commerce or marketplace companies who need to model post-purchase behavior, and for graduate students looking for a well-scoped benchmark problem with a public dataset and baseline implementations. Readers without background in CVR prediction or importance-weighting debiasing will need to consult the cited prior work first.

Authors’ abstract

In industrial recommender systems, conversion rate (CVR) is widely used for traffic allocation, but it fails to fully reflect recommendation effectiveness because it ignores refund behavior. To better capture true user satisfaction and business value, net conversion rate (NetCVR), defined as the probability that a clicked item is purchased and not refunded, has been proposed.Unlike CVR, NetCVR prediction involves a more complex multi-stage cascaded delayed feedback process. The two cascaded delays from click to conversion and from conversion to refund have opposite effects, making traditional CVR modeling methods inapplicable. Moreover, the lack of open-source datasets and online continuous training schemes further hinders progress in this area.To address these challenges, we introduce CASCADE (Cascaded Sequences of Conversion and Delayed Refund), the first large-scale open dataset derived from the Taobao app for online continuous NetCVR prediction. Through an in-depth analysis of CASCADE, we identify three key insights: (1) NetCVR exhibits strong temporal dynamics, necessitating online continuous modeling; (2) cascaded modeling of CVR and refund rate outperforms direct NetCVR modeling; and (3) delay time, which correlates with both CVR and refund rate, is an important feature for NetCVR prediction.Based on these insights, we propose TESLA, a continuous NetCVR modeling framework featuring a CVR-refund-rate cascaded architecture, stage-wise debiasing, and a delay-time-aware ranking loss. Extensive experiments demonstrate that TESLA consistently outperforms state-of-the-art methods on CASCADE, achieving absolute improvements of 12.41 percent in RI-AUC and 14.94 percent in RI-PRAUC on NetCVR prediction. The code and dataset are publicly available at https://github.com/alimama-tech/NetCVR.