MAC: A Conversion Rate Prediction Benchmark Featuring Labels Under Multiple Attribution Mechanisms

AI summary

Overview

• Research area: Machine learning for online advertising — specifically conversion rate (CVR) prediction, dataset/benchmark construction, and multi-task learning (referred to here as multi-attribution learning, MAL).
• Technical level: Advanced. The paper assumes familiarity with CVR prediction, attribution mechanisms, multi-task learning architectures (Shared-Bottom, MMoE, PLE), and ranking metrics (AUC, GAUC).
• Scope: The paper releases the first public CVR dataset with conversion labels from four attribution mechanisms (MAC), an open-source baseline library (PyMAL), a systematic empirical analysis of multi-attribution learning, and a new model (MoAE) designed from the resulting insights.

What This Paper Is About

CVR prediction models in advertising learn from conversion labels, and those labels are produced by an "attribution mechanism" — a rule that decides which ad clicks get credit for a conversion. Prior public datasets (Criteo, Ali-CCP, Taobao) only supply labels from one such rule, so models only ever see a partial view of why users convert. This paper builds a public benchmark where every ad click carries labels from four different attribution mechanisms, then uses it to study whether learning from all of them at once actually helps, and under what conditions.

Key Contributions

1. A new data resource — MAC. The authors release what they describe as the first public CVR benchmark with multi-attribution labels. Each ad click is annotated with attribution weights under four mechanisms: last-click, first-click, linear, and data-driven attribution (DDA). The dataset contains 79M clicks, 15.1M items, and 0.8M users, and adds behavior sequence features and multimodal features that the compared public datasets lack.
2. An open-source toolkit and systematic benchmarking — PyMAL. A PyTorch-based library implementing baseline families (BASE single-attribution, Shared-Bottom, MMoE, PLE, HoME multi-task models, and the state-of-the-art NATAL model), plus a thorough evaluation on MAC.
3. Three empirical insights about MAL. The analysis establishes that multi-attribution learning generalizes across attribution settings, that auxiliary objective choice matters (and can backfire), and that two architectural principles drive success.
4. A new model — MoAE (Mixture of Asymmetric Experts). The proposed model combines a mixture-of-experts backbone with main-task-prioritized asymmetric knowledge transfer, and beats all baselines on MAC.

Main Findings

• MAL generalizes across attribution mechanisms. Against the single-attribution BASE model, the strongest MAL models improve GAUC under all four target settings. The reported deltas for MoAE versus BASE are +2.12pt (last-click), +0.34pt (first-click), +1.74pt (DDA), and +0.80pt (linear).
• Gains are largest for last-click and DDA. Those settings see the most significant GAUC increases, at 2.12pt and 1.74pt respectively. Linear attribution shows a smaller 0.80pt GAUC uplift, which the authors attribute to MAL not increasing the number of positive samples in that setting.
• First-click labels are noisiest. The first-click setting shows the smallest GAUC improvement at 0.34pt and the lowest absolute AUC and GAUC scores; the authors tie this to the long time gap between initial click and conversion, which introduces unknown confounding factors.
• Users with long conversion paths benefit most. Grouping users by conversion path complexity, the authors find MAL delivers greater AUC improvements for users with more complex conversion paths across the last-click, linear, and DDA settings. Under first-click as the target, the improvement shows no clear monotonic trend with path complexity.
• More auxiliary targets usually help — but not always. Performance generally scales with the number of auxiliary targets, and the Cartesian product auxiliary target (CAT) adds further gains. However, when first-click is the target, only the last-click auxiliary target helps; DDA and linear auxiliary targets hurt performance, and combining all three also hurts.
• Two architectural principles matter. First, fully learning multi-attribution knowledge — achieved by mixture-of-experts structures in PLE and HoME, which improve both primary and auxiliary tasks. Second, leveraging that knowledge in a main-task-prioritized way — achieved by NATAL's asymmetric transfer, which wins on the primary task but underperforms PLE and HoME on auxiliary tasks. MoAE satisfies both.
• MoAE beats the state of the art. Versus NATAL, MoAE gains +0.23pt GAUC (last-click), +0.13pt (first-click), +0.27pt (DDA), and +0.39pt (linear). Under the last-click target, MoAE reaches 0.7636 GAUC versus NATAL's 0.7613, while also improving auxiliary-task metrics (for example 0.6877 on first-click auxiliary versus NATAL's 0.6842).
• Gains come from labels, not extra parameters. Disentangling parameter scaling from multi-attribution supervision by zeroing auxiliary task weights, the "w/o mal" variants — which have the same parameter scale as their MAL counterparts — perform near the BASE model. The authors report that vanilla parameter scaling yields no or marginal GAUC improvements, while the full MAL models gain substantially (MoAE +2.12pt GAUC).

Methodology in Plain English

The authors start from a practical data gap: to study multi-attribution learning you need labels from multiple attribution mechanisms, and no public dataset had them. So they sample 21 consecutive days of traffic from Taobao's advertising system, using stratified sampling that gives higher sampling rates to highly active users and lower rates to less active ones. For each ad click, they compute attribution weights under four rules — last-click (all credit to the final click), first-click (all credit to the initial click), linear (credit spread uniformly across touchpoints), and DDA (learned via the CausalMTA model) — using direct conversion attribution within a fixed, confidential attribution window. Data is anonymized by irreversibly hashing identifiers.

To evaluate models, they convert prediction into binary classification: clicks with positive attribution weight count as positive samples, zero-weight clicks as negatives. Performance is reported with AUC and Group AUC (GAUC), where GAUC weights each user's AUC by that user's click count.

They then build PyMAL to implement a spectrum of baselines and train everything with the Adam optimizer for one epoch, using the last day as the test set and preceding days for training, a batch size of 4096, a learning-rate search over {0.1x, x, 10x | x ∈ {0.0030, 0.0035, 0.0040}}, and an auxiliary loss weight λ searched over {0.1, 0.2, 0.3, 0.4}. Auxiliary objective combinations are chosen via a greedy forward-selection search. They also test the automatic weighting method GCS but find it does not contribute significantly.

From observed strengths and weaknesses — MoE models do well on all tasks, NATAL does best on the main task but worse on auxiliaries — they derive two design principles and build MoAE to satisfy both: an MoE backbone with a shared expert plus attribution-specific experts, a main-task-centric feature transfer module that asymmetrically aggregates auxiliary knowledge into the main predictor, and per-attribution MLP predictors.

Why This Matters

Impact on research. The paper's central claim is that it removes a data bottleneck. Because every prior public CVR dataset used a single attribution mechanism, no one could systematically compare multi-attribution approaches on public data. MAC makes that possible, and PyMAL makes the comparisons reproducible. The insights about auxiliary objective selection — especially the finding that adding auxiliary tasks can hurt under first-click — give a concrete design guideline rather than a generic "more tasks are better" assumption.

Real-world applications:

• Ad ranking and bidding systems that need to decide which clicks to bid on and how to allocate traffic; better CVR estimates feed directly into these decisions.
• Attribution budget allocation — advertisers and platforms choosing between last-click, first-click, linear, or data-driven credit rules can now see how much model accuracy depends on that choice.
• Multi-task recommendation systems — the two design principles (learn multi-attribution knowledge fully, then prioritize the main task) transfer to other settings where several related labels coexist.
• Cold-start / long-journey users — the finding that MAL helps most for users with long conversion paths points to where the approach should be deployed.

Industry relevance. The dataset comes from Taobao's advertising system at Alibaba, and GAUC is described as the primary evaluation criterion in the authors' production system, chosen because it aligns more closely with online performance than AUC. The authors note that improvements in the system-optimized attribution mechanism can drive user satisfaction and platform revenue. The dataset release is also deliberately anonymized and excludes real-time operational metrics.

Future Directions

• Design better auxiliary objectives. The paper shows that naive addition of auxiliary targets fails under first-click; the authors explicitly hint at the potential of "more sophisticated auxiliary targets," implying that automated or learned selection of auxiliary attribution views is open.
• Expand beyond four attribution mechanisms. MAC covers last-click, first-click, linear, and DDA. Whether the findings hold for other attribution rules, or for DDA models other than CausalMTA, is untested.
• Explain the auxiliary-task learning result. The paper poses RQ6 — how auxiliary-task learning (ATL) techniques affect MAL models — but the supplied content cuts off mid-discussion, so this remains a stated open question.
• Generalize the architecture principles. Whether the "MoE backbone plus asymmetric main-task transfer" recipe transfers to other multi-task domains outside advertising CVR is not established here.

Target Audience

Researchers and practitioners working on conversion rate prediction, online advertising, computational advertising, and multi-task recommendation systems. It is most useful to those who build or benchmark CVR models and want a public dataset with richer supervision signals, and to multi-task learning researchers interested in how auxiliary objective selection and mixture-of-experts structure interact. Benchmark designers and reproducibility-focused engineers will also find the PyMAL library and evaluation protocol directly actionable. The paper is not beginner-friendly: it assumes fluency in CVR prediction, attribution mechanisms, MoE/MTL architectures, and ranking metrics.

Authors’ abstract

Multi-attribution learning (MAL), which enhances model performance by learning from conversion labels yielded by multiple attribution mechanisms, has emerged as a promising learning paradigm for conversion rate (CVR) prediction. However, the conversion labels in public CVR datasets are generated by a single attribution mechanism, hindering the development of MAL approaches. To address this data gap, we establish the Multi-Attribution Benchmark (MAC), the first public CVR dataset featuring labels from multiple attribution mechanisms. Besides, to promote reproducible research on MAL, we develop PyMAL, an open-source library covering a wide array of baseline methods. We conduct comprehensive experimental analyses on MAC and reveal three key insights: (1) MAL brings consistent performance gains across different attribution settings, especially for users featuring long conversion paths. (2) The performance growth scales up with objective complexity in most settings; however, when predicting first-click conversion targets, simply adding auxiliary objectives is counterproductive, underscoring the necessity of careful selection of auxiliary objectives. (3) Two architectural design principles are paramount: first, to fully learn the multi-attribution knowledge, and second, to fully leverage this knowledge to serve the main task. Motivated by these findings, we propose Mixture of Asymmetric Experts (MoAE), an effective MAL approach incorporating multi-attribution knowledge learning and main task-centric knowledge utilization. Experiments on MAC show that MoAE substantially surpasses the existing state-of-the-art MAL method. We believe that our benchmark and insights will foster future research in the MAL field. Our MAC benchmark and the PyMAL algorithm library are publicly available at https://github.com/alimama-tech/PyMAL.