Dual-Stream LLM-Guided Reward Design for Reject-Inference-Aware Reinforcement Learning in E-Commerce Payment Fraud Detection

AI summary

Overview

Research area: Machine learning for financial fraud detection — specifically LLM-guided reinforcement learning (RL) reward design applied to e-commerce payment transaction blocking under censored (reject-inference) labels.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (REINFORCE), gradient-boosted models, constrained MDPs, and the reject-inference / positive-unlabeled learning literature.

Scope: A single study showing that one LLM can evolve a two-stream reward function — one stream for labeled approved transactions, one for legacy-blocked transactions whose fraud outcome is unobservable — for a two-step fraud-blocking policy, evaluated on a four-month held-out out-of-time window from a large e-commerce payment system.

What This Paper Is About

Production fraud models only ever learn from transactions their own legacy system allowed through, because a blocked transaction never produces a chargeback label. The model therefore trains on a narrower, lower-risk slice of traffic than it actually faces in deployment — the classical reject-inference problem.

The paper's goal is to attack that problem inside the reward function: extend the LLM-guided reward-design framework of Qu et al. (2025) so that a single LLM writes a reward with two internal streams, one scoring labeled approved rows and one scoring blocked, unlabeled rows using the newest gradient-boosted risk scores versus the legacy scores that caused the original block. The LLM decides for itself which of 14 offered signals to trust.

Key Contributions

1. Reject-inference-aware reward design. To the authors' knowledge, the first LLM-guided reward-design loop that learns from censored, never-approved samples; prior LLM reward work targets robotics and games where every rollout yields a reward.

2. A dual-stream reward designed semi-zero-shot. Stream 1 keeps a dollar-weighted Lagrangian on labeled rows, seeded from the human baseline of Qu et al. (2025). Stream 2 is authored zero-shot, because no human-designed reward exists for the blocked population. Both streams share one REINFORCE gradient and one policy.

3. Signal selection left to the LLM and auditable afterwards. A superset of 14 candidate evidence signals is exposed; the selected reward references four and relies on two, ignoring signals (including the legacy scores the authors' own schema assumed it would need). Because the reward is emitted as code, that choice is inspectable rather than buried in fitted weights.

4. Deployment-realistic setup and held-out temporal evidence. The reward designer is an open Qwen3-class ~27B-parameter model in FP8 on a single A100 (80 GB); the resulting policy is architecturally identical to the labeled-only baseline and adds no inference cost. Evaluation reports a four-month out-of-time window month by month at each target recall.

Main Findings

• Dual-stream beats the labeled-only agent at every target recall. On the four-month held-out window, dollar-weighted precision rises over the single-stream agent by +4.74, +2.36 and +2.97 points at 80%, 85% and 90% recall (+3.36 on average). Absolute dual-stream precision is 55.15%, 46.36% and 35.92% at those targets.

• Dual-stream beats the production GBM baseline at every target. Baseline precision of 47.09%, 39.95% and 29.91% becomes 55.15%, 46.36% and 35.92% — gains of +8.06, +6.41 and +6.01 points, +6.83 on average. Pooled averages: Base 38.98%, SS 42.45%, DS 45.81%.

• Where the blocking happens differs sharply. The paper's objective also asks that more fraudulent GMV be stopped pre-authorization than post-authorization. DS intercepts 67.1%, 94.8% and 64.8% of blocked fraud GMV at step 0 across the three targets, satisfying the constraint 3/3; SS achieves 64.8%, 81.6% and only 12.5% (2/3), pushing most fraud past the issuer authorization request at the 90% target.

• The advantage concentrates at the tight end of the recall band. At the 80% target DS leads SS in all four months (+2.80, +1.00, +1.91, +4.77). At 90% the ordering reverses and SS leads in three of four (−2.08, −1.36, −7.23, +3.81), which the authors trace to SS buying loose-recall precision by deferring blocking. Against the production baseline DS is positive every month at 85% and 90%, and in three of four at 80% (falling 0.62 points below in July); SS clears the baseline only once at 80%, falling 5.39 points below in July.

• Absolute levels shift sharply month to month, so only within-row comparisons are meaningful — the baseline runs 61.27% in April versus 19.27% in May at the 80% target.

• The LLM discarded the signal structure the authors sketched. It normalizes the legacy scores and then never mixes them into confidence, promoting the newest calibrated head to sole confidence term and documenting the omission as avoiding "stale-model noise." Payment instrument and the three secondary heads of each generation are never referenced; buyer segment is kept, conditioning block thresholds (0.70 for low-risk cohorts, 0.45 for the highest-risk one) and reward weights.

• Two unprompted reward behaviors emerged. The reward hinge-gates — paying for blocking only above threshold and passing only below, with symmetric crossover penalties — which its docstring records as a fix for false positives from a linear margin. It also applies a 1.8× step-0 multiplier plus a fixed early-block bonus, echoing early-stage up-weighting independently derived by Qu et al. (2025).

• Results stabilized across three independent training trials per agent, with exact operating points varying trial to trial. Both agents ran the full N_iter = 40 budget with no early stopping (best candidates at iterations SS 34, DS 25).

Methodology in Plain English

Transactions pass two checkpoints: pre-authorization, where the agent sees every transaction, and post-authorization, where it only sees ones it already passed. A compact neural policy — categorical embeddings plus the newest gradient-boosted scores feeding a two-class softmax — decides pass or block at each step, trained with REINFORCE at learning rate 10⁻⁴.

The training set is the union of two populations: ~2.06M labeled approved transactions with an observed fraud label, and ~3.91M legacy-blocked transactions (~1.9× the labeled stream, ~5.97M total) whose label is set to −1 because it can never be observed.

An LLM sits outside this loop and writes the reward function as Python code. On each of 40 outer iterations it samples 10 candidate get_reward functions, AST-validates each against a required 19-argument signature, and resamples if invalid; each valid candidate trains a fresh policy for 200 episodes; the best candidate by dollar-weighted precision at target recall is kept and its feedback (best / suboptimal / reflection) conditions the next iteration. LLM temperature adapts, decaying ×0.9 on improvement and growing ×1.1 on plateau. A small table of dollar-weighted fraud rates by buyer cohort and payment method is injected so the LLM can calibrate context weights rather than guess.

The key architectural rule: legacy scores enter the reward only, never the policy's input state. The policy keeps its original feature contract; the legacy scores act as reward-time evidence about a blocked row's latent risk. Because stream 2 is gated entirely by the is_blocked flag, it never touches approved traffic, so the deployed policy is unchanged in size and latency. Training uses February 2026 data, LLM candidate selection uses March 2026, and the out-of-time test is April 1–July 31, 2026, with the unauthorized-chargeback label matured as of 2026-09-01. The metric is dollar-weighted precision at fixed dollar-weighted recall against the production baseline score, with operating points obtained by sweeping block-score thresholds across both stages.

Why This Matters

Impact on research. The paper moves LLM-guided reward design out of robotics and game benchmarks — where every rollout yields a reward — into a setting where labels are missing by construction. It also offers an alternative to the explicit machinery of classical reject inference (Heckman correction, self-labeling, PU estimators): the disagreement structure between newest and legacy scores is already in the data, and the LLM synthesizes a reward from it. Because the reward is code, the resulting design decisions are auditable rather than opaque, inverting RLHF/RLAIF — the LLM designs a reward for an external policy and is never updated itself.

Real-world applications:

• E-commerce and card-not-present payment fraud, where blocking decisions censor the very labels needed to retrain the blocker.
• Credit scoring and lending, the original home of reject inference, where declined applicants never reveal their repayment outcome.
• Trust-and-safety and account-abuse systems that act on traffic before outcomes can be observed.
• Any cost-sensitive decisioning pipeline where the objective is dollar-weighted precision at a target recall and policy latency budgets are tight.

Industry relevance. All four authors are at eBay, and the setup is deliberately production-shaped: real transaction data, a production GBM baseline, a single A100 80 GB running an open-weight ~27B FP8 model (a simplification over the 2× V100 32 GB, 4-bit setup of prior work), no added inference cost, and the claimed sub-50 ms CPU inference of the underlying policy left unchanged. The censored population carries usable risk signal; recovering it nearly triples the usable training data.

Future Directions

• Guard against propagating legacy bias. The blocked population is a biased, non-random (MNAR) sample; stream 2 assumes the newest GBM scores carry genuine risk signal there. Where the original block was driven by features the newest models do not observe, that assumption can import legacy-policy bias — the authors explicitly caution against treating stream 2 rewards as ground-truth labels. Quantifying and correcting this is open.

• Test whether the discarded signals were genuinely uninformative. The reward ignored the legacy generations' secondary heads and the payment instrument; the authors draw no conclusion that these are useless, only that exposing a superset and letting the designer select is a workable way to find out. Re-running with altered supersets would test this.

• Close the loose-recall gap. The dual-stream advantage holds at the tight end of the band but reverses at 90% recall, where the labeled-only agent leads in three of four months by deferring blocking to post-authorization. Whether a reward can hold both tight-recall precision and early interception across the whole band is unresolved.

• Deployment safeguards. The ethics statement calls for human review, redress paths, and ongoing monitoring for differential impact across buyer cohorts before any deployment, given that false blocks harm legitimate buyers and missed fraud harms sellers and the platform.

Target Audience

Practitioners and researchers in applied fraud detection and risk modeling at payments companies, marketplaces and banks; RL researchers interested in reward design for censored or missing-label environments; and anyone working on LLM-driven reward engineering who wants evidence from a production-adjacent domain rather than a simulation benchmark. Readers without RL or credit-scoring background will find the methodology dense, but the results tables and the code listing in the appendix are readable on their own.

Authors’ abstract

Production fraud-detection models learn from a censored slice of their own traffic: transactions a legacy system blocked never receive fraud labels, so models train on a biased, approved-only distribution. We extend LLM-guided reward design, the framework of Qu et al. (2025) in which an LLM evolves the reward of a two-step REINFORCE fraud-blocking policy, to attack this reject-inference problem inside the reward. A single LLM designs a dual-stream reward: stream 1 applies…