Multi-Agent Reinforcement Learning from Delayed Marketplace Feedback for Objective-Weight Adaptation in Three-Sided Dispatch

AI summary

Overview

Research area: Reinforcement learning for large-scale marketplace logistics — specifically offline multi-agent RL applied to dispatch optimization in a three-sided (customer, merchant, courier) food-delivery marketplace.

Technical level: Advanced. The paper combines a decentralized multi-agent MDP formulation, offline Double DQN with a conservative regularizer, and a production switchback experiment, so it assumes familiarity with value-based RL and causal inference for online experiments.

Scope: A deployed case study from DoorDash (authors Haochen Wu, Yi Hou, and Shiguang Xie, the latter listed in the paper body as Ryan Xie) in which a store-level RL policy adapts the objective weights of an existing dispatch optimizer using delayed marketplace outcomes, evaluated in a two-week production switchback experiment.

What This Paper Is About

Food-delivery dispatch must constantly trade off delivery speed against batching efficiency, and in production that tradeoff is usually governed by static heuristic weights tuned globally and updated manually — making them brittle when local conditions change. This paper asks whether an RL policy can learn, from delayed real-world marketplace feedback, to pick a store-level multiplier that shifts an existing assignment optimizer's speed-versus-batching tradeoff, without replacing the optimizer or violating its feasibility constraints. The goal is safe, local, real-time adaptation of dispatch objectives at production scale.

Key Contributions

1. An RL architecture that learns from marketplace feedback while adapting dispatch objective weights through a low-dimensional control interface, rather than replacing the production assignment optimizer.
2. A formulation of objective-weight adaptation as an offline multi-agent decision-making problem, with store-level decentralized execution and delayed regional rewards drawn from the real-world marketplace.
3. A production switchback experiment showing increased batching and reduced courier-side time costs without degrading customer-facing delivery quality.
4. A full description of the offline training pipeline — controlled data collection, a shared value function with Double Q-learning targets plus a conservative regularizer, and reward-reweighting and drift-monitoring diagnostics used before and after deployment.

Main Findings

• Batching increased, significantly, across all day parts: the OWA-RL policy raised the share of batched deliveries from a 47.52% baseline to 48.14%, an average treatment effect of +0.495 percentage points (p < 0.001).
• Courier-side time costs fell: CAT (delivery duration measured from courier acceptance to dropoff) dropped from 1163.0 to 1159.8 seconds, an ATE of −1.261 seconds (p = 0.019), and CWT (courier wait time) dropped from 277.1 to 275.7 seconds, an ATE of −0.856 seconds (p = 0.004).
• Customer-facing delivery quality was preserved: ASAP (order creation to dropoff) was 1956.0 seconds at baseline versus 1960.0 under OWA-RL, an ATE of +0.972 seconds that was not statistically significant (p = 0.264), and the 20-minute lateness rate was unchanged at 2.09% in both arms (ATE −0.012, p = 0.237).
• Dinner-hour results were similar and slightly stronger on lateness: CAT fell from 1156.3 to 1153.1 seconds (ATE −1.289, p = 0.042), CWT from 262.9 to 261.6 seconds (ATE −1.030, p = 0.041), batching rose from 57.99% to 58.59% (ATE +0.600, p = 0.010), and the 20-minute lateness rate improved from 2.36% to 2.34% (ATE −0.037, p = 0.040). ASAP was not significantly changed (2168.3 versus 2173.0 seconds, ATE +0.869, p = 0.633).
• The policy is state-dependent, not a fixed global retune: inspecting predicted actions for the San Francisco Bay Area during Friday dinner peak showed probability mass shifting across ASAP-weight multipliers as outstanding deliveries, supply pressure, and courier wait time changed, with the lower multiplier becoming more likely in specific backlog and supply regimes.
• The policy responds directionally to reward design: offline reward-reweighting diagnostics showed that increasing the efficiency weight shifted probability mass toward lower ASAP-weight multipliers, while increasing the speed weight shifted mass toward higher multipliers.
• Conservative regularization trades offline fit for stability: the DQN baseline without the CQL penalty reached lower MSE faster, while OWA-RL maintained a higher training loss because the conservative objective penalizes high values on actions weakly supported by logged data.

Methodology in Plain English

The system has two nested layers. The inner layer is the existing production assignment optimizer, which turns orders, couriers, constraints, and objective weights into courier-order assignments. The outer layer is the learned policy, called OWA-RL, which does not make assignments itself — it only picks a multiplier that changes the optimizer's tradeoff.

Each store is treated as an agent. Every assignment cycle, the store observes three numbers: the number of outstanding deliveries, a localized supply-pressure feature (a regional supply signal rescaled by the store's own recent median count of feasible couriers relative to the regional median), and the median courier wait time. From those three numbers the policy chooses one of five discrete multipliers — 0.8, 0.9, 1.0, 1.1, or 1.2 — applied to the baseline delivery-speed (ASAP) objective weight. Lower multipliers make batching and more efficient routing more attractive to the optimizer; higher multipliers favor faster order completion. A multiplier of 1.0 exactly recovers the static production baseline.

Rewards come from delayed marketplace outcomes. For each delivery, ASAP is dropoff time minus order creation time, CAT is dropoff minus courier acceptance, and XCAT is CAT minus the direct pickup-to-dropoff travel time the delivery would have taken alone. The reward for a decision cycle is the negative average of α·ASAP + β·XCAT over the deliveries attributed to that cycle in the region, so rewards are aggregated regionally to capture network effects across nearby stores and couriers. The paper reports using α = 0.9 and does not report a value for β.

Training is offline and centralized. Logged online decisions are joined with delayed fulfillment outcomes into transition tuples (state, action, regional reward, next state), and a shared store-level Q-network is trained with a Double DQN target plus a discrete Conservative Q-Learning penalty, with the combined loss being the DDQN loss plus η times the CQL loss (the value of η is not reported). The network is a two-layer MLP with hidden dimension 16, mapping a 3-dimensional state to 5 action scores; at serving time the highest-scoring action is selected. Training ran for 30 epochs with mini-batches of 32, the Adam optimizer at a learning rate of 10⁻³, discount factor γ = 0.99, gradient clipping at maximum norm 10, and a target network refreshed every 2 epochs.

Data collection used a controlled regional rollout over two iterations of roughly two days each. Only 0.5% of global regions were randomly selected every two hours. The first iteration used pure exploration with uniformly random actions; the second used a 50% exploitation / 50% exploration mixture. The paper estimates roughly 4,320 dispatch runs per day, about 10 stores per run, and about 864,000 data points per day under the 0.5% rollout, yielding approximately 1.73 million transition samples per iteration and roughly 3.46 million across both.

Evaluation was a global switchback experiment over approximately 4,000 geographic regions randomly assigned to treatment or control at each two-hour interval, with about half of regions in each condition per interval over a two-week period. Effects were estimated with CUPED variance reduction and p-values clustered at the region-hour switchback bucket, with customer-experience quality, cancellations, and carryover effects monitored as guardrails.

Why This Matters

Impact on research. Most RL work on delivery operations learns direct operational decisions and is evaluated in simulation or offline. This paper studies a complementary setting: learning from genuinely delayed, noisy, and coupled feedback from a live economic and logistics system, while only modulating the objective of an existing optimizer. It offers production-scale evidence that offline RL with conservative value estimation can be deployed safely in a multi-agent marketplace, and it frames the objective-weight interface as a way to preserve feasibility constraints and operational safeguards that a decision-replacing policy would put at risk.

Real-world applications:

• Food-delivery and quick-commerce dispatch, where the same speed-versus-batching tradeoff recurs across every order cycle.
• Ride-hailing and courier marketplaces, where matching objectives must balance rider wait, driver utilization, and pooling.
• Freight and last-mile logistics routing, where consolidating shipments improves efficiency but can hurt service-time commitments.
• Any multi-sided platform with an existing combinatorial optimizer, where a small learned control layer can adapt weights rather than requiring a rebuilt decision system.

Industry relevance. The system serves hundreds of millions of daily inferences at a 20-second cadence, and the deployed change amounts to shifting a single multiplier on an existing objective weight — an intervention that requires no change to the optimizer, constraints, or serving infrastructure. That combination of minimal surface area and measurable operational gains is what makes the design pattern transferable to other production decision systems.

Future Directions

• Extending beyond a single weight interface. The authors propose introducing different decision layers to the dispatch system, which would require systematic methods for detecting distribution shift and understanding how RL decisions interact across agents in the multi-agent setting.
• Improving credit assignment. Because rewards are attributed from delayed regional outcomes, individual store-level actions receive noisy credit; reducing this noise while retaining the network-effect signal is an open problem.
• Maintaining reliability under marketplace drift. The offline-trained policy's reliability depends heavily on the stability of marketplace dynamics after deployment, which motivates continued drift monitoring of state, action, and reward distributions.
• LLM-based interpretability tooling. The authors suggest tools built on large language models that link RL policy decisions to dynamic states, to support automated hypothesis generation, debugging, and policy retraining.

Target Audience

Applied RL researchers and ML engineers working on reinforcement learning from real-world, delayed feedback; industrial practitioners building decision systems for marketplaces, logistics, and dispatch; and experimentation scientists interested in how offline-trained policies are validated through switchback experiments. Readers without an RL background can still follow the system design, the interface, and the production results, but the training formulation and conservative-regularization discussion require intermediate-to-advanced familiarity with value-based offline RL.

Authors’ abstract

Dispatch in three-sided marketplaces provides a natural setting for reinforcement learning from world feedback: decisions are evaluated by delayed operational outcomes such as delivery speed, courier utilization, and merchant congestion. We present a deployed reinforcement learning system at DoorDash that adapts dispatch objective weights in a large-scale food-delivery marketplace using delayed signals. Rather than replacing the combinatorial assignment optimizer, a store-level policy learned from logged marketplace data selects a discrete multiplier that shifts the dispatch optimizer's tradeoff between delivery quality and batching efficiency. This interface enables offline policy learning under noisy, delayed, and coupled feedback while preserving production feasibility constraints and operational safeguards. We train a shared value function using centralized offline data and decentralized store-level execution, with Double Q-learning targets and a conservative regularizer to reduce out-of-distribution value overestimation. In a production switchback experiment, the offline-trained policy increases batching and reduces courier-side time costs without degrading customer-facing delivery quality. Results illustrate how world feedback from a live economic and logistics system can be used to safely adapt decision policies online.