Cooperative Multi-Agent Reinforcement Learning for Market Making on Prediction-Market Order Books

AI summary

Overview

Research area: Multi-agent reinforcement learning applied to market making on prediction-market limit order books (Polymarket binary contracts), sitting at the intersection of market microstructure, stochastic control, and cooperative MARL.

Technical level: Advanced. The paper assumes fluency in POMDPs, policy-gradient RL, limit order book mechanics, and adverse-selection/inventory-risk theory.

Scope: An MSc research thesis (LSE, Department of Statistics, supervised by Tengyao Wang) that builds a data pipeline from Polymarket L2 updates into LOBSTER-format order data, then trains and evaluates single-agent and cooperative two-agent market makers in the JaxMARL-HFT simulator on 417 real markets.

What This Paper Is About

Market makers earn the bid-ask spread but carry the risk of the inventory they accumulate when only one side of a quote fills. On prediction markets like Polymarket, order books are thin and contracts are binary (paying exactly 0 or 1 at a known resolution date), so adverse selection and inventory risk are especially acute. The paper asks whether a market-making agent trained on replayed real Polymarket order flow can be made to quote more and lose less, and specifically whether splitting the market-making "seat" into a cooperating quoter and risk manager, with a deliberately designed reward split, improves performance over the JaxMARL-HFT baseline.

Key Contributions

1. An order-level data pipeline for prediction markets. Polymarket publishes only L2 book updates, not the individual order messages a faithful replay simulator needs. The authors reconstruct individual orders from those updates and write them in LOBSTER format, enabling replay-based training on 417 real markets.

2. A redesigned single-agent market maker. Relative to the JaxMARL-HFT baseline, the observation space is enriched with five book levels per side plus recent flow (order imbalance, relative-strength measure, volatility of step-to-step mid changes, high-low bar statistics, drift, signed volume shares) instead of just spread and own inventory; the action table widens from nine two-sided rows to seventeen rows reaching thirteen ticks (versus five) and admitting skewed and one-sided quotes plus a no-trade row; and inventory control moves out of the reward into a position cap and a loss stop.

3. A cooperative multi-agent decomposition of the market-making seat. One agent (the quoter) quotes continuously; a second (the risk manager) manages the quoter's inventory. They are independent learners acting simultaneously as peers, each given augmented observations of the partner's position (the quoter sees what the risk manager closed last step; the risk manager sees the quoter's net inventory and an eight-row table of open lots ordered worst-to-best by mark).

4. A reward design for the desk. A general-sum split of desk profit, a blend toward the desk total, and a markout term charging the quoter for the mid's subsequent move after each fill, intended to mitigate the credit-assignment problem that the split itself creates.

Main Findings

• The decomposition alone performs poorly. Training the quoter and risk manager as separate agents without reward shaping did not produce a good desk.
• Reward design is the decisive ingredient. Reward designs aimed at multi-agent credit assignment "substantially improve the resulting desk relative to the unshaped decomposition" in the reported experiments, making the shaped desk the paper's central result.
• The redesigned single agent beats the retrained baseline. It quotes more frequently and incurs smaller held-out losses than the JaxMARL-HFT baseline configuration retrained on the same data.
• That single-agent gain is not decomposed. Because observations, actions and risk constraints were changed jointly, the contributions of each are "not separately identified."
• No configuration was profitable on average. None of the models establishes positive average portfolio value in periodic held-out evaluation, scored on realised end-of-episode portfolio value on held-out markets.
• Statistical caveat. Each configuration was trained with only one seed.
• The no-trade collapse is cited as motivation, not solved by assertion. The paper notes that the JaxMARL-HFT maker selects the no-trade action between 61.1% and 97.7% of the time across three seeds, and treats this as a property of environment design (reward and action space) rather than of the learner.

Methodology in Plain English

The authors start from a raw data problem: Polymarket tells you how the book looked after every update, but not which individual orders were placed or cancelled, so you cannot faithfully replay the market. They infer those individual orders from the book updates and store them in LOBSTER, a standard order-message format, so that a matching engine can replay them message by message.

Those replays feed JAX-LOB's matching engine through JaxMARL-HFT. At each step every learning agent picks a row from a fixed quote table (bid offset, ask offset, and size on each side, measured in ticks from the best bid and ask with the agent's own resting orders removed). Those messages are shuffled together with the recorded market messages for that step, the matching engine produces trades, and each agent receives a new observation and reward. An episode is one market window; the recurrent summary is cleared at episode boundaries so history does not leak across unrelated markets.

Evaluation is deliberately out-of-sample: every model is scored on realised end-of-episode portfolio value on markets excluded from training, against the retrained JaxMARL-HFT baseline. Configurations are built as ablations of one design choice at a time where possible. Detailed experimental settings, learner choices (the paper argues for on-policy policy gradients over value-based methods in its section 4.1.1), and the per-configuration results tables appear in sections 4–6, which are not included in the content available here, so the specific numerical scores are not reported in this summary.

Why This Matters

Impact on research. Most RL market-making work trains in model-driven simulators with Brownian or mean-reverting prices and parametric fill probabilities, which the paper argues understates adverse selection and inflates apparent profitability. This work instead trains on replayed real order flow from a venue whose books are thin and whose contracts are binary, and it proposes a new axis for multi-agent market making: decomposing a single seat cooperatively, as opposed to the adversarial and competitive directions that dominate prior MARL work.

Real-world applications:

• Liquidity provision on prediction-market exchanges such as Polymarket, which the paper notes contributed $1 million to market-making rebates on the very markets studied.
• Building replay-grade datasets and simulators for venues that publish only L2 book updates.
• Multi-market or combinatorial market making, where many logically linked binary contracts are quoted together.
• Benchmarking learned quoting policies against closed-form stochastic-control makers under realistic adverse selection.

Industry relevance. The pipeline and protocol are framed explicitly as the foundation for makers that quote many linked markets at once, which is the operational shape of a real prediction-market desk rather than a single-book research toy.

Future Directions

• Identify the ablations separately. Re-run the single-agent redesign with observations, actions and risk constraints varied one at a time, since the current gain is a joint effect.
• Scale to many linked books. Extend the multi-agent desk from one market to portfolios of correlated contracts, exploiting the combinatorial constraints the paper cites as a source of edge.
• Strengthen the evidence base. Train multiple seeds per configuration and report periodic held-out evaluation, given that no configuration currently shows positive average portfolio value and each was trained once.
• Explicit coordination between desk agents. The current desk relies on implicit coordination through augmented observations and the reward split; communication actions, shared parameters, or alternative decompositions (e.g. hierarchical, as in prior work) remain untested here.

Target Audience

Researchers and graduate students in reinforcement learning for finance, market microstructure, and cooperative MARL; quantitative practitioners interested in automated market making on prediction markets; and anyone building replay-based simulators or order-level datasets from L2-only exchange feeds. Readers looking for a profitable, production-ready trading policy will not find one here; readers looking for an environment design, data pipeline, and reward-shaping study will.

Authors’ abstract

A market maker provides liquidity by quoting prices to buy and sell an asset, earning the spread while managing inventory risk. We study market makers trained with multi-agent reinforcement learning on prediction markets, specifically Polymarket, whose thin order books pose distinct challenges. Three challenges underpin this research: adverse selection may cause a policy trained on real data to stop quoting; the consequences of a quote can arrive many steps after the action; and prediction-market L2 data does not identify individual orders needed for simulation. We address these challenges by reconstructing order-level LOBSTER-format data from Polymarket and training agents in JaxMARL-HFT on 417 real markets. We enrich the baseline market maker's observation space with the book and recent flow, redesign its actions and inventory controls, and propose a cooperative multi-agent decomposition into a quoter and a risk manager trained using recurrent Independent Proximal Policy Optimisation (IPPO) with RNN policies. We investigate reward shaping, particularly reward blending and post-fill markout, to mitigate credit-assignment difficulties between the agents. Our redesigned single-agent market maker quotes more frequently and incurs smaller held-out losses than the retrained JaxMARL-HFT baseline, although the contributions of jointly changed observations, actions and risk constraints are not separately identified. Decomposition alone performs poorly; careful reward design substantially improves the desk relative to the unshaped split and yields our best result. Models are evaluated on realised end-of-episode portfolio value on held-out markets. None establishes positive average portfolio value in periodic held-out evaluation, and each configuration was trained with one seed. The environment, data and protocol lay a foundation for makers that quote many linked markets at once.