JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making

AI summary

Overview

Research area: AI agent evaluation, probabilistic forecasting, and cost-aware decision-making — specifically LLM agents operating in a historical sports-betting environment.

Technical level: Intermediate to Advanced. The protocol section is accessible to a motivated reader, but the evaluation metrics (log gain in nats, capped-Kelly settlement, cluster-robust inference) assume familiarity with probabilistic scoring and betting mathematics.

Scope: The paper introduces JockeyBench, a benchmark and evaluation protocol built from twelve years of Japanese horse-racing records, and uses it to test whether 28 LLM model–scaffold configurations can turn noisy historical evidence into forecasts that survive costs and fixed betting rules.

What This Paper Is About

LLM agents can now write their own predictive programs, revise them as outcomes arrive, and pay for the evidence they use. What remains unclear is when a predictive edge actually translates into value across a long sequence of decisions. The authors build a controlled replay of Japanese horse racing — with strict information cut-offs, priced data tools, and paired frozen baselines — so that forecast quality, adaptation, research cost, and realized betting outcomes can each be measured separately rather than conflated into a single score.

Key Contributions

1. A leakage-controlled historical replay resource. JockeyBench is assembled from the JRDB archive: 40,357 races and 564,224 runner entries spanning 2015–2026, linked across races, horses, jockeys, trainers, and pedigrees, with every record timestamped to the moment it would have become public. Agents see 97 whitelisted fields in five groups (race conditions, runner attributes, workouts, past starts, jockey and trainer statistics), while odds, popularity, dividends, expert ratings, and free text are withheld.

2. A dual-workflow protocol that relocates LLM reasoning. In Program, the LLM writes and revises an executable predictor that runs each race. In Adaptive Acquisition, the LLM writes an evidence-acquisition program that selects data, followed by a fresh per-race LLM forecast. Both share the same release clock and tariff.

3. Paired frozen-program controls. Each daily-revision trajectory is matched against its own frozen initial program running the same races under the same tariff, so revision gains are separated from final forecast quality.

4. A joint evaluation separating four things that aggregate scores merge: predictive quality (smoothed log gain, Top-1, Top-3), benchmark-specific net utility (Net = 100·Σ[g_r + I_r(h_r − 1/n_r)] − C, with a weighted research fee over evidence cells, tool calls, and CPU seconds), and retrospective betting returns and path risk under flat and capped-Kelly staking, with provider charges reported separately.

Main Findings

• Delivery is not competence. Agents produced 40,428 predictor outputs among 44,712 forecasts, but Granite-4.2-8B delivered a predictor 90% of the time and still scored a mean log gain of −0.087. Gemma-4-E4B and Nemotron-3.5-Lightning never delivered a predictor at all. The best configuration, GPT-6.1-Sol, reached 0.525 nats (between-run SD 0.002) with Top-1 of 28.5%.

• Reference strength determines the story. GPT-6.1-Sol's 0.525 exceeds the card-and-recent-history reference R2 at 0.343 and sits near the agent-inspired feature ensemble at 0.523. But broad factual features reach 0.608, the expert forecast 0.679, the pre-race market 0.700, and the closing market 0.757 — the highest in every period tested. R3, with expanded factual features, underperformed R2 in every period.

• Revision gains are not final quality. After Holm correction across 28 tests, there were seven significant revision gains and two declines. The largest gain, +0.359 for Nemotron-3-Ultra-550B, still left its revised mean below uniform at −0.083, while the two highest-scoring configurations showed no revision gain at all.

• No configuration made money. Under retrospective capped-Kelly settlement (JPY 100,000 initial bankroll, Kelly fraction capped at 1% of day-start bankroll per race, official JRA dividends including the 20% WIN takeout), every one of the 28 configurations had negative mean profit. Thirteen of 138 individual trajectories finished in profit, and a separately archived 54-trajectory, 22-model flat-WIN subset showed pooled ROI of −19.7%.

• Program beat Adaptive Acquisition on profit and fees — while both lost. In the four-model comparison (DeepSeek-V4-Flash, GLM-5.3-Flash, GPT-6-Astra, MiMo-V2.6-Pro), Program had higher mean terminal P&L under both staking rules and smaller maximum drawdowns in all eight comparisons. For DeepSeek, Program gained roughly 1,000 score-yen in gross reward while Adaptive incurred 4,551 more in research fees per trajectory — enough to erase its predictive advantage.

• Return and risk orderings disagree. For DeepSeek under Kelly, Program lost JPY 38,305 with drawdown JPY 39,405, versus JPY 39,395 lost and JPY 58,663 drawdown for Adaptive. Yet Adaptive had the less negative Sharpe (−6.33 versus −9.78) and slightly higher turnover ROI (−30.18% versus −31.14%).

• Probability bias reverses on selected bets. For DeepSeek Adaptive, predicted versus observed win rates were 18.14% and 21.25% across all non-refunded top picks, but 17.83% and 7.82% on the Kelly-selected subset. Mean bias over all forecasts concealed overprediction exactly where decisions were made.

• Profitable runs can be concentrated. DeepSeek Adaptive R1 earned JPY 1,470 under Kelly while the other five completed trajectories lost JPY 47,568 on average; three winning tickets accounted for 55.35% of R1's gross returns. MiMo's Program Kelly profits ranged from JPY −70,500 to JPY +73,080 across five trajectories.

• Fresh per-race inference is no guarantee. Adaptive Acquisition produced direct forecasts for 96.9–99.4% of outputs with positive mean log gain for all four models, but only DeepSeek exceeded Program on log gain (0.256 versus 0.232). Adaptive cohorts contained six, one, two, and one completed trajectories, so these are descriptive comparisons.

Methodology in Plain English

The researchers rebuilt a raw commercial archive of Japanese racing records into a replay database. Every field was tagged with the time it became public, identities were pseudonymised, and the agent's view was restricted by a whitelist. Odds and expert ratings were removed from inputs entirely, so forecasts had to be built from factual evidence like past performances, workouts, and trainer statistics.

Two agent designs were tested. In the Program workflow, an LLM writes a forecasting program during an exploration session using races from before the evaluation window, then each day it receives feedback and may revise the code. The program itself runs each race and outputs a ranking plus win probabilities. In the Adaptive Acquisition workflow, the LLM writes a smaller program that only decides which evidence to fetch, and a fresh LLM session then makes the forecast from that evidence.

The daily cycle is fixed: cards appear at 21:00 JST the day before, forecasts are sealed before the races, results are released at 20:30 JST, and only then can the agent reflect and revise. Exploration allows up to 40 model calls and daily reflection up to 20; conversations restart between sessions, so only code, fitted parameters, and notes persist.

To measure revision, every revising trajectory is paired with a frozen copy of its own starting program running the same races. To measure cost, evidence cells, tool calls, and CPU seconds are all charged under one published tariff. To measure consequence, sealed top picks are settled retrospectively against official dividends under two fixed rules — a flat 100-yen win bet and a Kelly bet capped at 1% of the day-start bankroll. Odds are used only for settlement, never shown to agents. Uncertainty is estimated by clustering races by day across the ten-day window.

Why This Matters

Impact on research. The paper makes a methodological argument that applies well beyond racing: benchmarks that report a single accuracy number can hide the difference between an agent that runs, an agent that predicts, an agent that improves, and an agent that is worth deploying. The paired frozen control is a cheap and reusable design for isolating adaptation, and the explicit tariff turns "cost-aware evaluation" from a slogan into an arithmetic comparison.

Real-world applications:

• Sports analytics and wagering: evaluating whether a model's probability estimates survive takeout and staking rules, not just whether they rank runners well.
• Financial forecasting agents: the same structure — time-gated data, priced evidence, sequential capital allocation, drawdown — maps directly onto trading and portfolio agents.
• Agent procurement and deployment: separating provider charges, research fees, and decision value lets buyers see which cost line is actually eating the return.
• Benchmark design for long-horizon agents: the frozen-control pattern is transferable to any setting where an agent revises itself over time.

Industry relevance. For anyone deploying LLM agents in sequential decision settings, the headline is cautionary: positive log gain coexisted with negative realized profit in every configuration tested. Teams that select a model on forecast score alone, or that pick the one lucky run out of five, are measuring the wrong thing.

Future Directions

• Test beyond horse racing. The authors state explicitly that generalization to other sports, unstructured information environments, and broader decision tasks remains to be established.
• Vary the pricing scheme. Costs are compared under one common tariff; how agents change their information acquisition and resource allocation under alternative pricing remains unexplored.
• Separate model capability from scaffold. Results reflect the specific model–scaffold configurations tested, and different tool interfaces, reasoning budgets, or later model versions may reorder the rankings.
• Address difficulty-based model routing. The authors note that cost-per-outcome comparisons in deployment can be confounded by routing, which their fixed-history evaluation does not assess.
• Extend the time window. The evaluation covers ten shared days; whether the short-term revision dynamics observed hold over longer horizons is untested.

Target Audience

Researchers building or evaluating LLM agents for sequential decision-making, benchmark designers interested in cost-aware and leakage-controlled evaluation, and quantitative practitioners in sports betting or finance who want to know whether a model's probabilities survive real settlement rules. Readers focused purely on prompting techniques will find less here; the paper's value is in measurement design, not in a new agent architecture.

Authors’ abstract

Repeated event forecasting requires extracting predictive signals from noisy evidence and evaluating their value across successive decisions. We introduce JockeyBench, a benchmark and evaluation protocol that studies this problem through historical Japanese horse racing. The protocol combines time-gated evidence, priced tools, paired frozen-program controls, and two workflows: executing an LLM-written predictor or acquiring evidence for a fresh per-race LLM forecast. Evaluation separates forecast quality, benchmark-specific net utility, and retrospective betting returns and risk under fixed decision rules. Across 28 model--scaffold configurations on 324 races, paired tests identify seven revision gains and two declines after multiple-testing correction. Yet every configuration has negative mean profit under capped-Kelly settlement. In a four-model comparison, the reusable-program workflow has higher mean betting profit and lower research fees, although both workflows lose on average. Further diagnostics reveal that mean probability bias reverses on selected bets and that profitable individual runs can depend on concentrated returns. These findings motivate evaluating predictive competence, adaptation, resource use, and decision value separately when assessing agents for repeated event forecasting.