LOLPlay: Benchmarking Real-Time Visual Agents in League of Legends

AI summary

Overview

Research area: AI agents / computer-use and game-playing benchmarks — specifically real-time visual agents that must reason and act while an environment keeps changing.

Technical level: Intermediate. The paper is readable without deep RL background, but assumes familiarity with agent benchmark concepts (task success rate, subgoal checks, observation–action loops, reasoning-effort settings).

Scope: The paper introduces LOLPlay, a benchmark of 36 champion-specific tasks with 82 executable subgoal checks in a live, unpaused League of Legends client, and reports 324 task evaluations of three models at three reasoning-effort settings.

What This Paper Is About

Most agent benchmarks let the world pause while the model thinks. In real settings — and in a live MOBA — the scene changes between the screenshot the agent sees and the moment its click lands. LOLPlay tests whether visual agents can turn an understanding of a scene into timely actions that complete a sustained, multi-part goal while gameplay continues uninterrupted from screenshot capture through reasoning to action execution.

Key Contributions

1. A native, unpaused MOBA benchmark. 36 champion-specific tasks spanning six families — economy and farming (6 tasks, 7 subgoals), development (9, 26), sustain (7, 18), combat (6, 13), cooperation (3, 8), and map objectives (5, 10) — with 5–20 minute budgets, run in a real League of Legends client prepared automatically through the local League Client Update (LCU) API.

2. Executable task- and subgoal-level scoring. 82 subgoal checks use Live Client Data API state snapshots and timestamped events to verify final-state, deadline, attribution (who killed or assisted whom), and ordering requirements — for example, that a personal Voidgrub kill precedes destruction of the enemy top outer turret. The agent's own completion message does not determine success.

3. An empirical study of reasoning effort under real-time pressure. Three models (GPT-6 Astra, GPT-6.1 Sol, DeepSeek-v4.1-Flash) at low, medium, and high reasoning effort across all 36 tasks: 324 task runs and 17,221 completed model responses.

4. Trajectory-level diagnosis of observation–action mismatch. Case studies link stale observations, action-context changes, incomplete subgoal composition, and response-budget exhaustion to concrete failures.

Main Findings

• Lowest ceiling was 25.0%. GPT-6 Astra at low and medium effort shares the highest task success rate at 25.0% (9/36 passes each). Across the cohort, 40/324 runs pass (12.3%) and 284 fail, covering 12 distinct passing tasks.

• Success is concentrated in economy and development. Those two families account for 19 passes each. The remaining two passes are on Malphite's combat-category task, which requires reaching level 5 with at most one death rather than securing a kill. Sustain, cooperation, and map-objective tasks have no successful run.

• Partial progress is common but rarely becomes completion. Subgoal success rate is 28.8% averaged across all nine configurations; GPT-6 Astra at medium effort reaches the highest at 40.3%. Of the 284 failed runs, 108 (38.0%) satisfy a proper subset of requirements without completing the task.

• More reasoning effort did not help. High effort improves neither task success rate nor subgoal success rate over low effort for any model. Astra completes 9, 9, and 6 tasks at low/medium/high effort (25.0%, 25.0%, 16.7%); Sol 5, 4, 4 (13.9%, 11.1%, 11.1%); DeepSeek-v4.1-Flash 2, 1, 0 (5.6%, 2.8%, 0%). Astra's subgoal success moves 37.7% → 40.3% → 33.7%, so medium effort buys partial progress but not more completed tasks.

• Longer reasoning shrinks the interaction budget. For Astra, mean request latency rises from 7.47 to 8.39 to 16.67 seconds while mean responses per episode fall from 43.9 to 37.8 to 27.1 — fewer decision rounds as thinking time grows.

• Speed alone does not rank agents. DeepSeek-v4.1-Flash (low) averages 2.81 seconds per request and 89.8 responses per episode versus 7.47 seconds and 43.9 responses for GPT-6 Astra (low), yet scores 5.6% task / 22.2% subgoal against 25.0% / 37.7%.

• Budget exhaustion reveals ineffective interaction. Across all configurations, 129 runs end at the time budget and 75 at the 100-response cap; every capped run ends before its wall allowance. Of DeepSeek's 55 capped low/medium runs, 32 achieve no subgoal, and across these runs only 5/93 positive-attainment checks pass. One Malphite episode uses 100 responses containing 101 right-click commands yet finishes at level 1 with zero deaths — the survival constraint passes but the level-5 objective does not.

• Which requirements agents can actually satisfy. Checks for minimum learned ability levels pass in 25/36 instances and level thresholds in 52/108, versus 4/9 for item possession, 10/144 for minimum creep score, 0/45 for personal kills, and 0/27 for allied assists.

• Development progress outpaces completion. Development tasks average 55.5% subgoal success but only 23.5% task success; sustain tasks average 27.0% subgoal success with zero successful episodes.

• A successful episode is not clean play. Cho'Gath (task 013) succeeds with level 6, R rank 1, and a maximum-health increase of 795.3 after 420.8 seconds of execution against a 540-second budget — but across 33 responses and two deaths.

• Stale observations produce misdirected actions. In Warwick (task 035), a 144.641-second request spans a personal Voidgrub kill at 09:19.65 and a death at 09:31.74; the agent's CLICK(810,465) then lands inside an inactivity dialog in the changed frame at 11:42. Only one of three checks passes.

• Human feasibility was established. Two players with more than 200 prior League of Legends matches each completed all 36 tasks (100% task completion) under the same configurations, budgets, and evaluator.

Methodology in Plain English

Each task is a tuple of instruction, starting configuration, time budget, a set of required subgoals, and a dedicated evaluator. The harness uses the local League Client API to spin up a fresh custom match — creating the lobby, inserting beginner-difficulty bots, locking the specified champion, setting summoner spells, rune page, skin, and Q/W/E/R to normal casting — then waits for the Live Client Data service and validates the initial state before the clock on the agent's budget starts. Preparation time is excluded.

The agent sees only screenshots plus the instruction and recent history; no unit list, coordinates, health bars, or evaluator state are provided. It emits mouse–keyboard actions — the 13 pointer, click, scroll, keyboard, and shortcut action types of the computer_13 adapter plus WAIT, FAIL, and DONE markers — and a single response may contain several actions executed in sequence. Crucially, the game never pauses: it runs through observation, generation, and execution, so an action can arrive after the encounter it was planned for has ended. Episodes end at the task's time budget, at 100 model responses, or when the agent signals DONE or FAIL.

Scoring is entirely external. Evaluators query the Live Client Data API for final-state, deadline-snapshot, event-attribution, and event-ordering evidence. Task success rate weights every task equally; subgoal success rate averages the fraction of verified subgoals per task. The paper also reports mean request latency (including service, network, retry, and parsing overhead) and mean responses per episode as descriptive, non-causal comparisons.

Why This Matters

Impact on research. Existing computer-use and game benchmarks (OSWorld, OSWorld 2.0, VideoGameBench, GameWorld, GameVerse) mostly score task completion or progress milestones. LOLPlay's contribution is the evaluation unit: champion-specific objectives with jointly required state, participant, and event-order checks in an environment that never pauses. That conjunction exposes the gap between "the agent did something useful" and "the agent completed the goal," and it turns reasoning effort into a variable that can be measured against decision budget rather than assumed to help.

Real-world applications implied by the setting:

• Live-UI automation where dialogs, notifications, and shifting layouts can invalidate a planned click between screenshot and execution.
• Robotic and embodied control, where the scene evolves during inference and stale observations are the default condition.
• Operations and incident response, where an agent must satisfy several simultaneous constraints — act, preserve safety margins, and hit a deadline — rather than a single end-state.
• Evaluating agent cost/latency tradeoffs, since the paper shows faster responses and more interaction do not compensate for ineffective goal-directed control.

Industry relevance. The benchmark provides a reproducible harness (LCU for setup, Live Client Data for scoring) that separates preparation from evaluation, which is directly reusable for anyone shipping agents into stateful, time-sensitive software. The finding that higher reasoning effort reduces the number of decision rounds without improving completion is a concrete argument for tuning effort per deployment rather than maximizing it. The paper explicitly notes the intended setting is custom bot matches and that uncontrolled deployment of desktop input could enable unwanted gameplay automation.

Future Directions

• Replanning after intervening events. The Warwick case motivates refreshing observations and replanning when the scene changes mid-request; the paper does not establish that latency alone caused the death or the failure, so a fixed-model delay intervention remains open.
• Agents that retime rather than over-reason. Astra's low-to-high latency more than doubles while task success falls by one-third, but Sol's latency is non-monotonic (8.90, 11.44, 10.26 seconds), so effort is not a uniform latency intervention. Isolating a causal effect of delay is an explicit open question.
• Composing subgoals instead of accumulating them. With 108 failed runs satisfying a proper subset of requirements and 0/45 personal-kill and 0/27 allied-assist checks passing, the gap between local progress and completed objectives is the central target.
• Broadening task coverage. The authors note the 36 tasks cover one objective per champion rather than the full diversity of MOBA play, and that some requirements can be satisfied through passive progress — limitations they flag without changing the observed outcomes.

Target Audience

Researchers and engineers building computer-use, GUI, or game-playing agents; benchmark designers who need executable, state- and event-based scoring rather than judged outcomes; and practitioners tuning inference-time reasoning effort for latency-sensitive deployments. It is also useful for readers tracking evaluation methodology for agents operating in continuously changing environments.

Authors’ abstract

Visual agents need to coordinate reasoning and action while their environment evolves during inference. We introduce LOLPlay, a benchmark in native \emph{League of Legends} with 36 champion-specific tasks spanning resource management, combat, cooperation, and objectives. Agents interact through screenshots and mouse--keyboard actions; automated initialization and 82 executable subgoal checks distinguish partial progress from completion. Across 324 task evaluations of three models at three reasoning-effort settings, GPT-6 Astra achieves the highest task success rate of 25.0\% at low and medium effort, and the highest subgoal success rate of 40.3\% at medium effort. However, high effort improves neither metric over low effort for any model. For Astra, longer requests leave fewer interaction rounds, suggesting a cost to additional reasoning in real-time control. Trajectories reveal incomplete subgoal composition and actions executing after the observed scene changes, motivating agents that combine visual reasoning with timely action.