Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?

AI summary

Overview

Research area: Evaluation of large language model (LLM) agents — specifically, how well agents acquire and reuse new knowledge through interaction in unfamiliar environments (test-time learning without weight updates).

Technical level: Intermediate. The paper is mostly about benchmark design and empirical comparison, but it uses formal metrics (normalized scores, learning gain, learning slope) and several named agent methods.

Scope: The paper introduces Learn2Play Bench, a benchmark of 20 newly designed text-based games with hidden rules, and uses it to compare 11 backbone models, 6 self-evolving memory methods, 3 agent harnesses, and human players.

What This Paper Is About

Existing benchmarks for "self-evolving" agents mostly use tasks whose rules are either stated in the instructions or already known to pretrained models, so it is hard to tell whether an agent improved by learning something new or by recalling pretraining knowledge. The authors build a benchmark of text-based games with deliberately novel or counterintuitive hidden rules — for example, a game in which stealing a colleague's lunch helps the player get promoted while working hard does not — so that agents must discover rules from feedback. The goal is to measure, in a controlled and reproducible way, how effectively LLM agents learn from experience and whether they can transfer what they learn to new instances of the same game.

Key Contributions

1. A controlled benchmark with hidden rules. Learn2Play Bench consists of 20 manually designed game templates, each able to generate multiple instances with different random seeds. Games have deterministic feedback and automatic scoring, allowing repeated, reproducible evaluation of learning without model weight updates.

2. A transfer setting. Ten of the 20 games are "fixed" (the same instance repeats across episodes) and ten are "re-shuffled" (the visible instance changes while the underlying rule stays the same), which tests whether an agent can apply learned knowledge to new situations.

3. A systematic comparison. The authors compare 11 backbone models under a common harness, 6 self-evolving methods with a fixed backbone, and different agent harnesses (OpenCode, Claude Code, Codex), and they also measure inference cost.

4. A human study on the same games. College students play the same games (21–30 participants per game), and their scores and interaction records are compared with those of the agents.

Main Findings

• Experience retention: raw history beats summarization. With Kimi K3 held fixed, Memory — which cumulatively adds complete, unmodified interaction histories — leads all four metrics (Max 63.3, Mean 48.0, LG +16.9, LS +4.69) among the six methods, ahead of EvoTest (62.4 / 48.0 / +1.8 / +1.01), Reflexion (61.6 / 40.4 / +14.8 / +3.35), ReasoningBank (59.9 / 40.1 / +8.3 / +2.59), AWM (56.1 / 37.0 / +4.8 / +0.62), and Naive (54.7 / 31.9 / +0.3 / +0.42). The paper's explanation is that distilling incomplete evidence into rules or workflows creates beliefs that later episodes keep following, as in a GemForge run where an EvoTest–Opus 5 prompt told the agent to avoid combinations it had never tried, including one that would have succeeded.

• OpenCode beats the self-evolving methods on the same backbone. Even with the backbone fixed, OpenCode scores higher than the evaluated self-evolving methods, because it lets the model decide when to review feedback, run calculations, or check a plan, rather than following preset steps.

• Backbone results are not the same as learning results. Under OpenCode, Claude Opus 5.5 leads Max (80.1) and Mean (61.0), but Qwen 3.8 Max leads Learning Gain (+25.2) and Learning Slope (+6.30). The full backbone Max/Mean/LG/LS set is: Claude Opus 5.5 80.1/61.0/+23.0/+5.22; GPT-6 Astra 76.2/56.1/+23.8/+5.88; Claude Opus 5 74.5/53.8/+21.3/+5.13; Qwen 3.8 Max 72.5/51.1/+25.2/+6.30; Gemini 3.5 Flash 72.0/52.2/+19.8/+5.37; Kimi K3 68.5/50.2/+17.1/+4.24; Gemini 3.7 Flash 58.3/39.9/+10.7/+2.47; GPT-5.6-SOL 57.0/41.1/+14.5/+3.71; Claude Sonnet 5 55.9/35.6/+15.7/+3.72; DeepSeek V4 Pro 54.2/38.2/+14.7/+3.66; GPT-5 Mini 34.0/22.3/+9.2/+1.83.

• Top humans outperform the best agent. Human Top-1 reaches Max 84.3, Mean 53.5, LG +21.6, LS +7.33; Top-3 reaches 79.1/50.1/+18.9/+5.54; Top-5 reaches 75.1/47.1/+17.1/+4.62. In fixed games, consecutive action sequences have an average similarity of 0.54 for humans versus 0.66 for agents, and after a lower-scoring episode humans reach a new personal best 33% of the time versus 22% for agents.

• Harness choice matters as much as the model. With Opus 5, Claude Code improves on OpenCode across all four metrics (81.8/62.3/+27.1/+6.59 versus 74.5/53.8/+21.3/+5.13). With GPT-5.6-SOL, Codex gives 74.3/54.8/+22.2/+5.66 versus OpenCode's 57.0/41.1/+14.5/+3.71. The paper attributes this to models being trained on their harnesses and harnesses being adapted to model capabilities.

• Changing surface details slows learning. In re-shuffled games the learning slope drops even though the underlying rule is unchanged. The drop is 0.59 for Claude Opus 5 and 0.88 for Kimi K3, but 1.73 for Gemini 3.5 Flash, 1.94 for GPT-5.6-SOL, 2.15 for Gemini 3.7 Flash, and 2.67 for Claude Sonnet 5; Qwen 3.8 Max shows a negative drop of -0.41 and Codex (GPT-5.6-SOL) -0.19.

• Higher cost does not mean better performance. Kimi K3 achieves higher performance at lower overall cost than Claude Sonnet 5 despite higher token prices, using 0.56M tokens per episode on average versus 0.73M for Claude Sonnet 5. With Kimi K3, OpenCode uses 56% fewer input tokens than Memory and 71% and 41% fewer generated tokens than EvoTest and ReasoningBank respectively.

• Agents can learn rules but still fail to plan. In one Opus 5 Haunted Inn run the agent states the relevant taboos but assigns rooms greedily and leaves a later guest with no safe room; only in the next episode does it reserve rooms for guests with fewer safe options.

Methodology in Plain English

The authors designed 20 text-based games from scratch rather than reusing existing ones, and deliberately gave them hidden rules that conflict with ordinary real-world expectations. A player is shown a state and a list of valid actions, picks one action, and gets textual feedback; each episode ends with an automatically computed score. The games use fixed seeds so outcomes are deterministic and can be replayed and compared.

Each game is evaluated over repeated episodes — five episodes for normal games and ten for the five harder "challenge" games (Catnip, DreamGarden, Mola Tea, RedEye, Roadside Observatory) — with three independent trials per game, each trial starting from no memory. Raw scores are normalized to a common scale using a per-game reference score, then four metrics are computed: Max (average peak score across trials), Mean (average across episodes), Learning Gain (late episodes minus early episodes, using the first and last k episodes where k = min(3, floor(T/2))), and Learning Slope (the slope of a linear fit through the episode scores). Games are weighted 3 for challenge games and 1 for normal ones, so the five challenge games and fifteen normal games each carry 50% of the weight.

Against this protocol the authors swap in different components while holding others fixed: 11 backbone models under OpenCode; 6 experience-retention methods (Naive, Memory, Reflexion, EvoTest, ReasoningBank, AWM) each tested with Kimi K3, Claude Opus 5, and GPT-5 Mini; alternative harnesses (Claude Code for Opus 5, Codex for GPT-5.6-SOL); and fixed versus re-shuffled versions of the same games. Separately, human participants recruited from college students played the same games through the same interface, and their Top-1, Top-3, and Top-5 records by Max were compared with the agents.

Why This Matters

Impact on research. The paper's central methodological point is that improvement on a familiar task does not prove that an agent learned anything — it may simply be recovering pretraining knowledge. By using rules that are new or counterintuitive, and by comparing identical games with and without instance changes, the benchmark offers a cleaner way to separate learning from recall. This matters for the fast-growing literature on self-evolving agents, where methods are often evaluated with different models, tasks, and protocols, making them hard to compare directly.

Real-world applications.

• Robots or software agents facing an unfamiliar command interface, where the agent must try commands and observe effects before acting competently.
• Enterprise workflow automation where an agent is deployed into a new tool, API, or internal system whose conventions are not in any training corpus.
• Customer-facing agents that must adapt to a specific user's feedback and preferences over repeated sessions without retraining.
• Scientific or diagnostic assistants that must infer the rules of a new instrument, simulator, or stateful machine by probing it.

Industry relevance. The finding that harness choice can raise performance and learning while also lowering estimated inference cost is directly actionable for teams deploying agents, since it suggests gains are available without switching to a more expensive model. The result that retaining raw interaction history can beat summarization into rules or workflows is also relevant to anyone building memory systems for production agents, and the human-agent gap points to exploration/exploitation balance as an engineering target.

Future Directions

• How should agents balance exploration and exploitation? Because Human Top-1 reaches a higher Max than every agent configuration evaluated, the authors argue future work should let agents test new strategies even when their current strategy is already scoring well.
• How should learned rules be stored? The authors suggest linking each learned rule back to the original interactions behind it, so agents can revisit the evidence and revise a rule when new feedback arrives.
• How can agents learn what stays constant as environments change? The transfer results show surface changes reduce learning slopes even when rules are fixed; methods could help agents compare experiences across situations, separate shared rules from instance-specific details, and check when rules still apply.
• Can agents plan ahead rather than act greedily? The Haunted Inn case shows an agent that knows the rules but chooses actions that work now while preventing success later.

Target Audience

Researchers and engineers working on LLM agents, test-time adaptation, and self-improving or self-evolving agent systems, as well as anyone designing evaluations for agent memory, harnesses, or continual learning. It is also useful for practitioners choosing a backbone model or agent harness for a deployment where the environment's rules are not known in advance, and for readers interested in human-versus-agent comparisons of exploration behavior.

Authors’ abstract

Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments. Evaluating this ability is therefore important for understanding how effectively agents acquire and use new knowledge. Existing benchmarks have sought to evaluate this ability, but they primarily evaluate tasks whose rules are provided in the instructions or already familiar to pretrained models, making it difficult to distinguish learning from interactions from reasoning with existing knowledge. To address this, we introduce Learn2Play Bench, a benchmark of newly designed text-based games, whose rules are novel or counterintuitive, requiring agents to acquire knowledge through interaction rather than rely solely on pretrained knowledge. These games provide reproducible feedback and automatic scoring, enabling controlled evaluation of learning across repeated attempts. We also vary game instances to test whether agents can apply what they have learned to new situations. Therefore, we evaluate how backbone models, self-evolving methods, and agent harnesses affect agents' learning ability, revealing three findings: (1) Experience retention: Retaining complete records of actions and feedback can support more effective learning than summarizing these experiences into rules or strategies. (2) Human agent gap: Top-performing human players achieve higher peak scores than the evaluated agents. Human explore more varied strategies, and repeat actions less. (3) Harness matters: With the backbone fixed, changing the harness can improve performance while reducing estimated inference cost. Together, these findings provide insights into how LLM agents learn from experience and suggest directions for future work to improve their learning ability. Project website: https://liushiliushi.github.io/learn2play-bench-website/