S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

AI summary

Overview

• Research area: Large language model agents, interactive benchmark design, and experience-driven self-improvement.
• Technical level: Intermediate. Readers should be comfortable with LLM agent loops, in-context learning, memory summarization, and supervised fine-tuning, though no mathematics beyond the paper's own notation is required.
• Scope (one sentence): The paper introduces S3Gym, an interactive benchmark with seven text-based games that tests whether an LLM can explore, judge its own experience, and convert that experience into measurably better behavior under strict held-out evaluation.

What This Paper Is About

Existing agent benchmarks measure how capable a model is at evaluation time, treating it as a fixed policy, so they say little about whether a model can actually learn from its own interactions. The authors argue that experience only becomes useful when an agent actively generates diagnostic evidence (Self-Testing), interprets what succeeded or failed (Self-Judging), and turns that interpretation into changed behavior (Self-Improvement). S3Gym is built to separate these three steps and to measure whether the last one really happens, by giving the agent permissive exploration episodes and then testing it on stricter, held-out configurations while hiding the environment's ground-truth rewards during exploration.

Key Contributions

1. A unified formulation of experience-driven self-improvement. The paper formalizes self-improvement as an iterative cycle — Explore → Judge → Consolidate → Update → Evaluate — and instantiates it in S3Gym with separate exploration and held-out evaluation phases.
2. A systematic comparison of experience-integration pathways. It evaluates three mechanisms under one protocol: History ICL (raw score-annotated trajectories in context), Summary Memory (model-generated rules, mistakes, and next directions), and parameter Training (supervised fine-tuning on exploration trajectories and self-judgments).
3. An explicit evaluation of Self-Judging. S3Gym withholds verifier-computed rewards from the agent during exploration and records both the agent's self-judged immediate scores and the environment's verifier-computed step rewards, so the two can be compared directly.
4. Comprehensive empirical and diagnostic analyses. It reports results for representative LLMs across seven interactive environments, along with analyses of improvement dynamics, judgment reliability, summary effectiveness, cross-game consistency, and common failure modes. (Only the first analysis subsection, RQ1 on parameter training, is included in the provided excerpt; the remaining analyses are not reported there.)

Main Findings

• Self-improvement is neither automatic nor uniform. The paper states that context-level experience improves performance for several model–game pairs, but the most effective pathway depends strongly on task structure.
• Summary Memory is a selective compression mechanism, not an unconditional upgrade. It helps some models substantially: Gemini-2.5-Flash raises its Minesweeper AUC+ from 0.000 under ICL to 7.794 with Summary Memory, and its PvZ AUC+ from 24.402 to 238.501; GPT-5.5 raises its Chess AUC+ from 0.474 to 16.840. But GPT-5.5's PvZ AUC+ falls from 548.499 to 33.219, and Gemini-3.5-Flash falls from 12.280 to 0.000 on Chess and from 200.000 to 55.095 on Trust Evolution.
• Different models lead under different pathways. Under History ICL, Gemini-3.5-Flash gets the strongest Chess results and the largest AUC on Snake and Trust Evolution, while GPT-5.5 leads the more brittle Minesweeper, Nullify, and Tetris tasks and has the largest PvZ AUC. Under Summary Memory, GPT-5.5 becomes strongest on Chess, Nullify, and Tetris and reaches the highest average and maximum scores on Snake, while Gemini-3.5-Flash remains strongest in average performance on PvZ and Trust Evolution.
• High base capability does not determine self-improvement. The authors attribute differences to how well models retrieve, compress, and apply their own experience.
• Parameter training yields substantial gains on Trust Evolution. For Qwen3-8B, the score rises from 0 to a maximum of 30 and stays above the initial baseline at 18 of the 19 updated checkpoints, with a post-training average of 8.684 and AUC+ of 163.5.
• Training gains on Chess and Snake are limited or transient. Chess stays at zero for most checkpoints but reaches 0.0667 at the final checkpoint; Snake reaches a score of 1 at epochs 8, 12, and 15 but then returns to zero.
• Minesweeper, Nullify, and Tetris show no measurable improvement under training. Their evaluation scores remain zero throughout training. The authors suggest one possible explanation is that exploration produces too few successful trajectories to bootstrap an effective supervised signal, though the excerpt truncates before they finish stating this hypothesis.
• Games expose different improvement regimes. Chess, Minesweeper, and Nullify have sparse or highly discrete scores, so one successful episode can inflate Max. while moving Avg. and AUC+ little; Tetris, Snake, PvZ, and Trust Evolution give denser sequential feedback and show clearer sustained adaptation.
• Metrics are complementary and must be read within a game. Max. identifies occasional breakthroughs, Avg. reflects overall performance, and AUC+ distinguishes persistent improvement from isolated peaks. The paper states that raw AUC+ values should be compared within the same game, not across games, and that cross-game consistency should be judged by within-game ranks or the number of games improved.

Methodology in Plain English

The benchmark runs each model through a repeated cycle. First comes an exploration phase, where the agent plays the game under relatively permissive configurations — smaller boards, extra lives, weaker failure penalties, or additional trial opportunities — across 30 exploration episodes. During these episodes the agent receives observations and environment feedback, but the verifier-computed rewards and final episode scores are withheld; instead, the model must output, alongside each action, its own estimate of the immediate reward under the game rules. That self-judged score is what the agent sees in its history.

The collected trajectories and self-judgments are then consolidated in one of three ways. History ICL appends the score-annotated trajectories directly to the context, with a 100,000-token input limit and a 14,000-new-token output cap. Summary Memory asks the model to compress the judged trajectories into a summary consisting of strategies to retain, mistakes to avoid, and directions for the next cycle. Parameter Training converts the trajectories and judgments into supervised fine-tuning examples, keeping high-scoring actions as positives and filtering or correcting low-scoring ones.

Finally, the updated agent is evaluated on stricter, held-out configurations using disjoint random seeds; evaluation trajectories are never added to history, memory, or training data. Evaluation runs every 3 exploration episodes, with 3 strict-mode episodes per checkpoint at checkpoints x ∈ {0, 3, 6, ..., 30}, and the controller caps interaction at 64 steps per episode unless a game's own horizon is smaller. The paper reports three metrics per model–game pair: Avg. (mean over checkpoints), Max. (best score), and AUC+ (area of sustained improvement above the initial baseline, computed by piecewise-linear interpolation between adjacent checkpoints).

The seven games are Chess, Minesweeper, Nullify, Tetris, Snake, Plants-vs-Zombies (PvZ), and Trust Evolution, chosen to cover latent-rule induction, constraint satisfaction, numerical transformation, spatial planning, long-horizon control, resource allocation, and multi-agent strategy. The main context-level comparison covers seven proprietary models: GPT-4o, GPT-4.1, o3-mini, Gemini-2.5-Flash, Gemini-2.5-Pro, GPT-5.5, and Gemini-3.5-Flash. The training pathway is studied separately with Qwen3-8B over 20 consecutive checkpoints (epoch 0 being the original model and epochs 1–19 being updated models), evaluated without History ICL or Summary Memory at inference time.

Why This Matters

The paper argues that existing benchmarks answer a static question — how capable is the model at evaluation time — while hiding the work that dominates real deployment. It separates the agent's judgments from verifier-computed outcomes so that failures can be attributed either to inaccurate self-evaluation or to an inability to convert correct feedback into transferable behavior. The reported results indicate that recognizing successful actions is insufficient; agents must also transform feedback into executable, transferable policies.

Real-world applications:

• Agent deployment and operational tuning: The paper's Background section frames this via the forward-deployed engineer role popularized by Palantir, where a general-purpose model must be adapted to a customer's data formats, workflows, and constraints. S3Gym targets the layer of that work concerning the feedback signal and the improvement loop it enables.
• Automated research and software engineering: The introduction names these as application areas where agents accumulate behavioral experience through repeated observation–reasoning–action–feedback loops.
• Games and strategic decision-making: The benchmark covers planning, resource allocation, survival, and multi-agent strategy, which map onto long-horizon control settings.
• Self-training pipelines: The training pathway is directly relevant to teams that build supervised fine-tuning data from a model's own trajectories and need to know when that data helps and when it causes negative transfer.

Industry relevance: the work comes from ByteDance Seed, M-A-P, and TokenWave.AI, and it gives practitioners a diagnostic for choosing between context-level memory and parameter updates rather than assuming either one works. The reported negative transfer under parameter training (unstable improvement on some tasks, severe negative transfer on others) is a practical warning for teams internalizing self-generated experience into weights.

Future Directions

• Diagnosing why training fails on some games. The authors hypothesize that Minesweeper, Nullify, and Tetris produce too few successful trajectories to bootstrap an effective supervised signal; whether this holds, and how to generate better evidence in sparse-reward games, remains open in the provided text.
• Improving Self-Judging reliability. The benchmark records both self-judged scores and verifier-computed rewards, and the paper notes that Summary Memory may preserve incorrect judgments. The excerpt does not report the judgment-reliability analysis, so how self-judgment error maps to improvement failure is not quantified there.
• Choosing between context-level and parameter-level improvement. Since summary compression sometimes helps greatly and sometimes destroys performance, an open question is when to compress, when to keep raw history, and when to train.
• Cross-game transfer and generalizable experience. The games use different score scales and the paper advises comparing AUC+ only within a game. Whether experience transfers across games or environments, rather than memorizing isolated actions or seeds, is the broader target the benchmark is designed to probe.

Target Audience

Researchers and engineers building LLM agents that accumulate experience, benchmark designers interested in measuring adaptation rather than static capability, and practitioners deciding between context-level memory (History ICL or Summary Memory) and parameter training for self-improvement. It is also useful for evaluation specialists who care about separating a model's own judgments from executable ground truth, and for teams assessing the risk of negative transfer when training on self-generated trajectories.

Authors’ abstract

Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience, yet existing agent benchmarks largely evaluate them as fixed policies. It therefore remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions. We introduce \textbf{S\textsuperscript{3}Gym}, an interactive benchmark for evaluating LLM self-improvement through three coupled capabilities: \textbf{Self-Testing}, \textbf{Self-Judging}, and \textbf{Self-Improvement}. S$^3$Gym separates permissive exploration from strict held-out evaluation and instantiates this protocol in seven text-based games with executable environment verifiers. We evaluate three pathways for incorporating interaction experience: direct History ICL, score-conditioned Summary Memory, and parameter Training. Our experiments reveal that self-improvement is neither automatic nor uniform. Context-level experience improves performance for several model--game pairs, but the most effective pathway depends strongly on the task structure: summaries are beneficial when experience can be compressed into reusable strategic rules, yet often underperform raw history when success depends on precise, state-contingent information. Parameter training produces substantial gains on some tasks, but also exhibits unstable improvement and severe negative transfer on others. These findings show that recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies. S$^3$Gym provides a unified framework for diagnosing this process and identifying the bottlenecks that prevent agents from translating interaction experience into reliable self-improvement.