Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents

AI summary

Overview

• Research area: Natural Language Processing / post-training of small language models for interactive dialogue game agents (dialogue games as agentic benchmarks).
• Technical level: Intermediate (assumes familiarity with supervised fine-tuning, DPO, LoRA, and benchmark scoring, but the argument is presented narratively).
• Scope: A single-author case study of a post-training recipe — broad SFT, turn-local DPO repair, and training-free delta scaling — applied to a 2B open-weight model in the LM Playschool Challenge.

What This Paper Is About

Interactive dialogue games require a model to carry state across turns, interpret feedback, and choose valid actions as constraints accumulate. The author finds that a 2B model's failures in this setting are not only broad knowledge deficits but also local decision failures: repeated guesses, malformed actions, and violations of feedback the model has just received. The goal is to place supervision precisely where those decisions break down, without sacrificing the model's general static-benchmark capability.

Key Contributions

1. A diagnosis-guided three-part recipe — "Acquire, Repair, Preserve" — that separates broad game participation (SFT), repair of mechanically verifiable failures within one game family (turn-local preference pairs), and preservation of general capability (training-free delta scaling).
2. An empirical demonstration that supervision locality is the hinge. Turn-local preference pairs anchored to post-feedback decisions produced the clearest positive preference-tuning signal, while whole-dialogue DPO destroyed protocol compliance.
3. A training-free preservation method (delta scaling, W(s) = W_base + s · Δ_LoRA) that selects an operating point along the learned LoRA update subject to a paired interactive gate, reported as aggregate-level trade-off control rather than preservation of every capability.
4. A candid reporting of path-dependence and eval noise, including two nominal training stages that were found to have updated no weights at all, and sign-inconsistent phase marginals across two evaluation rails.

Main Findings

• Official endpoint: The submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, both above the official 4B (26.66 public) and 9B (31.91 public) baselines. Aggregate static performance is approximately preserved: 44.14 versus the baseline's 44.24.
• Scale comparison: Scaling the untrained official baseline from 2B to 9B raises public clemscore from 10.67 to 31.91 (+21.24) with no training at all.
• In-domain gain is two-sided: Public Played rises 27.22 → 78.63 and Quality 39.20 → 49.50; closed in-domain Played rises 47.32 → 82.62 and Quality 28.33 → 49.83.
• Out-of-domain transfer is weak and structurally different: OOD clemscore rises from 3.72 to 7.88, but OOD Played falls below the untrained base (35.19 → 33.08), so the gain reflects only higher conditional Quality (10.56 → 23.81) on fewer played episodes.
• Transfer is concentrated in related variants: Of 84.86 total positive out-of-domain movement, three held-out Wordle-crazy configurations contribute 70.02, or 82.5%; the remainder comes from TA Frozen Lake (+9.17), TA Mastermind (+5.00), and Cryptolect (+0.67).
• The largest hidden in-domain gains are in untargeted games: TMW GraphReasoning (+71), ImageGame (+60), and AdventureGame (+58), while the targeted Wordle family remains near zero on hidden in-domain episodes (3.3, 10.0, and 3.3).
• Acquisition dominates: Phase A (broad SFT) lifts the pinned public-development score from 13.05 to 43.85, the largest single step in a trajectory that continues through 46.83 and 45.52 (Phase B passes), 48.52 (Phase C), to a mean 50.43 after delta scaling.
• Locality matters: Unfiltered whole-dialogue DPO scored 12.35 and produced natural-language preambles violating game protocols across Wordle, Codenames, and other games; filtering the chosen side to stronger source models partially recovered the score to 25.76 but left the principal abort patterns intact.
• Phase marginals are sign-inconsistent: Only turn-DPO pass 1 is positive on both rails (+2.98 and +1.49); pass 2 is −1.31 / +1.34 and branch-DPO is +3.00 / −2.07. Branch-DPO reduces clem by 2.07 and raises stat by 1.23 in the controlled campaign.
• Preservation hides redistribution: BBH Quality rises from 0.00 to 30.47, whereas IFEval falls from 68.52 to 51.85 and EQ-Bench Quality from 64.78 to 56.43.
• Selection signal: Across the eleven submissions, public clemscore correlates strongly with closed in-domain score (r = 0.93) but only weakly with closed out-of-domain score (r = 0.30).
• Late continuation did not help: Five continuation arms from the selected s = 0.85 parent (solver corrections, success-versus-failure pairs, teacher corrections) all failed to exceed the 50.43 development endpoint, scoring between 44.15 and 48.92.
• Path dependence: On a matched 4B probe, low-dose SFT on 700 off-policy successful episodes reduced clemscore from 40.75 to 24.36; a separate 4B probe trained on 303 self-generated successful dialogues reached 35.06.
• Two nominal stages trained nothing: Nominal Stage 2's output adapter is byte-identical to the broad-SFT checkpoint (372 of 372 tensors equal, zero optimizer parameters); nominal Stage 3's GRPO run reported 16.8M trainable parameters and 54 non-zero gradient norms while every gradient was discarded and the KL term was identically zero across all 54 steps.

Methodology in Plain English

The author starts from a 2B open-weight vision-language model (Qwen3.5-2B), served and fine-tuned in text-only mode.

Acquire. Phase A trains the model by imitation on success-only episodes from the training split of playpen-data, capped at 700 examples per game (9,522 rows total; 9,045 after a 95/5 split). This teaches the surface form of game play and lifts the pinned development score from 13.05 to 43.85, but raw Wordle quality stays at zero: the model still repeats rejected guesses and violates feedback it just received.

Repair. Phase B targets Wordle specifically because its failures are frequent, local to a single decision, and mechanically checkable. Preference pairs are built by taking a clean successful episode, extracting the eligible assistant turn that follows guess_feedback:, and synthesizing a rejected completion via one of four deterministic corruptions: repeat_last, bad_length, invalid_word, or duplicate_guess_keyword. Because the dialogue history is held fixed, the DPO contrast lands on the few decision tokens that actually determine the failure. Pass 1 handles repeated guesses and bad length (1,905 pairs, β = 0.1); pass 2 handles invalid words and duplicated answer fields (1,906 pairs, β = 0.1). Phase C then addresses the current model's own residual errors: it plays 27 seeded Wordle instances with branching factor 2 at rounds 1–3 and temperature 0.7, scores the downstream episodes, and pairs the immediate diverging responses of the highest- and lowest-scoring siblings (77 DPO pairs at β = 0.20).

Preserve. By this stage the model pays a specialization tax on static benchmarks. Rather than adding replay data, the recipe treats preservation as weight-space model selection: scale the learned LoRA delta by a scalar s ∈ [0, 1] and pick the s that maximizes static score subject to passing an interactive gate on the pinned public-development set. The selected value is s = 0.85. All trainable phases use LoRA (r = 16, α = 32, all-linear, dropout 0.05), AdamW, bf16, and one epoch.

Measurement. Development decisions use a pinned public-development set (dataset revision 557d8caf, clembench commit ed39486, 14 games, 67 episodes) with paired same-host comparisons, because that sample is small and noisy. The official final evaluation (snapshot 2dd5a533) is reported as the primary result: a public suite of 502 episodes (72 interactive, 430 static), a closed in-domain suite of 1,272 episodes, and a closed out-of-domain suite of 360 episodes, all run at temperature 0 with a 5,000-token generation limit and thinking mode disabled. Clemscore multiplies the average Played percentage by the macro-average Quality over games with at least one played episode; applying the same aggregation to 430 static instances from BBH, CLadder, EQ-Bench, IFEval, and MMLU-Pro yields Statscore.

Why This Matters

The paper's value is less in the headline numbers than in the diagnostic framing: it argues that for small interactive agents, the question is not whether dialogue game data helps but which part of the trajectory should receive supervision. It also documents in unusual detail how fragile post-training pipelines can be, including two stages that silently optimized nothing and a 0.74-point evaluation-noise spread measured directly on identical weights.

Real-world applications:

• Local and on-device assistants constrained by memory, latency, or cost, where a 2B model is the practical deployment target.
• Post-training budget allocation for teams deciding between scaling model size and spending compute on targeted fine-tuning — the paper quantifies the trade with the 2B-to-9B comparison.
• Interactive tool-use and API-calling agents, where malformed or constraint-violating actions in multi-turn state are the dominant failure mode.
• Preference-data construction pipelines, where the paper's contrast between turn-local pairs and whole-dialogue pairs is directly actionable.

Industry relevance: The recipe trains in approximately 5.3 hours on a single A100-80GB GPU, and the preservation step requires no gradient computation at all. That makes the approach attractive for teams that cannot afford large-scale RL or frontier-model distillation. The negative result on out-of-domain transfer is equally relevant: public leaderboard progress was a strong predictor of hidden in-domain progress (r = 0.93) but a weak predictor of broad transfer (r = 0.30).

Future Directions

• Extend repair beyond mechanically verifiable failures. The paper states the method has not been tested on failures requiring semantic or strategic judgment, and that demonstrated coverage is restricted to the Wordle family.
• Vary the acquisition-stage parent. The study evaluates repair only after one specific, capped broad-SFT recipe and cannot establish whether turn-local repair complements a stronger imitation stage or compensates for weaknesses specific to the submitted parent.
• Test scale applicability. The submitted artifact uses a single Qwen3.5-2B base; transfer to larger or differently trained models remains untested.
• Replace aggregate preservation with component-wise constraints. Since Eq. 3 optimizes an aggregate static score subject to an aggregate interactive gate, gains in one component can offset losses in another; a stronger claim would require component-wise floors or a multi-objective selection criterion.
• Build diagnostics or selection signals that reach beyond the Wordle family and the public development distribution, which the author identifies as a prerequisite for broader transfer.

Target Audience

Researchers working on post-training and alignment for small language models, particularly those studying preference optimization and where to place supervision. Practitioners entering agentic or interactive benchmarks (clembench, Playpen, LM Playschool) who want a reproducible reference recipe and an honest account of its noise floor. Also useful for evaluation-focused readers, because the paper's treatment of sign-inconsistent phase marginals, byte-identical checkpoints, and development-set variance is a model of careful reporting about what a sequential training trajectory can and cannot establish.

Authors’ abstract

Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, and choose valid actions under changing constraints. We study this setting in the LM Playschool Challenge with a 2B open-weight model, and find that many failures are not only broad knowledge failures but also local decision failures: repeated guesses, malformed actions, and violations of feedback that the model has just seen. These diagnostics motivate a training recipe organized around three steps: acquire broad game participation through supervised fine-tuning, repair mechanically verifiable failures within one targeted dialogue-game family using turn-local preference pairs, and preserve general capabilities beyond these dialogue games. In the official final evaluation, our submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while approximately preserving aggregate static performance (44.14 vs. 44.24 for the baseline). Out-of-domain clemscore remains low at 7.88, with the largest gains concentrated in unseen variants of the targeted family. Our results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.