DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agents

AI summary

Overview

• Research area: Large language model (LLM) agents, long-horizon tool use, agent memory and context compression, preference optimization.
• Technical level: Intermediate. The paper assumes familiarity with LLM agents, tool calls, and preference training (DPO), but its central idea is simple enough to follow without a deep math background.
• Scope: The paper identifies a specific failure mode ("loops," where an agent re-issues an earlier tool call and receives the same observation), shows that context compression makes it worse, and proposes DeLoop — a self-supervised training method that turns an agent's own loops into preference-training data.

What This Paper Is About

Long-running LLM agents often redo work: they call the same tool with the same arguments, get the same observation back, and learn nothing new. The paper shows this happens even with full history, becomes far more common once the agent's history is compressed to fit a context window, and is strongly associated with task failure. The goal is to fix the agent's policy itself — not just the memory system — by training it to prefer the recovery action it eventually took over the looping action it repeated.

Key Contributions

1. Defining and characterizing loops. The authors formalize a loop as a step whose canonicalized tool call and byte-identical observation both appeared earlier in the trajectory. They show loops occur with full history, grow sharply under compression, and correlate with failure.
2. Mining loops as free supervision. They introduce a rule that turns a looping action into a rejected response and the first later non-looping action in the same rollout into a chosen response — requiring no task rewards, gold answers, step labels, or LLM judges.
3. Training the policy with DPO on these pairs. DeLoop trains with direct preference optimization (LoRA DPO, β = 0.05, one epoch, 8,000 self-mined CodeGym pairs for Qwen3-8B), plus a DeLoop-Mix variant that adds 4,635 cross-domain pairs for 12,635 total.
4. Broad evaluation. Results span CodeGym, ALFWorld, Open-AgentRL, AppWorld, ScienceWorld, Terminal-Bench, and BrowseComp-Plus, across Qwen3-8B/14B/32B, Qwen3.5-9B, and multiple Gemini models.

Main Findings

• Compression drives loops. On CodeGym, Qwen3-8B's loop rate rises from 29.0% with full history to 57.3% under summarization at 4,096 tokens, and to 64.7% at 2,048 tokens. On Gemini 3.1 Pro the increase is 2.9×.
• Loops are costly. Looping Qwen3-8B runs contain 6.5 looping actions on average, execute 10.5 more actions per task, and use 11k more transcript tokens than loop-free runs.
• Loops predict failure. Averaged across six Qwen3 and Gemini models, 59.5% of looping runs fail versus 33.4% of loop-free runs. Qwen3-8B fails 85% of looping runs versus 68% loop-free; Gemini 3.1 Pro fails 48% versus 12%.
• Main CodeGym result. DeLoop cuts Qwen3-8B's loop rate from 57.3% to 6.1% (a 9.4× reduction), lowers steps per task from 15.00 to 4.67, cuts tokens per task by 47%, and holds task success (25.1% to 25.9%).
• The gap is large against alternatives. Training-free methods stay between 42.7% and 65.3% loop rate; the strongest labeled baseline (gold-label DPO) reaches 22.3%.
• Larger models gain in success, not just cost. Qwen3-14B: loops 42.9% to 12.3%, success 45.1% to 52.3% (+7.2 points). Qwen3-32B: loops 38.5% to 12.9%, success 37.7% to 44.8% (+7.1 points). Qwen3.5-9B: loops 54.7% to 26.0%, success 49.7% to 55.3%.
• Both halves of the pair matter. Supervised fine-tuning on the chosen action alone leaves a 37.7% loop rate, versus 6.1% with the DPO contrast against the looping action.
• DPO beats other objectives on the same signal. KTO reaches 12.6%, IPO 11.7%, and an online GRPO looping-action penalty 15.0%, against 6.1% for DPO.
• Cross-domain pairs help at 8B. DeLoop-Mix raises success from 25.1% to 30.3% under compression and from 31.7% to 36.2% with full history, with the loop rate at 10.1% in both settings.
• The policy often ignores records it has. With the action record present, Qwen3-8B re-issues the call in 3.3% of samples; with the record deleted, 48.8%. Yet 25.2% of compressed runs repeat a call still present verbatim in context, and action-ledger summaries still loop 62.7% of the time.
• Blockers do not fix it. With an exact duplicate blocker returning earlier results, Qwen3-8B still proposes an already executed call in 71.3% of tasks (69.8% without the blocker); DeLoop lowers this to 13.6%.
• Generalization. The loop rate falls in all five context settings tested (4.7% to 12.8%) and under all seven alternative memory policies. Loops drop in all four additional environments, by up to 32.8 points on AppWorld. On Terminal-Bench, unproductive duplicate calls fall from 16.6 to 6.9 per task; on all 830 BrowseComp-Plus queries, DeLoop-Mix lowers the loop rate from 22.4% to 1.2% while matching accuracy.
• No capability regression reported. Performance is preserved on LiveCodeBench, MT-Bench, and IFEval.

Methodology in Plain English

The approach has three stages.

First, detect. Each tool call is normalized so formatting differences do not count, then paired with its observation. If that exact call-observation pair appeared earlier in the episode, the step is a looping action — it returned information the agent already had.

Second, mine. For each looping action, the paper looks forward in the same rollout for the first action that is not itself a loop and differs from the looping call. That later action becomes the "preferred" example and the repeated call becomes the "rejected" one. Both are paired with the same preceding history. If the agent never recovers, no pair is built. Importantly, a pair is usable even if the rollout ultimately fails the task — the recovery is identified from actions and observations alone.

Third, train. Direct preference optimization nudges the policy to raise the probability of the recovery action and lower the probability of the loop, relative to a frozen reference model. At inference, the trained model both acts and writes its own summaries when history exceeds 4,096 tokens; only the action policy is trained, summary writing has no separate objective.

Evaluation uses 300 held-out CodeGym tasks with greedy decoding and a 30-round budget, measuring loop rate, task success, steps per task, and estimated tokens per task.

Why This Matters

The work reframes a common agent failure as a free training signal: no human labels, no reward model, no stronger teacher. It also separates two questions the field usually conflates — what history to keep, and whether the policy actually uses what it keeps. The paper shows that better compression alone does not solve looping, which redirects effort toward policy training.

Real-world applications:

• Coding and terminal agents that inspect repositories and run commands over many steps, where repeated calls burn budget and stall progress.
• Web research agents that re-run identical searches instead of advancing, as measured on BrowseComp-Plus.
• Enterprise tool-use assistants (ticketing, CRM, workflow automation) where each redundant tool call is a real latency and cost hit.
• Embodied and simulated agents (ALFWorld, ScienceWorld) that must explore without re-treading visited states.

Industry relevance: the method reduces steps per task to roughly one third on Qwen3-8B and cuts tokens per task by 47%, directly lowering inference cost and latency for deployed agents. Because it needs only the agent's own rollouts, it fits into existing post-training pipelines without annotation spend.

Future Directions

• Why do agents ignore records they already hold? With the record verbatim, the baseline still re-issues 15.3% of the time and DeLoop 3.4%, so looping is not purely a memory problem; the residual failure is unexplained.
• Beyond exact repeats. The detector only catches identical call-observation pairs; near-duplicate calls, semantically equivalent calls, and repetitive reasoning are outside its scope.
• Combining with online RL and better compressors. A per-step loop penalty in GRPO reached 15.0% versus 6.1% for DPO, and the authors position DeLoop as complementary to compression research rather than a replacement.
• Scaling and transfer. Zero-shot transfer from CodeGym to ALFWorld on Qwen3-32B lowered loops from 64.9% to 54.5% and raised success from 33.6% to 42.5% — meaningful but far from closed, leaving open how much cross-domain and cross-harness generalization is achievable.

Target Audience

Researchers and engineers working on LLM agents, agent post-training, and context management. It is most useful to practitioners who deploy long-horizon tool-using agents and need to control cost and failure rates, and to those studying self-supervised or preference-based training where labeled data is unavailable. Readers without agent or DPO background will still follow the core argument but may need to consult the cited preference-optimization work for the training details.

Authors’ abstract

Language-model agents that run long tasks often redo work they have already done: they re-issue a tool call they made earlier and get back exactly the same observation, a step that adds no new information. We identify this failure as a loop. Loops appear natively in Qwen, Llama, and Gemini agents and become far more common once the agent's history is compressed to fit its context window. To reduce these loops, we propose DeLoop, a self-supervised training method. DeLoop builds on a simple observation: the agent's own rollout can provide the supervision needed to learn from its loops. We use a looping action as a negative example and the first later non-looping action in the same rollout as a positive example. DeLoop learns from these pairs through direct preference optimization and does not require task rewards, gold answers, or LLM judges. On CodeGym under compression, DeLoop reduces the share of Qwen3-8B tasks containing loops from 57.3% to 6.1%. It also uses about one third as many steps per task, with similar task success. On Qwen3-32B, DeLoop reduces loops and steps while raising task success from 37.7% to 44.8%.