Repetition as Reinforcement: Enhancing Sample Efficiency via Instant Episode Repetition in Reinforcement Learning

AI summary

Overview

Research area: Reinforcement learning (RL) for continuous control, specifically sample efficiency and experience reuse in off-policy actor-critic methods (SAC and TD3).

Technical level: Intermediate. The core idea is simple, but the paper assumes familiarity with Markov Decision Processes, replay buffers, off-policy actor-critic algorithms, and self-imitation learning.

Scope: The paper introduces and empirically evaluates Instant Episode Repetition (IER), a mechanism that re-executes the action sequence of a new best-reward episode for a fixed number of subsequent episodes, tested across eight simulated continuous-control tasks and one real-world robotic manipulation task.

What This Paper Is About

Reinforcement learning agents typically need millions of environment interactions to learn, far more than biological learners. Existing techniques such as experience replay and Self-Imitation Learning (SIL) reuse past experience only passively — they update the policy from stored transitions but never change how the agent behaves in the environment. The paper's goal is to make reuse active: when the agent discovers a new highest-reward episode, it immediately replays that episode's full action sequence during data collection, so successful behaviour is reinforced at the moment it is found.

Key Contributions

1. Instant Episode Repetition (IER): A biologically inspired mechanism that modifies the RL interaction loop by immediately re-executing the action sequence from a prior high-reward episode, replacing policy-selected actions during that repetition phase.
2. Episode-level behavioural consolidation: Rather than treating experience as independent transition samples, IER repeats complete action sequences, shifting experience reuse from passive replay-buffer sampling toward active episode-level action replay while new transitions are still being collected.
3. General, architecture-free integration: IER is integrated into the off-policy continuous-control algorithms SAC and TD3 (yielding IER-SAC and IER-TD3) with no changes to network structures or loss functions.
4. Broad empirical evaluation: Comparisons against standard off-policy baselines and SIL-enhanced baselines on MuJoCo, the DeepMind Control Suite, and a real-world robotic manipulation task, under consistent training settings.

Main Findings

• IER-SAC beats SAC on most simulated tasks: IER-SAC improves over the SAC baseline in six out of eight tasks and performs comparably in the remaining two. The strongest gains are in dynamic locomotion tasks — Ant-v4, HalfCheetah-v4, Walker-Walk, and Cheetah-Run.
• IER-TD3 also improves, but by less: IER-TD3 outperforms standard TD3 in six out of eight tasks and exceeds SIL-TD3 in five tasks, with notable improvements in HalfCheetah-v4, Humanoid-v4, and Walker-Walk. Relative gains are generally smaller than for IER-SAC.
• SIL is inconsistent: SIL-SAC performs competitively in selected environments such as HalfCheetah-v4 and Walker-Walk, but is inconsistent on more unstable tasks such as Humanoid-v4 and Cartpole-Swingup, sometimes falling below the SAC baseline — attributed to the risk of reinforcing suboptimal trajectories when exploration is incomplete.
• Algorithm difference explained: SAC's entropy regularization promotes broader exploration and policy diversity, which the repetition mechanism complements. TD3's deterministic policy updates may be more sensitive to repeated behaviour sequences and more prone to premature convergence if repetition is excessive.
• Moderate repetition is best (RN sensitivity): Averaged jointly over SAC and TD3 across all eight tasks, ΔAUC% peaks at RN3 (10.66%), followed by RN2 and RN4, then declines at larger repetition levels. The unimodal pattern suggests small RN values under-reinforce good episodes while large RN values reduce state diversity.
• Task-specific RN trends: Finger-Turn-Hard and Walker-Walk show substantial gains at moderate or higher RN values; Walker-Walk and Cheetah-Run often improve with increasing RN and typically peak at RN3–RN5. Hopper-v4 and Cartpole-Swingup show smaller and more variable changes. For TD3, performance commonly peaks at RN2–RN3; SAC tends to benefit from smaller RN values.
• RN=0 is consistently weak: Across nearly all tasks, RN=0 is one of the weakest settings, confirming that learning without repetition underutilizes successful trajectories. Even RN=1 consistently improves performance. Performance occasionally declines at RN=7, likely from overexploitation or reduced experience diversity.
• Domain-level pattern: In MuJoCo locomotion environments, improvements are greatest at moderate repetition levels, particularly under TD3. In DeepMind Control tasks, repetition effects are smoother and less sensitive to RN magnitude.
• Cross-algorithm agreement: Using a 95% near-best criterion, intersection-based agreement was found for Ant-v4 (RN1), Cheetah-Run (RN3), Walker-Walk (RN5), and Cartpole-Swingup (RN2); rank-based compromise was needed for HalfCheetah-v4 (RN4), Hopper-v4 (RN2), Humanoid-v4 (RN3), and Finger-Turn-Hard (RN5). RN2, RN3, and RN5 appear most frequently, suggesting RN2–RN4 is a stable and effective operating range.
• Real-world task results: On the physical in-hand manipulation task, IER-SAC achieves faster convergence and higher final performance than both SAC and SIL-SAC. IER-TD3 also outperforms TD3 and SIL-TD3, though with narrower margins. SIL-TD3 shows a temporary early improvement but saturates quickly, allowing TD3 and IER-TD3 to surpass it after relatively few interactions.
• Repetition is not duplication: Because each episode starts from an initial state sampled from a reset distribution, re-executing the same stored action sequence produces a family of locally related but non-identical episodes. This increases sampling density around high-reward regions without duplicating identical transitions.

Methodology in Plain English

IER changes one thing in the standard RL loop: what actions the agent takes during data collection.

Normally, an off-policy agent (here, SAC or TD3) picks actions from its policy, stores every transition in a replay buffer, and periodically updates its networks from random batches of that buffer. IER adds an episode-boundary rule on top of this. When an episode ends, the agent computes the episode's total reward. If that reward is higher than any episode reward seen so far, it saves the entire sequence of actions from that episode and enters a repetition phase. For the next RN consecutive episodes, it ignores the policy and executes that stored action sequence step by step instead. RN is a hyperparameter controlling repetition strength. If no new best episode occurs, or once the repetition phase ends, the agent returns to sampling actions from its policy (with exploration noise).

Crucially, this alters only the interaction strategy, not the learning objective: every transition collected during repetition and exploitation still goes into the replay buffer and is used for the same standard off-policy updates. There are no changes to network architecture, reward function, loss, or optimizer.

The evaluation covers eight continuous-control tasks — Ant-v4, HalfCheetah-v4, Humanoid-v4, Hopper-v4 (MuJoCo, vector observations), and Walker-Walk, Cheetah-Run, Cartpole-Swingup, Finger-Turn-Hard (DeepMind Control Suite) — plus a real-world task where a custom-built gripper moves a cube laterally while keeping it suspended. That setup uses LED ring lighting and a camera tracking the cube's pose with ArUco markers, with rewards for maintaining a stable grasp and increasing lateral displacement. Baselines are SAC, TD3, SIL-SAC, and SIL-TD3. Simulated learning curves report mean episodic return over five independent seeds with shaded standard deviation; the RN ablation curves in the supplementary material use ten independent seeds and are smoothed with a sliding window of size 5. The paper notes that some baseline results are slightly lower than values in the original publications due to simulator stochasticity and seed variation, but relative differences remain consistent. Code is available at https://github.com/UoA-CARES/instant-episode-repetition.

Why This Matters

Impact on research: The paper argues a conceptual point about where experience reuse happens. Replay buffers and SIL reuse data after it has been stored; IER intervenes in how data is generated. This reframes repetition as an interaction-level mechanism rather than an update-level one, and shows it can be bolted onto existing actor-critic algorithms without architectural changes or external demonstrations. It also provides a tunable, interpretable hyperparameter (RN) whose optimal range is characterized across tasks and algorithms.

Real-world applications:

• Robotic manipulation where a stable, successful motion must be consolidated quickly — demonstrated here on in-hand cube translation with a gripper.
• Industrial or field robotics where each physical trial is expensive and slow, making sample efficiency directly tied to cost.
• Any continuous-control setting with a repeatable reset procedure (the paper's real-world setup uses an autonomous reset mechanism), where a good trajectory can be safely re-attempted.
• Sim-to-real pipelines where reducing the number of real interactions lowers the burden of simulation fidelity.

Industry relevance: The method requires no new networks, losses, or demonstration data, and only changes how the environment is driven during training. That makes it a low-integration-cost add-on to existing off-policy RL training stacks, and it transfers from simulated benchmarks (MuJoCo, DeepMind Control Suite) to a physical robotic system in the same paper.

Future Directions

• Adaptive RN selection: The results show the optimal RN varies by task, algorithm, and environment stability (TD3 peaks at RN2–RN3, SAC prefers smaller values). The paper treats RN as a tunable hyperparameter rather than a fixed setting, leaving open the question of how to set or schedule it automatically.
• Understanding the TD3/SAC asymmetry: Why IER gives smaller relative gains under TD3's deterministic policy updates, and whether repetition interacts badly with premature convergence, is offered as a likely explanation but not resolved.
• Repetition criteria beyond "new maximum reward": IER only triggers on a new highest episode reward, which may underutilize near-best episodes; alternative triggers are not explored.
• Scope of environments tested: Evaluation is limited to vector-based, low-dimensional observations (visual input is explicitly excluded) and to eight simulated tasks plus one real task, so generalization to high-dimensional visual control and to a wider set of real-world stochastic systems remains open.

Target Audience

Researchers and practitioners working on sample-efficient reinforcement learning, experience replay, and self-imitation learning, particularly those applying off-policy actor-critic methods to continuous control. It is also relevant to robotics engineers who need to train policies on physical hardware where interactions are costly, and to readers interested in biologically inspired mechanisms for RL. Readers without background in off-policy RL will need to consult the cited SAC, TD3, and SIL papers first; beginners may find the simulation and ablation sections accessible but the methodology section demanding.

Authors’ abstract

Repetition is a fundamental mechanism in human learning, where revisiting successful experiences strengthens memory, consolidates skills, and improves future performance. Motivated by this biological principle, we introduce Instant Episode Repetition (IER), a simple and novel mechanism that improves sample efficiency by immediately repeating action sequences from successful episodes during environment interaction. Unlike conventional approaches such as Experience Replay and Self-Imitation Learning (SIL), which passively reuse past experience during training updates, IER directly influences the data collection process. Upon identifying a high-reward episode, the agent repeats its action sequence for a fixed number of subsequent episodes, reinforcing valuable behaviors through renewed interaction with the environment. We integrate IER into state-of-the-art SAC and TD3 algorithms and evaluate its effectiveness on continuous-control benchmarks, including MuJoCo, the DeepMind Control Suite, and a real-world dynamic object translation task with a robotic manipulator. Experimental results demonstrate that this simple mechanism improves learning performance over standard and self-imitation-based baselines.