Periodic Skill Discovery

AI summary

Overview

• Research area: Unsupervised reinforcement learning (RL), specifically skill discovery — learning diverse behaviors without external reward — with a focus on temporally periodic (rhythmic) behavior.
• Technical level: Advanced. The paper assumes familiarity with Markov decision processes, mutual information objectives, constrained optimization with Lagrange multipliers, and standard RL algorithms (SAC, PPO).
• Scope: The paper proposes Periodic Skill Discovery (PSD), a framework that learns periodic skills of controllable period by mapping states into a circular latent space, evaluated on MuJoCo locomotion tasks in both state and pixel domains.

What This Paper Is About

Existing unsupervised skill discovery methods reward agents for visiting diverse states — either by maximizing mutual information between skills and states, or by maximizing how far the agent travels in a learned latent space. This biases them toward learning where to go, not how to move, so they largely ignore the periodic structure that underlies most locomotion. PSD instead trains an encoder that arranges states on a circle in latent space, where the circle's diameter encodes how long a movement cycle takes, so that simply moving along that circle produces a periodic behavior. This lets a single, simple reward signal produce skills at many different timescales.

Key Contributions

1. A new skill discovery objective (PSD) that maps states to a circular latent space in which the period variable L determines the circle's diameter, so that following a larger circle corresponds to a longer period. The paper proves (Theorem 1) that an optimal encoder forms a regular 2L-gon of diameter L centered at the origin.
2. A single-step intrinsic reward derived from that geometry: r_PSD(s_t, s_{t+1}, L) = exp(−κ Δ²), where Δ is the deviation of the one-step latent transition from the optimal step length L sin(π/2L). This avoids needing whole-rollout objectives tailored to each period.
3. An adaptive sampling method that dynamically expands or shrinks the range [L_min, L_max] of period variables during training based on how well the policy follows the circular path, so the agent discovers its own feasible range of periods without prior knowledge.
4. Demonstrations that PSD scales to pixel observations and composes with an existing method. Because it encodes temporal distance rather than a specific state representation, PSD works from raw pixels, and summing its reward with that of METRA yields a policy π(a | s, z, L) that independently controls movement direction (z) and period (L) in a fully unsupervised manner.

Main Findings

• Circular latent space forms as intended: With 1k transitions sampled from the replay buffer in the Ant environment, both the 1-step distance ‖φ_L(s_t) − φ_L(s_{t+1})‖ and the L-step distance ‖φ_L(s_t) − φ_L(s_{t+L})‖ converged close to their theoretical optima, L sin(π/2L) and L respectively, with a small relative error.
• Periods are controllable: In Ant with L = 20, the learned behavior completed approximately five full cycles of period 2L = 40 over 200 timesteps. When the period variable L was switched mid-episode at fixed intervals for Ant and Walker2D, the policy's behavior changed to match the new period of 2L.
• Broader frequency coverage than baselines: Applying a Fourier transform to 16 uniformly sampled skills per method and extracting the four highest-amplitude frequencies, PSD consistently discovered skills across a wide range of frequencies. METRA and CSD (distance-maximizing methods) concentrated on narrow frequency bands and often produced inconsistent, indistinguishable behaviors in Hopper and Walker2D, while MI-based methods often produced static or partially random behaviors.
• Better downstream task performance (Table 1): A high-level PPO policy was trained with frozen skills, averaged over 10 seeds. PSD led on all four reported tasks:
  • HalfCheetah-hurdle: PSD 3.8 ± 2.0 vs. METRA 1.9 ± 0.8, DADS 0.9 ± 0.3, CSD 0.8 ± 0.6, DIAYN 0.6 ± 0.5.
  • Walker2D-hurdle: PSD 5.4 ± 1.4 vs. CSD 4.1 ± 1.3, METRA 3.1 ± 0.5, DIAYN 2.6 ± 0.5, DADS 1.9 ± 0.3.
  • HalfCheetah-friction: PSD 43.4 ± 19.1 vs. METRA 30.1 ± 13.1, DIAYN 13.2 ± 3.4, CSD 12.5 ± 3.8, DADS 12.4 ± 2.9.
  • Walker2D-friction: PSD 8.7 ± 1.7 vs. CSD 5.3 ± 0.3, METRA 5.2 ± 1.6, DIAYN 4.6 ± 1.2, DADS 1.6 ± 0.1.
• Works from pixels: PSD learned periodic behaviors in pixel-based Ant and HalfCheetah environments without any modification to its objective or reward formulation.
• Composes with METRA: Combining PSD's reward with METRA's allowed the agent to modulate both movement direction (z) and period (L), producing a more diverse behavioral repertoire than either method alone. The paper characterizes METRA as adjusting the temporal direction of skills and PSD as modulating their temporal length.

Methodology in Plain English

The agent plays episodes with no external reward. At the start of each episode, a "period variable" L is drawn uniformly from a range [L_min, L_max] and held fixed; the policy is conditioned on it, and the target behavior is a cycle lasting 2L timesteps.

Alongside the policy, an encoder maps each state (and its L) into a latent vector. Two constraints shape that latent space: states L steps apart must be at most L apart in the latent space, and consecutive states must be at most L sin(π/2L) apart. Subject to those limits, the objective pushes L-apart states as far apart as possible (so they land on opposite sides of a circle of diameter L) while pulling their midpoint toward the origin (so circles of different diameters share a center). The paper proves this yields a regular 2L-gon: evenly spaced points around a circle, returning to the start after 2L steps.

Because the geometry is fixed, the policy can be steered with a one-step reward: it is rewarded when its single latent step matches the ideal arc length L sin(π/2L), which keeps it moving evenly around the circle and therefore repeating after 2L steps. The encoder and policy are trained jointly, with the policy optimized by SAC and the constraints handled by dual gradient descent with Lagrange multipliers. The period range itself adapts: the method measures the average total intrinsic reward at the current L_min and L_max, and expands the range when the policy handles those boundaries well and contracts it when it does not.

For evaluation, the authors compare against DIAYN, DADS, CSD, and METRA on five MuJoCo locomotion tasks (Ant, HalfCheetah, Humanoid, Hopper, Walker2D), with pixel-based experiments in Ant and HalfCheetah. Downstream usefulness is tested by freezing the skills and training a PPO high-level policy to pick skills for hurdle-jumping and variable-friction terrain; an external velocity-based reward is added and shared with all baselines for fairness.

Why This Matters

The paper reframes unsupervised skill discovery around timing rather than just spatial coverage, giving a principled way to produce rhythmic behaviors at many timescales without hand-designed gait schedules, phase variables, or offline motion data.

Real-world applications:

• Legged robot locomotion, where the paper notes that conventional approaches rely on foot contact schedules, central pattern generators, or manually tuned reward functions and foot trajectories.
• Gait and rehabilitation robotics, where rhythmic movement must be modulated across timescales for energy-efficient and adaptive motion.
• Autonomous navigation over irregular terrain, such as the hurdle and variable-friction scenarios the paper evaluates, which require switching between behaviors of different rhythms.
• Learning from camera input on physical platforms, since PSD's reward depends on temporal distance rather than a specific state representation.

Industry relevance: The method is fully unsupervised in its core formulation and can be extended to remain fully unsupervised when combined with METRA, which reduces dependence on hand-engineered reward functions — a major cost in robotics deployment. Its reported scaling to pixel observations is directly relevant to companies that train policies from cameras rather than proprioceptive state, and its compatibility with existing skill discovery objectives means it can be added to existing pipelines rather than replacing them.

Future Directions

• Extending beyond periodic tasks, such as robotic manipulation, by generalizing the latent geometry past circular structures — the authors note PSD may underperform where large persistent external disturbances make periodic behavior infeasible.
• Integrating frequency-domain analysis directly into training, for example Fourier representations, to better capture temporal patterns.
• Broadening domain coverage, since the experiments focus on locomotion tasks chosen to showcase multi-timescale behavior, while the framework is claimed to apply to any domain with periodic structure.
• Further work on the METRA combination, which the paper presents as a way to jointly modulate temporal direction and period but describes only as a validation of the combination idea.

Target Audience

RL researchers and graduate students working on unsupervised skill discovery, intrinsic motivation, or representation learning; robotics researchers interested in gait generation and legged locomotion; and practitioners who train policies from high-dimensional observations such as pixels and want temporally structured behavior without hand-designed rewards. Readers unfamiliar with constrained optimization or mutual information objectives will find the appendices and equations challenging, though the core circular-latent idea is intuitive.

Authors’ abstract

Unsupervised skill discovery in reinforcement learning (RL) aims to learn diverse behaviors without relying on external rewards. However, current methods often overlook the periodic nature of learned skills, focusing instead on increasing the mutual dependence between states and skills or maximizing the distance traveled in latent space. Considering that many robotic tasks - particularly those involving locomotion - require periodic behaviors across varying timescales, the ability to discover diverse periodic skills is essential. Motivated by this, we propose Periodic Skill Discovery (PSD), a framework that discovers periodic behaviors in an unsupervised manner. The key idea of PSD is to train an encoder that maps states to a circular latent space, thereby naturally encoding periodicity in the latent representation. By capturing temporal distance, PSD can effectively learn skills with diverse periods in complex robotic tasks, even with pixel-based observations. We further show that these learned skills achieve high performance on downstream tasks such as hurdling. Moreover, integrating PSD with an existing skill discovery method offers more diverse behaviors, thus broadening the agent's repertoire. Our code and demos are available at https://jonghaepark.github.io/psd/