Learning Parameterized Skills from Demonstrations

AI summary

Overview

• Research area: Reinforcement learning / learning from demonstration, specifically skill discovery and hierarchical policy learning for robot manipulation.
• Technical level: Intermediate. The paper assumes familiarity with the options framework, behavior cloning, variational inference, and KL divergence, though the high-level ideas are explained accessibly.
• Scope: The paper introduces DEPS (Discovery of GEneralizable Parameterized Skills), an end-to-end algorithm that discovers discrete skills modulated by continuous parameters from multitask expert demonstrations, and evaluates its ability to generalize to unseen tasks on the LIBERO and MetaWorld-v2 benchmarks.

What This Paper Is About

Learning from demonstration typically produces either a single flat policy that maps state and task to an action, or a library of skills that are purely discrete or purely continuous. Discrete-only skills are structured but inflexible, while continuous-only skills are flexible but unstructured and hard to interpret. This paper proposes learning parameterized skills — discrete skills that take continuous arguments (the authors' example is slice_fruit(x, y, z)) — so that a single skill compactly represents a family of behaviors and can adapt to new situations.

Key Contributions

1. An end-to-end algorithm (DEPS) for discovering parameterized skills from demonstrations. DEPS jointly trains a three-level hierarchy: a discrete skill policy, a continuous parameter policy conditioned on the chosen discrete skill, and a low-level subpolicy conditioned on both, using temporal variational inference on a derived lower bound.
2. Information-theoretic and architectural fixes for degeneracy. The paper identifies that a standard implementation of this hierarchy is under-specified and admits degenerate solutions that minimize behavior cloning loss without meaningful abstractions, and addresses this with an information bottleneck, prediction of continuous parameters per discrete skill rather than per timestep, a Skill Parameter Norm Penalty, and an information asymmetry between high-level policies and the subpolicy.
3. A view of skills as parameterized trajectory manifolds with aggressive state compression. The subpolicy receives only a compressed one-dimensional state, defined as s'_t = tanh(w_(k_t, z_t) · s_t^proj + b_(k_t, z_t)), forcing it to rely on the discrete skill and continuous parameter for skill-specific guidance.
4. Empirical demonstration of generalization and interpretability. DEPS outperforms multitask and skill-learning baselines on LIBERO and MetaWorld-v2, and produces interpretable skills such as a grasping skill whose continuous arguments define the grasp location.

Main Findings

• Best across all reported evaluation sets. In Table 1 (averaged across 5 seeds), DEPS achieves the highest mean success and mean highest success on LIBERO-OOD, LIBERO-10, LIBERO-3-shot, MW-Vanilla, and MW-PRISE.
• Strong out-of-distribution generalization. On LIBERO-OOD, DEPS reaches 0.34 ± 0.08 mean success and 0.66 ± 0.12 mean highest success, versus BC at 0.15 ± 0.04 and 0.36 ± 0.08, and PRISE at 0.10 ± 0.09 and 0.27 ± 0.23. The paper states this is more than double the standard BC result and more than triple PRISE.
• Advantage in low-data regimes. On LIBERO-3-shot, DEPS achieves 0.26 ± 0.03 mean success and 0.49 ± 0.03 mean highest success, compared to PRISE (0.07 ± 0.07 and 0.19 ± 0.14) and BC (0.11 ± 0.05 and 0.22 ± 0.08).
• The MetaWorld-v2 results follow the same pattern. On MW-Vanilla, DEPS reaches 0.45 ± 0.03 / 0.65 ± 0.03; on MW-PRISE, 0.32 ± 0.03 / 0.53 ± 0.03 — both the highest reported.
• Pretraining matters substantially. The BC-Untrained baseline scores 0.01 ± 0.00 mean success on LIBERO-OOD and 0.00 ± 0.00 on LIBERO-10, which the paper reads as underscoring the importance of pretraining for adaptation in complex environments.
• Robust to smaller pretraining budgets. In Table 2 (50-shot finetuning on LIBERO-OOD, averaged across 3 seeds), DEPS leads at 5, 10, and 15 pretraining epochs. At 5 epochs DEPS gets 0.24 ± 0.08 / 0.64 ± 0.09 versus BC at 0.07 ± 0.03 / 0.30 ± 0.08. The paper notes the margin over baselines increases with limited pretraining, suggesting the method may improve pretraining data efficiency.
• Consistent across finetuning budgets. Figure 6 shows DEPS outperforming baselines irrespective of the number of finetuning gradient steps on LIBERO-OOD and MW-Vanilla.
• Interpretable skill structure. Learned discrete skills correspond to primitive behaviors such as grasping, moving, and releasing objects, applied consistently across visually diverse tasks; less frequent subtasks such as closing drawers and microwaves get their own discrete skills.
• Continuous parameters encode skill-relevant, not task-relevant, information. Varying the continuous parameters of a single grasping skill smoothly varies the resulting grasp location, and there is high overlap in the continuous parameters used across tasks.
• The compressed state behaves like an index. Figure 8 shows the one-dimensional compressed state changing monotonically at roughly uniform rates within a discrete skill, with sudden changes when the selected discrete skill and continuous parameter switch.
• Compression is essential. Appendix results indicate that compression to a one-dimensional state is essential to DEPS' performance, that changing the limit on the maximum number of discrete skills considerably improves performance, and that learning only discrete or only continuous skills does not replicate DEPS' performance.

Methodology in Plain English

The authors start from the assumption that a long task can be broken into a sequence of discrete skills, each modulated by a continuous parameter vector. They write down the joint likelihood of a demonstrated trajectory together with its (unobserved) skill and parameter sequence, then use temporal variational inference to derive a tractable lower bound on the log-likelihood of the demonstration data. A bidirectional GRU "variational network" sees entire trajectories and infers posterior distributions over skills and parameters; the discrete policy, continuous policy, and subpolicy are trained against that posterior.

To keep this from collapsing into a trivial solution, several design choices are made. Continuous parameters in the variational network are predicted once per discrete skill rather than once per timestep, so they cannot encode the action at each step, and a Skill Parameter Norm Penalty discourages large-magnitude parameters. The low-level subpolicy never sees the full image observation — only the robot's proprioceptive state, compressed to a single scalar through a skill-and-parameter-specific linear projection followed by tanh. This bottleneck forces the skill and parameter variables to carry the information needed to choose actions, and it makes the subpolicy's inputs overlap more across tasks. High-level policies, by contrast, see rich observations and the task description.

During deployment, the agent samples a discrete skill, then a continuous parameter, then an action. The continuous policy refines the parameter at every timestep from the most recent observation, while the variational network's per-skill parameterization is used only during training.

Why This Matters

• Impact on research. The paper reframes parameterized skills as trajectory manifolds and argues that aggressive state compression, not just a better training objective, is what prevents degenerate skill discovery. It also reports that a state-of-the-art skill-learning baseline (PRISE) performs poorly under rapid generalization, suggesting prior evaluations with more extensive finetuning may not reflect transfer ability.
• Robotics manipulation. A single grasping skill whose continuous arguments specify grasp location could cover many objects without a separate policy per object.
• Low-data deployment. The advantage on LIBERO-3-shot (3 demonstrations per task) points toward adapting robots in settings where collecting expert data is expensive or slow.
• Factory and warehouse automation. The paper's pretrain-then-finetune setup, with 500 gradient steps of finetuning, matches scenarios where a base skill library is trained once and adapted to new bins, parts, or workcells.
• Domestic and assistive robots. The learned skills for grasping, moving, releasing, and closing drawers and microwaves are exactly the primitives a household robot would need to recombine in unfamiliar kitchens.
• Industry relevance. Because DEPS learns from offline demonstration datasets and pretrains on a fixed set of tasks offline, it fits the common industry pipeline of pretraining on logged robot data and shipping a checkpoint that customers or integrators finetune with small amounts of task-specific data.

Future Directions

• Hyperparameter sweeps over the skill library size. The paper reports that changing the limit on the maximum number of discrete skills considerably improves performance, leaving open how much headroom remains with tuning.
• Scaling the pretraining set. The authors motivate the work with the gap between Atari policies needing over 10 million samples and humans reaching effective gameplay in 20 episodes; whether DEPS-style compression closes more of that gap with larger demonstration corpora is untested here.
• Removing remaining architectural assumptions. The derivation assumes (k_t, z_t) is independent of earlier skill/parameter pairs given prior states and actions, and the implementation drops action history from the high-level policies; the practical limits of both simplifications are not characterized.
• Limitations in general. The paper states that limitations are discussed in Appendix J, but that content is not included in the provided material, so specific limitations are not reported here.

Target Audience

Researchers and practitioners working on imitation learning, hierarchical reinforcement learning, and robot manipulation, particularly those interested in skill discovery, variational methods for sequential decision-making, or pretraining-and-finetuning pipelines for robots. Readers should be comfortable with behavior cloning and variational inference; the paper connects to the options framework and to parameterized-action MDP literature.

Authors’ abstract

We present DEPS, an end-to-end algorithm for discovering parameterized skills from expert demonstrations. Our method learns parameterized skill policies jointly with a meta-policy that selects the appropriate discrete skill and continuous parameters at each timestep. Using a combination of temporal variational inference and information-theoretic regularization methods, we address the challenge of degeneracy common in latent variable models, ensuring that the learned skills are temporally extended, semantically meaningful, and adaptable. We empirically show that learning parameterized skills from multitask expert demonstrations significantly improves generalization to unseen tasks. Our method outperforms multitask as well as skill learning baselines on both LIBERO and MetaWorld benchmarks. We also demonstrate that DEPS discovers interpretable parameterized skills, such as an object grasping skill whose continuous arguments define the grasp location.