SLAP: Shortcut Learning for Abstract Planning

AI summary

Overview

Research area: Robotics — task and motion planning (TAMP), hierarchical reinforcement learning, and robot skill discovery.

Technical level: Advanced. The paper assumes familiarity with Markov decision processes, options/skills, PDDL-style operators, abstract state spaces, and policy-gradient RL.

Scope: The paper introduces Shortcut Learning for Abstract Planning (SLAP), a method that uses model-free RL to learn new low-level "shortcut" options inside the abstract planning graph induced by an existing set of TAMP skills, yielding shorter plans than pure planning and higher success rates than flat or hierarchical RL in four simulated robotic environments.

What This Paper Is About

Classical task and motion planning (TAMP) systems rely on manually programmed skills such as pick, place, and move, which limits robots to behaviors their human engineers already knew how to write and often produces long, wasteful plans. The goal of this work is to let an agent automatically discover new, physically dynamic behaviors — such as slapping or wiping obstacles aside — that shorten the plans produced by an existing abstract planner, without any additional assumptions, inputs, or hand-designed skills.

Key Contributions

1. The SLAP method. SLAP identifies promising shortcut connections between pairs of abstract states in the planning graph induced by existing options, then trains a separate RL policy (PPO) for each shortcut in a self-contained goal-based MDP, and finally adds the learned policies to the option set at evaluation so the planner can select them when they shorten plans.

2. A pruning mechanism for shortcut candidates. Because the number of potential shortcuts is O(|S|²), SLAP runs a small number of random rollouts from sampled initial states for each candidate shortcut MDP and prunes any shortcut whose terminal abstract state is reached in fewer than a threshold number of rollouts, based on the intuition that RL needs some initial success to bootstrap.

3. Object generalization machinery. Each state is treated as a set of objects with feature vectors, and each abstract state as a set of relational atoms. SLAP computes the relevant atoms (add and delete effects) and relevant objects for a shortcut, trains the policy on a projection of the state containing only those objects, and at test time matches shortcuts to new tasks through a type-preserving, injective object substitution.

4. Empirical evaluation in four simulated robotic environments (Obstacle 2D, Obstacle Tower, Cluttered Drawer, Cleanup Table), with comparisons against Pure Planning, Pure RL with PPO, Pure RL with SAC+HER, a Hierarchical RL (PPO) baseline, and the state-of-the-art hierarchical RL method SOL. The authors state that this is the first method that learns low-level skills for improving the execution time of an abstract planner.

Main Findings

• SLAP reduces plan length substantially across all four environments. Relative to Pure Planning, SLAP achieved reductions of 32% ± 7% in Obstacle 2D (17.6 ± 1.5 steps versus 25.9 ± 1.7), 68% ± 2% in Obstacle Tower (79.2 ± 3.2 versus 245.8 ± 10.4), 53% ± 14% in Cluttered Drawer (165.8 ± 43.6 versus 352.1 ± 49.5), and 73% ± 4% in Cleanup Table (115.2 ± 12.3 versus 431.8 ± 33.1). The paper's abstract summarizes this as reducing overall plan lengths by over 50%.

• SLAP reached 100% ± 0% success in every environment, while flat RL baselines reached 0% everywhere. Pure RL (PPO) and Pure RL (SAC+HER) recorded 0% ± 0% success in all four environments, hitting the maximum trajectory length (100.0 ± 0.0 in Obstacle 2D, 500.0 ± 0.0 elsewhere).

• Hierarchical RL and SOL only succeeded in the easiest environment. Hierarchical RL (PPO) and the adapted SOL method both succeeded only in Obstacle 2D (100% ± 0%), with plan lengths of 25.3 ± 1.8 (a 2% ± 9% reduction) and 24.9 ± 1.2 (a 4% ± 8% reduction) respectively — far smaller reductions than SLAP's 32% ± 7% in the same environment. Both failed (0% success) in Obstacle Tower, Cluttered Drawer, and Cleanup Table. The authors attribute this to high-level controllers struggling with the large number of grounded skills in manipulation tasks, noting SOL needs to deal with 216 grounded skills in Obstacle Tower, versus 2–3 option policies in the NetHack Learning Environment it was evaluated on in prior work.

• The number of discovered shortcuts grows with training and plan lengths fall correspondingly. After random rollout pruning, SLAP identified 11 shortcuts in Obstacle 2D, 92 in Obstacle Tower, 74 in Cluttered Drawer, and 54 in Cleanup Table. Average plan lengths continued to decrease toward the end of the 500,000 training steps as shortcut policies became more stable, with marginal benefit varying by environment complexity.

• SLAP generalizes to new object configurations and altered dynamics. Trained in Obstacle Tower with three stacked obstacles (each with mass 0.5 kg and friction coefficient 0.9), SLAP maintained short plan lengths when evaluated with varying numbers of stacked obstacles and distractor objects and with mass and friction doubled, whereas Pure Planning scaled poorly with each additional obstacle. The learned "slap" policy treats only a subset of obstacles as relevant yet physically affects the entire tower.

• Independent shortcut policies outperformed shared-policy schemes. In a comparison of three shortcut policy learning schemes, the default Independent scheme consistently produced shorter plans than Abstract Subgoals and Abstract HER, especially in the PyBullet environments, despite the latter two sharing representations across shortcuts. The authors speculate that universal policy learning implicitly devotes resources to learning infeasible shortcuts.

• Compute footprint varied by environment. Shortcut learning used (4.1 ± 1.2) × 10⁶ environment interactions and roughly 1.5 minutes of RL time in Obstacle 2D, (4.8 ± 0.6) × 10⁷ interactions and roughly 9 hours in Obstacle Tower, (3.3 ± 0.5) × 10⁷ interactions and roughly 6 hours in Cluttered Drawer, and (3.2 ± 0.6) × 10⁷ interactions and roughly 8 hours in Cleanup Table, all on a single H100 GPU with 4 CPU cores.

• SLAP spans a spectrum between planning and RL. If the given options are already optimal or shortcut learning is too hard, SLAP reduces to pure planning; if the environment is simple enough that a shortcut policy can go straight from the initial state to the goal, the plan collapses into a single shortcut and SLAP reduces to pure RL.

• Reported evaluation protocol. Training sampled 10 tasks per environment; evaluation sampled 10 held-out tasks per environment and measured success rate and plan length. RL policies were trained for 500,000 steps using stable-baselines3. Note that the Table 1 caption states averages over 10 random seeds, while the experimental details section states that all reported metrics are averaged over 5 random seeds with standard deviations.

Methodology in Plain English

The researchers start from an abstract planner they deliberately keep simple — no heuristics or continuous skill optimization — so that any gains come from shortcut learning rather than better planning. The planner builds a two-level abstract planning graph: the top level has abstract states as nodes and predefined options (skills) as edges, while the bottom level has environment states and actions, with the levels linked because executing a top-level option corresponds to a bottom-level trajectory. Searching this graph with a shortest-path algorithm gives an execution-time-minimizing solution using only the original skills.

To improve on that, SLAP looks for pairs of abstract states that no existing option connects directly, and treats each such pair as a candidate "shortcut." Cheap random rollouts from states collected in the training graphs prune candidates that are essentially unreachable, which keeps the number of learning problems manageable and gives RL some initial success to build on. Each surviving shortcut becomes its own small MDP with a reward that penalizes every step by −1 and terminates when the target abstract state is reached; SLAP trains these MDPs in parallel as separate subprocesses using PPO. Because the initial state distribution is drawn from states encountered in the training planning graphs rather than sampled directly from the abstract state, no extra assumptions are needed.

At evaluation time, the abstract planner runs exactly as before, but with the learned shortcut policies added to the option set. Since learned policies can fail, the system checks whether each shortcut actually reached its terminal abstract state within a fixed step budget and removes the corresponding graph edges if it did not; shortcuts that work are selected automatically whenever they yield a shorter plan.

For generalization, the method exploits the object-and-relation structure that TAMP already uses. Each shortcut has a small set of relevant objects derived from the atoms it adds and deletes, and policies observe only the feature vectors of those objects. When a new task contains different objects, the system searches for a type-preserving, injective mapping from the training shortcut's relevant objects to objects in the evaluation shortcut that is consistent with the added and deleted atoms; if a match is found, the learned policy is deployed with the substituted object inputs.

Why This Matters

Impact on research. The paper argues for a middle ground between model-based hierarchical planning and model-free RL: rather than learning skills tabula rasa from low-level cues, it uses the high-level structure of existing skills to guide discovery of new ones. It positions itself as the first method that learns low-level skills specifically to improve the execution time of an abstract planner, and it shows that a learned skill can be physically applicable to more objects than its abstract representation deems relevant — a departure from the STRIPS-style assumption that each skill affects only a small, pre-specified set of objects.

Real-world applications (as motivated or implied by the paper's environments and framing):

• Robotic manipulation in cluttered spaces, such as retrieving an object from a drawer packed with other items rather than clearing neighbors one at a time.
• Warehouse or tabletop organization, such as collecting and binning a set of objects of irregular shape, mirroring the Cleanup Table environment built from Objaverse objects.
• Non-prehensile manipulation, including pushing, slapping, or wiping obstacles aside when grasping is infeasible or inefficient.
• Generalizing a trained manipulation skill to scenes with different numbers of objects, distractor items, or changed mass and friction properties.

Industry relevance. SLAP is presented as a plug-and-play module: given any domain with an abstract planner, it learns shortcuts without additional user input, and it degrades gracefully to pure planning when shortcuts are hard to learn. That makes it attractive for organizations that already have TAMP stacks and hand-engineered skill libraries and want to reduce execution time without replacing their planning infrastructure. The reported compute footprint — up to roughly 9 hours of RL time on a single H100 with 4 CPU cores in the largest environment — is modest by modern standards.

Future Directions

• Softening the problem decomposition. A stated limitation is that SLAP cannot deviate from the decomposition induced by user-provided options; the authors suggest using those options instead as a "soft" decomposition that hierarchical RL could further improve.
• Scaling to larger abstract spaces. The planner used here is intentionally simple from a TAMP perspective, so the authors identify integrating more advanced planning techniques as a route to very large abstract spaces.
• Relaxing the sufficiency assumption. The paper assumes the given options suffice for solving tasks, which underpins the guarantee of task success. Without that assumption SLAP still applies, but success is no longer guaranteed — although learned shortcuts could then improve success rate over pure planning.
• Safety and predictability. Because shortcut behaviors fall outside the manually defined option set, execution can be less predictable than traditional TAMP; incorporating safety constraints into shortcut learning is proposed as future work.
• Better shortcut policy learning schemes. The authors plan to continue exploring alternatives to independent per-shortcut policies, since shared or universal policy learning underperformed in their tests, and to combine SLAP with methods that learn state and action abstractions.
• Moving beyond simulator access. A final direction is to remove the assumption of access to a simulator by using real-to-sim-to-real techniques to reconstruct approximate simulators from real-world data and learn shortcut policies inside them.

Target Audience

This paper is most valuable to robotics and AI researchers working at the intersection of task and motion planning and reinforcement learning, particularly those interested in skill discovery, option learning, and long-horizon manipulation with sparse rewards. It is also relevant to practitioners who already have a TAMP system or a library of programmed skills and want to improve execution efficiency without redesigning their planner, and to hierarchical RL researchers studying how prior high-level structure can guide low-level policy learning. Readers need a background in MDPs, options, abstract planning graphs, and PPO/SAC to follow the technical sections, though the core idea — learn shortcuts between abstract states — is accessible at a high level.

Authors’ abstract

Long-horizon decision-making with sparse rewards and continuous states and actions remains a fundamental challenge in AI and robotics. Task and motion planning (TAMP) is a model-based framework that addresses this challenge by planning hierarchically with abstract actions (options). These options are manually defined, limiting the agent to behaviors that we as human engineers know how to program (pick, place, move). In this work, we propose Shortcut Learning for Abstract Planning (SLAP), a method that leverages existing TAMP options to automatically discover new ones. Our key idea is to use model-free reinforcement learning (RL) to learn shortcuts in the abstract planning graph induced by the existing options in TAMP. Without any additional assumptions or inputs, shortcut learning leads to shorter solutions than pure planning, and higher task success rates than flat and hierarchical RL. Qualitatively, SLAP discovers dynamic physical improvisations (e.g., slap, wiggle, wipe) that differ significantly from the manually-defined ones. In experiments in four simulated robotic environments, we show that SLAP solves and generalizes to a wide range of tasks, reducing overall plan lengths by over 50% and consistently outperforming planning and RL baselines.