Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning

AI summary

Overview

Research area: Offline goal-conditioned reinforcement learning (GCRL), temporal distance learning, quasimetric value function architectures, and real-world robot manipulation.

Technical level: Advanced. The paper builds on Bellman-style value iteration, contrastive/ Monte-Carlo value estimation, quasimetric distance theory, the Metric Residual Network (MRN) architecture, and Bregman divergences with LINEX losses.

Scope: This paper introduces Multistep Quasimetric Estimation (MQE), an offline GCRL method that combines multistep Monte-Carlo returns with quasimetric distance parameterization, and evaluates it on simulated long-horizon benchmarks (OGBench, including a new "colossal" antmaze) and on real-world robotic manipulation (BridgeData).

What This Paper Is About

Learning to reach goals over long horizons is hard because an agent must estimate the temporal distance between pairs of observations. Prior methods split into two camps: temporal difference (TD) methods with local updates that carry optimality guarantees but degrade over long horizons, and Monte Carlo (MC) methods with global updates that work well in practice but generally only recover the behavior Q function Q_β rather than the optimal Q*. MQE aims to get both benefits at once by fitting a quasimetric distance using multistep Monte-Carlo returns, without needing an explicit hierarchy.

Key Contributions

1. MQE algorithm. The authors propose Multistep Quasimetric Estimation (MQE), an offline GCRL method that incorporates both multistep value learning and quasimetric architectures without requiring an explicit hierarchy. They state it is the first method capable of taking advantage of multistep TD returns with global value propagation through quasimetric architectures.

2. Horizon generalization and stitching. MQE shows stronger horizon generalization than prior methods, enabling "stitching" behavior, and is described as the only method with nonzero success rates in the new colossal-sized antmaze mazes.

3. A value-learning mechanism via action invariance. Rather than a tuned expectile value loss, MQE enforces action invariance, d(ψ(s), φ(s,a)) ← 0, through a squared-exponential loss that scales with the magnitude of the deviation and avoids the trivial solution in which both embeddings collapse to zero.

4. Real-world end-to-end offline GCRL with multistep stitching. The authors state this is the first end-to-end offline GCRL method enabling multistep stitching in the BridgeData real-world manipulation domain from an unlabeled offline dataset of visual observations, demonstrating robust horizon generalization.

5. A policy improvement result. Theorem 1 states that under a tabular setting with a full-support behavior policy, fitting the distance by the MQE objective and extracting a policy yields V_g^π(s) ≥ V_g^{π_β}(s) for all states s and goals g, i.e., improvement over the behavior policy. The proof is given in Appendix C.

Main Findings

• State-of-the-art on challenging OGBench tasks: On the aggregate OGBench success rates across a set of the most challenging evaluation tasks (90 tasks total, per the figure caption), MQE achieves the best performance among the compared methods. The broader evaluation setup is described as 13 state-based and 5 pixel-based environments (each pixel-based observation being 64×64×3) with 5 tasks each, described as a total of 95 tasks.

• Large margins on some tasks: On humanoidmaze-giant-stitch, MQE exhibits a 10× improvement over the previous best methods, including HIQL and n-SAC+BC, which perform explicit policy horizon reduction.

• One exception: Vision-based manipulation is the only case where HIQL performed better than MQE.

• Horizon generalization: With training trajectories fixed to only 4 meters long (via stitch datasets), MQE retains strong horizon generalization on antmaze of medium, large, giant, and colossal sizes, and is the only method with nonzero success rates in colossal mazes. Colossal mazes are described as 50% larger than the giant-sized mazes available on OGBench, with tasks whose horizons are up to 1000% longer than those in the training set.

• Real-world compositional scaling: On BridgeData tasks with a 6DoF, 5Hz WidowX250 manipulator, MQE performs comparably on single-stage tasks (single pick-and-place, open drawer) against GCBC, while TRA and GCIQL also perform competitively there. As the number of tasks that must be performed in sequence increases (up to 4), MQE retains relatively high task progress while GCIQL and TRA regress.

• Hardest real-world tasks: For the two most difficult tasks (quadruple pick-and-place, and drawer open and place), only MQE and TRA displayed positive success rate among all tested methods. The paper notes that such a BridgeData task had never been completed without hierarchical policies or high-level planners.

• Action invariance matters: On humanoidmaze-giant-stitch, using the action-invariance loss ℒ_I gives 26.5% (±1.3) success, versus 7.9% (±0.7) without it. Expectile losses were worse: κ=0.7 gives 11.3% (±1.1) and κ=0.9 gives 8.8% (±0.7).

• Waypoint sampling matters: The mixture sampling of k′ in Eq. 8 gives 26.5% (±1.3), compared to 22.1% (±1.1) for k′ ~ Geom(1−λ), 18.9% (±0.9) for k′ ~ Unif[1,K], 17.8% (±1.3) for k′ ~ Unif[1,50], 1.7% (±0.5) for k′ = 50, and 0% (±0.0) for k′ = 1.

• Both hyperparameters must be high: The success-rate sweep over (p, λ) (with α = 0.01, averaged over 4 seeds) suggests MQE needs a waypoint far enough for value to propagate rapidly and a high enough p for local consistency to be respected; the one-step objective 𝒯 alone is not sufficient to learn a good distance for policy learning.

• Long-horizon control scale: MQE achieves state-of-the-art performance on tasks requiring complex control and long-horizon reasoning, reported as up to 21 degrees of freedom and up to 4000 timesteps.

Methodology in Plain English

The method treats goal-reaching as learning a distance between a state (or state-action pair) and a goal, where smaller distance means higher value. Q and V are written as V_g(g)·e^(−d(·,g)), so the value at the goal itself is positive and the reward signal is baked into the parameterized distance.

The distance is parameterized with a quasimetric architecture (the Metric Residual Network, MRN), which splits learned embeddings into components and sums an asymmetric part (a maximum of ReLUs over coordinate differences) with a symmetric part (an l2 norm of differences). This guarantees the triangle inequality and non-negativity properties that define a quasimetric.

Where prior quasimetric distance learning used single-step TD updates or contrastive Monte-Carlo updates, MQE regresses the exponential negative distance not just toward the immediate next state but toward a sampled "waypoint" further along the trajectory, scaled by γ^(k′). The waypoint offset k′ is drawn from a mixture: with probability p it is 1 (a one-step backup), and with probability 1−p it is drawn from a geometric distribution capped at the index of the future state. The objective, called 𝒯_β, therefore behaves like an n-step return but with a random number of steps rather than a fixed one.

Two additional pieces make it work. First, an action-invariance loss pushes d(ψ(s), φ(s,a)) toward zero so the value function behaves like a max over actions of the Q function; the squared form (e^(−d) − 1)² is used instead of plain L1/L2 so gradients scale with the size of the violation and the representation does not collapse to zero. Second, the policy is extracted with behavior-regularized deep deterministic policy gradient (DDPG + BC), minimizing the learned distance evaluated at the policy's action plus a behavior-cloning term weighted by α.

Training uses a single critic and a single goal-reaching policy — waypoints are sampled randomly rather than being produced by a separate high-level policy — so the method has no hierarchical components. For analysis, the authors show the tabular version of MQE can be seen as (1) fitting the distance under the behavior distribution, (2) projecting it into the quasimetric space via a path-relaxation operator 𝒫(d)(x,z) = min_y [d(x,y) + d(y,z)], and (3) extracting a policy, which together yield the policy-improvement guarantee.

Why This Matters

Impact on research. The paper argues that the usual separation between TD-style local value propagation (which can recover Q* in theory) and Monte Carlo global value propagation (which recovers Q_β but tends to work better in practice) does not have to be a hard tradeoff. MQE shows an empirical and theoretical route to combining multistep returns with quasimetric constraints, and extends offline GCRL evidence into a real-world manipulation domain where prior end-to-end methods had limited success.

Real-world applications:

• Robot manipulation skill composition: chaining several pick-and-place primitives from a dataset that contains only individual primitives, without a high-level planner.
• Dependency-ordered task execution: for example, opening a drawer and then placing an object inside, conditioned on a single goal image of the finished state.
• Long-horizon navigation and locomotion control in large environments.
• Vision-based goal-reaching from image observations, since the method is evaluated with pixel-based observations as well as state-based ones.

Industry relevance. Offline learning from existing, unlabeled robot data — rather than online trial and error — is directly relevant to warehouse automation, manufacturing, and service robotics, where data collection is expensive and safe exploration is difficult. The paper's claim that MQE's training stability allows application to real-world robot learning "without additional design choices" addresses a common barrier to deploying RL on physical hardware.

Future Directions

1. Replacing waypoint heuristics. The authors note that MQE samples waypoints based on heuristics, which could incur more computation costs when finding the optimal sampling scheme for environments outside their evaluation range. Future work can investigate the theoretical connection between sampling waypoints and successor distances.

2. Different policy classes. The paper suggests investigating the effect of policy learning on action-chunking policies.

3. Beyond offline RL. The authors propose applying the same method in offline-to-online RL or online RL settings.

4. Open questions. The horizon generalization results on the colossal mazes and the real-world compositional scaling raise the question of how far the approach extends as horizons and the number of composed skills grow, and how the sampling distribution should be chosen in new domains.

Target Audience

This paper is best suited to reinforcement learning researchers and graduate students working on offline RL, goal-conditioned RL, and temporal distance or successor-representation learning. It is also relevant to roboticists who deploy learned manipulation policies from offline datasets of visual observations, and to practitioners interested in whether quasimetric architectures can be combined with multistep returns in a single, non-hierarchical algorithm. Readers should be comfortable with Bellman backups, Q/V functions, contrastive learning objectives, and quasimetric or metric learning concepts; the paper is not introductory.

Authors’ abstract

Learning how to reach goals in an environment is a longstanding challenge in AI, yet reasoning over long horizons remains a challenge for modern methods. The key question is how to estimate the temporal distance between pairs of observations. While temporal difference methods leverage local updates to provide optimality guarantees, they often perform worse than Monte Carlo methods that perform global updates (e.g., with multi-step returns), which lack such guarantees. We show how these approaches can be integrated into a practical offline GCRL method that fits a quasimetric distance using a multistep Monte-Carlo return. We show our method outperforms existing offline GCRL methods on long-horizon simulated tasks with up to 4000 steps, even with visual observations. We also demonstrate that our method can enable stitching in the real-world robotic manipulation domain (Bridge setup). Our approach is the first end-to-end offline GCRL method that enables multistep stitching in this real-world manipulation domain from an unlabeled offline dataset of visual observations and demonstrate robust horizon generalization.