Learning Reach-Avoid Task with Reinforcement Learning: Vectorized Simulation and Benchmark

AI summary

Overview

Research area: Robotics / deep reinforcement learning — specifically benchmarking DRL agents on robot-arm reach and reach-avoid motion tasks (moving an end-effector to a target while avoiding collisions with the robot itself and with obstacles).

Technical level: Advanced. The paper assumes familiarity with reinforcement learning algorithms (PPO, SAC), robot kinematics, and GPU-accelerated simulation frameworks (MuJoCo MJX, Brax, JAX).

Scope in one sentence: The paper introduces a GPU-vectorized simulation benchmark that evaluates PPO and SAC on reach and reach-avoid tasks using the full workspace of the UR5e and Franka Emika Robot arms, rather than the small tabletop workspaces common in prior work.

What This Paper Is About

Deep reinforcement learning is often used to teach robotic arms to reach a target while avoiding collisions, but the benchmark environments used to evaluate these agents are typically simplified — small tabletop-style workspaces where self-collision and obstacle avoidance are barely a challenge. The authors argue this creates a gap between reported benchmark performance and the difficulty of realistic scenarios, because an agent that appears near-perfect in a constrained setup may fail badly when the robot's entire reachable space is used. Their goal is to build a fast, vectorized, open-source environment that exposes this gap and provides a fair benchmark for the reach-avoid task without simplifications.

Key Contributions

1. A reach-avoid task environment built in a GPU-accelerated, vectorized manner (MuJoCo MJX physics plus the Brax training pipeline) suitable for high-speed reinforcement learning experimentation, achieving more than 10x speed-up in training compared to using Stable Baselines 3.
2. A comprehensive benchmark for the reach-avoid task that, according to the authors, is the first to capture real-world complexity without simplifications — full robot workspace, randomly sampled start poses, random reachable targets, and randomly placed spherical obstacles.
3. Open-source release of all environments and benchmarking code (the paper states the code is available at a blind-review link).
4. An empirical study spanning action spaces, reward formulations, observation spaces, network sizes, workspace sizes, and obstacle sizes, with cross-evaluation showing where trained policies fail to generalize.

Main Findings

• Best reach-task results: The best-performing agent achieved a 96.1% target reached rate with the UR5e (1 cm target size) and 98.8% with the Franka Emika Robot, with self-collision rates of 2.2% and 0.3% respectively.
• Best reach-avoid results: 86.8% target reached on the UR5e (10 cm obstacle radius, 2 cm target) with 2.7% self-collisions and 4.8% obstacle collisions; the Franka Emika Robot reached 95.2% with 0.1% self-collisions and 2.5% obstacle collisions.
• Algorithm differences: SAC struggled on the UR5e, reaching only 61.6% with a 17.3% self-collision rate, largely from wrist–forearm collisions that are physically impossible on the Franka Emika Robot. SAC did better on the Franka (87.9% target reached, 2.7% self-collisions). Halving the UR5e joint range for initialization from 720° to 360° raised SAC to 84.8% target reached with only 3.5% self-collisions. PPO was used for all remaining experiments.
• Action space: Both control schemes reached statistically identical target rates (96.5% absolute vs. 96.4% relative), but absolute position control had a 1.3 times higher self-collision rate (3.0% vs. 2.3%) and was much less precise: 0.012 m distance to target, 41.5 steps in target, and 21.7 times leaving the target per episode, versus 0.005 m, 99.1 steps, and 0.05 times for relative control.
• Reward formulation: Contrary to prior work suggesting sparse rewards are superior, the sparse reward agent failed to learn the task at all in a large workspace with a small target — the probability of a random policy reaching the 1 cm target is only 0.006%, requiring an average of 2 million steps to receive the first non-zero reward. Semi-sparse and dense rewards both worked; dense learned faster and reached higher precision. An acceleration penalty of -0.0001 reduced average acceleration by up to 12% with no statistically relevant effect on overall performance but slowed early learning; larger penalties caused the agent to deliberately collide to end episodes.
• Observation space: For relative position control, a minimal observation set learned slowly (150–200 million steps to reach 80% target reached) with high variance; adding the end-effector's Cartesian position gave a 96% target reached rate and 0.5 cm final distance versus 2–4 cm. Increasing network depth from 3 to 5 layers did not fix the minimal-observation problem, suggesting it is not a capacity issue. For absolute position control, including the end-effector position and previous action was best.
• Network size: A 3×256 architecture sufficed for relative position control; 4×256, 3×512, and larger first layers gave no benefit, while 2-layer networks were insufficient. For absolute position control, 4×256 gave a slight improvement over 3×256 and trained faster than 3×512.
• Workspace size is decisive: An agent trained in the PandaGym workspace achieved 100% target reached with zero collisions in just 1.5 million steps, whereas the Front workspace agent needed 10 million steps to reach 99% and never eliminated a residual 1% collision rate. The gap was even more pronounced with SAC.
• Generalization is asymmetric: Policies trained in larger workspaces generalize perfectly to smaller ones, but not the reverse. A PandaGym-trained agent dropped to 13.9% on the Front workspace and 4.0% on the Full workspace; a Front-trained agent dropped to 64.5% on the Full workspace, while an Upper-trained agent held 94.5%. Initialization also mattered: a Full-workspace policy trained with a limited joint range dropped from 98.9% to 60.1% target reached (self-collisions rising from 1% to 17.8%) when evaluated with full-range initialization.
• Reach-avoid specifics: The semi-sparse reward, though slower to learn, converged to a safer policy with 1.5% fewer self-collisions and 1% fewer obstacle collisions than dense reward. Adding the Euclidean distance from each robot link to the obstacle was critical, cutting obstacle collisions from 8% to 4.5%; including obstacle size gave no benefit because size was not randomized during training. A larger 4×256 network outperformed 3×256, and 5 layers added nothing. Restricting to the Front workspace raised the UR5e to 93.5% target reached (1.1% self-collisions, 3.0% obstacle collisions) and the Franka to 96.1% (0.1% self-collisions, 1.7% obstacle collisions).
• Obstacle size: Across obstacle radii of 0.025, 0.05, 0.1, 0.2, and 0.3 m, the final obstacle collision rate stayed at roughly 4%, while the target reached rate steadily declined (from 0.880 at 0.025 m to 0.781 at 0.3 m). Agents overfit to the obstacle size they trained on — small-obstacle agents transfer poorly to larger obstacles, while large-obstacle agents transfer better to smaller ones. A reach agent with no obstacle training dropped from 84.4% at 0.025 m to 60.3% at 0.3 m, with obstacle collisions rising from 12.5% to 37.3%.
• Baselines: On the reach task, a pseudo-IK baseline reached 0.369 (1 cm target) and 0.510 (2 cm target) with 0.222 self-collision rate; random and standstill agents reached essentially 0.000–0.001. A reach agent placed in an environment with an obstacle dropped to 86.6% target reached, with obstacle collisions in 10.3% of episodes.

Methodology in Plain English

The authors built a simulation framework on top of MuJoCo MJX and the Brax library so that both the physics simulation and the reinforcement learning algorithms run as a single just-in-time compiled process on one GPU. The environment is written in a "stateless" style, following the Gymnasium API, where all dynamic data lives in a state object passed into and returned from pure functions — which is what allows JAX to vectorize thousands of simulations at once.

They tested two robots (the UR5e and the Franka Emika Robot) using MuJoCo Menagerie model files, with several practical fixes: replacing mesh or cylindrical collision geometry with capsules for speed and MJX compatibility, adding joint damping to prevent velocities exceeding the real robot's limits by up to 5x, and reducing solver iterations to 1 with 6 line-search iterations and a dense Jacobian. Simulations used a 2 ms timestep with 20 physics substeps per environment step, giving episodes of 125 steps, or 5 seconds of simulated time; the agent typically reaches the target within 20 to 50 steps.

They then ran controlled comparisons. Training used PPO (batch size 512, 64 minibatches, learning rate 8·10⁻⁴, discount factor 0.97, 5 or 7 updates per batch, unroll length 10, no action repeat) and SAC (128 parallel environments, batch size 256, learning rate 1·10⁻⁴, reward scaling 20, replay buffer between 102,400 and 2,048,000 transitions). Each experiment ran across 5 random seeds, and final performance was averaged over those seeds, evaluating each on 4096 environments. Performance was measured by target reached rate, self-collision rate, obstacle collision rate, and average end-effector distance to target over the last 10 steps. Start poses were sampled from the robot's full joint range with collision-causing poses rejected; targets were sampled either uniformly in the spherical workspace or from the end-effector position of a random collision-free robot pose; obstacles were sampled uniformly in a 1.2 m × 1.2 m × 1.2 m cube centered at the origin, again rejecting colliding placements.

Why This Matters

The paper shows that strong benchmark numbers on simplified environments can hide a substantial drop in real-world-relevant performance. DRL agents that solved a constrained reach task perfectly collapse when the workspace is enlarged or the initialization distribution is changed. This challenges how the community reports and compares results, and argues that workspace size, initialization distribution, and sampling procedure must be documented for results to be reproducible and comparable.

The paper does not enumerate specific industry applications; the task it studies underlies several areas of robotics, and the following are reasonable extrapolations from the paper's framing:

• Industrial robot arms performing pick-and-place and precise object manipulation in cluttered cells, where self-collision and link-level obstacle avoidance matter.
• Motion planning and control for articulated manipulators, where the paper contrasts learned joint-level control with Cartesian control plus classical inverse-kinematics solvers.
• Safety-critical manipulation where collision rates must stay near zero, since the benchmark explicitly reports self-collision and obstacle collision rates separately.
• Simulation-based training pipelines that need fast, large-scale data generation on a single GPU.

Industry relevance: The environment targets a single NVIDIA RTX 3090 Ti with an Intel Core i7-13700k, sustaining roughly 185,000 steps per second for the simple reach task and around 125,000 steps per second for reach-avoid with 8,192 parallel environments — throughput that makes rapid experimentation feasible outside large compute clusters. The open-source release of environment and benchmarking code lowers the barrier for others to reproduce and extend the results.

Future Directions

1. Bridging the sim-to-real gap. The authors list this as an explicit limitation and call for extending the benchmark to physical robotic platforms and incorporating realistic noise models.
2. Dynamic and more complex obstacles. The current benchmark is restricted to static spherical obstacles. The authors propose introducing dynamic obstacles and more complex obstacle geometries.
3. Improving generalization across workspace and initialization distributions. The cross-evaluation results show policies trained in small workspaces or limited joint ranges fail when evaluated outside their training distribution. The paper notes that approaches such as reusing the previous episode's final pose as the next initial pose may lead to poor generalization, leaving open the question of how to train policies that transfer.
4. Closing the remaining performance gap. Even the best reach-avoid agents leave roughly 13% (UR5e) and 5% (Franka) of targets unreached, and the authors state that additional research is still required before claiming the robotic arm reach-avoid task has been successfully solved with DRL.

Target Audience

Reinforcement learning researchers working on manipulation and motion planning; robotics engineers evaluating whether DRL is ready for full-workspace control; and benchmark designers who need to understand how workspace size, initialization distribution, reward formulation, and network architecture interact with reported agent performance. Readers need a working knowledge of RL algorithms and robot kinematics to get full value from the experimental sections, though the high-level conclusions about simplified benchmarks misleading evaluation are accessible to a broader audience.

Authors’ abstract

Deep reinforcement learning (DRL) has a longstanding tradition in addressing the reach-avoid task problem, especially for controlling robotic arms. While this task serves as a baseline environment within the research community, the ability of DRL to effectively learn the each-avoid task in complex and realistic scenarios beyond simplified and restricted tabletop settings remains uncertain. In this paper, we present, for the first time, a comprehensive benchmark for the reachavoid task that accurately captures real-world complexities without simplifications. We demonstrate a diverse range of settings for robotic arm reach-avoid task, which can be used for evaluating DRL research. We achieved this by utilizing the MuJoCo MJX physics engine and parallelizing both the simulation environment and DRL algorithms using the Brax library. We achieved state-of-the-art results with success rates of 96.1% (UR5e) and 98.8% (Franka Emika Robot) for the reach task and 86.8% (UR5e) and 95.2% (Franka) for the static reachavoid task. Our results indicate that while in previous works DRL agents could solve, for example, a reach task in a simplified setting perfectly, their agents performance collapses when evaluated in realistic scenarios. Overall, this work identifies that additional research is still required to claim the successful resolution of the robotic arm reach-avoid task using DRL. The environment and benchmarking code is available as open source at the following link