The Pulse
Apple Benchmark Finds a Sharp Failure Point in Reset-Free RL
REVERSAL-BENCH uses a continuous reversibility parameter and reset oracle to show how reset-free reinforcement-learning agents become trapped in irrecoverable states.
AI.info Team ·
Reset-free agents fail when one mistake cannot be undone
Reset-free reinforcement-learning agents can reach a sharp failure point as environments become less recoverable, according to a new Apple benchmark. When an agent enters an irrecoverable state, no permitted action can return it to the task, ending further learning.
The benchmark, called REVERSAL-BENCH, gives researchers a continuous reversibility parameter, ρ, ranging from 0 to 1. Apple uses that control to vary the recoverability of an environment and study how reset-free policies respond as irreversible states become more likely.
The study appears on Apple Machine Learning Research’s website in September 2026 and is authored by Riyaaz Shaik and Chandru Venkataraman.
“We introduce REVERSAL-BENCH, a benchmark that controls reversibility via a continuous parameter ρ∈ [0, 1] and provides a reset oracle, a ground-truth verification mechanism to test state recoverability across eight manipulation settings in five physics engines.”
Riyaaz Shaik and Chandru Venkataraman, authors of REVERSAL-BENCH
A benchmark for the reversibility cliff
REVERSAL-BENCH covers eight manipulation settings across five physics engines. Its reset oracle provides ground-truth verification for whether a state remains recoverable. The benchmark also includes a large multi-simulator dataset labeled with recoverability.
The distinction matters because reset-free learning depends on the environment remaining usable after each action. A transition into an irrecoverable state can leave the agent permanently trapped, preventing additional interaction from improving the policy. REVERSAL-BENCH makes that property an experimental variable rather than an unstated assumption.
Apple describes the benchmark as a way to measure the point at which reset-free learning stops progressing because the environment has absorbed the agent into an unrecoverable condition.
Irreversibility, not obstacle complexity
The researchers evaluate a broad range of policy architectures, including standard actor-critic algorithms, safe reinforcement learning and specialized reset-free frameworks. As ρ increases, reset-free agents are consistently absorbed into irrecoverable states, while episodic agents maintain steady learning because external resets prevent a single failure from ending their interaction with the task.
Apple also compares irreversible environments with geometrically identical reversible counterparts. The comparison is designed to isolate reversibility from other sources of difficulty, such as obstacle complexity. The company says the results show that the breakdown is causally driven by irreversibility rather than by a harder geometry.
The finding applies across autonomous reset-free baselines and constrained reinforcement-learning systems. In the benchmark’s full-physics simulations, learned manipulation policies display the same absorption pattern seen in simpler settings.
Prediction does not guarantee recovery
REVERSAL-BENCH also evaluates a safety shield intended to intervene before irreversible failures occur. The results indicate that recoverability can be predicted accurately, but active recovery works mainly when the agent still has a physically available way to steer clear of the trap.
That distinction separates recognizing danger from preventing it. A system may identify that a state is becoming unrecoverable without possessing an action capable of reversing the underlying physical event. Once the agent has crossed the reversibility boundary, a warning or prediction alone cannot restore the task.
A test for reset-free learning
Apple presents REVERSAL-BENCH as a benchmark suite, reset oracle and labeled dataset for studying this failure mode. Its central contribution is a controlled way to vary reversibility and test whether a policy can continue learning without external resets.
The benchmark does not claim that every reset-free system will fail under every condition. Instead, it identifies a recurring relationship between environmental recoverability and continued learning: as irreversible regions become more severe, reset-free agents are increasingly likely to become permanently absorbed, while episodic agents can continue after being reset.
For researchers working on autonomous reinforcement learning and physical manipulation, the benchmark offers a way to evaluate whether a policy’s success depends on an environment that can undo its mistakes. That question becomes especially important in settings where pushing, spilling or otherwise disturbing an object can permanently change what actions remain possible.