Real-World Reinforcement Learning of Active Perception Behaviors

AI summary

Overview

Research area: Robot learning — reinforcement learning for active perception (information-gathering) behaviors, with a focus on real-world manipulation under partial observability.

Technical level: Intermediate overall; the theoretical section on POMDPs and advantage weighted regression is Advanced.

Scope: The paper introduces Asymmetric Advantage Weighted Regression (AAWR), an offline-to-online RL recipe that uses privileged training-time sensors to train policies that actively move to gather task-relevant information, and evaluates it on 8 simulated and real manipulation tasks across 3 robots.

What This Paper Is About

A robot's instantaneous sensors often do not reveal the information needed to act — a wrist camera cannot see a toy hidden behind a cabinet door. In such partially observed settings, good behavior requires the robot to deliberately move to gather missing information, a capability called active perception. The paper's goal is to train these active perception policies efficiently on real robots, using a small number of suboptimal demonstrations, an easy coarse policy initialization, and extra "privileged" sensors available only during training.

Key Contributions

1. A new algorithm, AAWR (Asymmetric Advantage Weighted Regression): It extends advantage weighted regression (AWR) by giving critics and value functions privileged, training-time-only observations, so the policy receives better supervision than it could get from partial observations alone.

2. Theoretical justification: Theorem 1 shows that maximizing expected policy improvement in a POMDP yields exactly the AAWR objective, whereas the symmetric variant (SAWR) does not recover the correct solution. The authors also show in Appendix E that privileged value functions are the fixed point of the IQL Bellman equations while unprivileged ones are not.

3. Broad empirical validation: AAWR learns active and interactive perception behaviors across 8 tasks, spanning simulated and real robots (Sim. Koch, Sim. xArm, Real Koch, Real Franka), different types of partial observability, and different privileged sensor types, and it outperforms all prior approaches in the evaluations.

4. A practical pipeline for real robots: AAWR works both in purely offline training and in offline-to-online fine-tuning, and it can act as a "helper" policy that hands off to a generalist foundation policy once the target is found.

Main Findings

• Simulated tasks: AAWR outperforms its non-privileged counterpart AWR and behavior cloning (BC) by approximately 2x in Camouflage Pick and 3x in Fully Obs. Pick. The authors note that gains appear even in the fully observable task, which they hypothesize is because the non-privileged critic must learn to extract object position from pixels.

• Active Perception Koch task: Only AAWR reaches 100% success by learning to scan the workspace. Distillation gets high initial success but plateaus at 80%, learning a local-maximum strategy of approaching the center of the workspace that fails when objects are in corners out of camera view. VIB collapses during evaluation because privileged information is unavailable at deployment.

• Blind Pick (real Koch robot, open-loop picking): Online AAWR achieves 94% grasp and 89% pick; offline AAWR achieves 88% grasp and 71% pick. By comparison, online AWR reaches 71%/55%, offline AWR 65%/62%, and BC 47%/41%.

• Handoff tasks on a real Franka: AAWR is best on all three metrics across Bookshelf-P (Search 92.4, Completion 44.4, Steps 36.6), Bookshelf-D (81.3, 44.4, 26.9), Shelf-Cabinet (78.2, 40.0, 46.3), and Complex (73.2, 50.0, 43.0).

• Generalist policies fail at search: The foundation policy π₀ scores Search 11.0 on Bookshelf-P, 66.7 on Bookshelf-D, 10.0 on Shelf-Cabinet, and 29.6 on Complex, and takes far more steps (263.3, 229.7, 280.0, 252.5 respectively). A VLM+π₀ variant using Gemini-2.5 also searches poorly, with the highest step counts (322.3, 281.8, 382.0, 374.7).

• AAWR beats a hand-engineered searcher on efficiency: The Exhaustive controller has decent search and completion rates but is slow; when normalized for time taken, AAWR scores 2-8 times higher in Search and Completion than Exhaustive.

• Learned qualitative behavior: In the Bookshelf tasks, AAWR learns to zoom out to see multiple shelves, scans bottom-to-up, then approaches the target. AWR and BC follow a relatively fixed path and fail to fixate on objects even when they appear in frame. AWR's AWR baseline achieves 0.0 Completion on Bookshelf-P.

• Training data used: Bookshelf-P and Bookshelf-D used 250 demonstrations split among four demonstrators, roughly a 50% π₀ success rate. Shelf-Cabinet used 35 trajectories at 74% success and Complex used 50 trajectories at 94% success. Simulated results are reported over 10 seeds per method.

Methodology in Plain English

The researchers model active perception tasks as partially observed Markov decision processes (POMDPs), where the robot sees only observations o but the underlying state s includes hidden facts like object location.

Their key move is asymmetry. During training, extra sensors are available — object detectors, bounding boxes, segmentation masks, or true object positions. These feed the critic and value networks, which estimate how good an action is. The policy itself only ever sees the partial observation, so it remains deployable without those extra sensors. The advantage estimate produced by these privileged critics is then used as a weight in a behavior-cloning-style loss: actions with higher estimated advantage get up-weighted, actions with lower advantage get down-weighted.

To train the value functions, the authors use implicit Q-learning (IQL) with its expectile regression objective for the value network and 1-step TD error for the Q network. The full procedure is offline-to-online: first the policy and value functions are pretrained on offline data for a set number of gradient steps, then the policy is run in the environment to collect online transitions, and both buffers are sampled 50/50 for further updates. At deployment, the privileged sensors are removed and only the partial-observation policy runs.

In the real robot "handoff" experiments, the active perception policy searches until an object detector spots the target across two consecutive intervals, at which point the robot switches to a generalist foundation policy to perform the grasp.

Why This Matters

Impact on research: The paper shows that privileged-information RL can be made sample-efficient enough to run on real robots, without requiring the heavy instrumentation (such as task-specific volumetric maps) that prior real-world active perception RL needed. It also provides a formal argument for why asymmetric advantage estimation is the correct objective in POMDPs, which strengthens the theoretical footing of asymmetric actor-critic methods generally.

Real-world applications:

• Warehouse and logistics picking, where items are occluded in bins, shelves, or totes and a robot must look before it can grasp.
• Home robotics and assistive devices, where a robot searches cluttered rooms for a requested object.
• Search and rescue or industrial inspection, where a robot must position its sensors to reveal occluded areas.
• Retail and lab automation, where a helper policy can hand off to an existing generalist grasping policy once the target is located.

Industry relevance: The method reduces the need for expensive sensor instrumentation at deployment and reduces reliance on hard-to-collect optimal demonstrations. The demonstrated ability to improve a struggling "generalist" foundation policy makes it directly relevant to teams that have already invested in large teleoperation-trained policies but find them inadequate for search tasks.

Future Directions

• Directly fine-tuning generalist foundation policies with AAWR instead of switching between a helper policy and the foundation policy at handoff.
• Using other forms of privileged information, such as outputs from foundation models, rather than hand-specified privileged sensors.
• Selecting useful features from privileged information automatically through representation learning, instead of relying on prespecified signals.
• Scaling AAWR to longer-horizon tasks where information-gathering challenges compound over time, and applying it more broadly to other partially observed robotic tasks.

Target Audience

Robotics and reinforcement learning researchers, particularly those working on POMDPs, asymmetric actor-critic methods, offline-to-online RL, and real-robot manipulation. Also useful for practitioners who already have a foundation manipulation policy and need a way to add search and information-gathering behavior. Readers without RL background will find the theoretical section on POMDPs and Bellman fixed points challenging.

Authors’ abstract

A robot's instantaneous sensory observations do not always reveal task-relevant state information. Under such partial observability, optimal behavior typically involves explicitly acting to gain the missing information. Today's standard robot learning techniques struggle to produce such active perception behaviors. We propose a simple real-world robot learning recipe to efficiently train active perception policies. Our approach, asymmetric advantage weighted regression (AAWR), exploits access to "privileged" extra sensors at training time. The privileged sensors enable training high-quality privileged value functions that aid in estimating the advantage of the target policy. Bootstrapping from a small number of potentially suboptimal demonstrations and an easy-to-obtain coarse policy initialization, AAWR quickly acquires active perception behaviors and boosts task performance. In evaluations on 8 manipulation tasks on 3 robots spanning varying degrees of partial observability, AAWR synthesizes reliable active perception behaviors that outperform all prior approaches. When initialized with a "generalist" robot policy that struggles with active perception tasks, AAWR efficiently generates information-gathering behaviors that allow it to operate under severe partial observability for manipulation tasks. Website: https://penn-pal-lab.github.io/aawr/