A Primer on SO(3) Action Representations in Deep Reinforcement Learning

AI summary

Overview

Research area: Robot learning / deep reinforcement learning for orientation control, specifically how to represent actions on the rotation group SO(3).

Technical level: Intermediate. The paper is explicitly written as a primer that prioritizes clarity and implementation over exhaustive manifold mathematics, but it assumes familiarity with continuous-control RL algorithms (PPO, SAC, TD3) and basic rotation representations.

Scope: A systematic empirical study of SO(3) action representations across three continuous-control algorithms (PPO, SAC, TD3) under dense and sparse rewards, validated on a pure-rotation environment plus three robot benchmarks.

What This Paper Is About

Robots frequently need policies that command orientations, but SO(3) admits no global, smooth, minimal parameterization, so every choice (Euler angles, quaternions, rotation matrices, Lie algebra coordinates) comes with distinct constraints and failure modes. While these trade-offs are well studied for supervised learning and for observation/input representations, the authors argue that the most suitable SO(3) action representation in deep RL remains unclear, since actions shape exploration dynamics, entropy regularization, and action clipping. The paper's goal is to evaluate these representations systematically and distill implementation-ready guidelines for practitioners.

Key Contributions

1. Systematic evaluation across three algorithms: The authors analyze the most popular continuous-control RL algorithms — PPO, SAC, and TD3 — in action spaces that include orientations, under both dense and sparse rewards.
2. Mechanistic explanation of performance differences: Beyond intuitions about smoothness and uniqueness, they attribute observed performance and sample-efficiency gaps to the map between Euclidean network outputs and SO(3), highlighting how representation-induced action projections affect exploration, action scaling, and regularization.
3. Concrete guidelines: They provide practical, algorithm- and representation-dependent recommendations for choosing policy representations and mitigating representation-induced pitfalls.
4. Validation on three robot platforms: They test their findings on drone control (trajectory tracking and drone racing), a RoboSuite manipulation suite, and goal-conditioned arm control with pose goals.

Main Findings

• Delta tangent vectors win overall: Across algorithms and reward formulations, the delta tangent vector representation (increments in the local frame) almost always produces the best final policy with minor variance between runs. The authors recommend it as the generally most reliable choice.
• Global matrices are second-best, with a caveat: Global rotation matrix representations achieve the second-best performance in the pure-rotation study, except for SAC with sparse rewards, where they perform poorly.
• Sparse rewards amplify representation failures: Other representations often perform poorly, particularly in sparse reward environments, despite the use of HER for the off-policy algorithms.
• Uniqueness and smoothness matter only locally: Hypothesis 1 was only partially confirmed. Rotation matrices are both unique and smooth yet are not uniformly best; quaternions suffer from the double-cover, and delta matrices underperform global matrices because they must learn the relation between current orientation and goal. However, local tangent spaces outperform global matrices because the maximum step angle limits the policy to a region where the tangent space is unique, discontinuity-free, and the Exp map is almost linear.
• Projections warp exploration distributions: Samples from Gaussian policies are generally off-manifold, and projecting them back produces action distributions that concentrate around small regions of the action space, harming exploration. This is most visible with sparse rewards. Euler angles and quaternions are most affected, matrices to some degree, and local tangent spaces least.
• Entropy regularization drives magnitude, not diversity: For PPO and SAC, maximizing entropy pushes actions toward larger norms in Euclidean space, but with a maximum step angle these do not correspond to more random actions — instead they lead to more stable rotation directions. Scaling delta tangent actions to the allowed range mitigates this. Attraction toward Euler singularities only becomes relevant with entropy coefficients raised by two orders of magnitude.
• Unit-rotation centering helps PPO: Centering delta actions around the unit rotation improves performance for quaternion and matrix representations in PPO; results for SAC and TD3 were mixed. Delta tangent and Euler actions are unaffected as they are centered by construction.
• Scaling tangent vectors improves performance: Unscaled tangent increments consistently exhibit a performance difference of around −1.5 compared to scaled tangent vectors across PPO, SAC, and TD3.
• Drone benchmarks: On trajectory tracking and drone racing with PPO, local tangent space actions consistently converge faster and achieve higher rewards. Surprisingly, Euler angles were second, attributed to the limited range of angles required (the drone cannot deviate far from upright without crashing). Absolute quaternion and matrix actions are highly random at initialization, leading to fast crashes; results were averaged over 25 seeds.
• RoboSuite benchmark: With SAC and shaped rewards over nine tasks, trained for 5M steps across five random seeds each, global actions do well (dense rewards compensate for exploration issues), and quaternions outperform the matrix representation on several tasks. Local tangent actions are competitive on most tasks but do not exceed quaternions. Narrow performance gaps between representations within a task versus larger gaps between tasks indicate that reward design and task difficulty dominate this benchmark.
• Goal-conditioned arm control: On the reach task, matrix and tangent representations converge quickly, quaternions follow slightly delayed, and Euler angles lag. On the harder pick-and-place task, the tangent representation reaches a 69.8% success rate, matrix 54.1%, quaternions 46.7%, and Euler angles 32.3% — a roughly 2x increase in success rate between Euler and tangent representations.

Methodology in Plain English

The authors first build an idealized environment with only rotational dynamics: a goal-conditioned MDP whose state is a pair of orientations (current and goal), with flattened rotation matrices as observations following Geist et al. (2024). The agent rotates toward the desired orientation with a maximum step length, using the shortest geodesic path; dense rewards are the negative angle to the goal, and sparse rewards are 0 when the angle to the goal is at most 0.1 and −1 otherwise, with episodes terminating after 50 steps.

They then compare four action representations — rotation matrices, unit quaternions, tangent-space rotation vectors, and Euler angles — each used both as global actions (a desired absolute orientation) and as delta actions (an intrinsic rotation relative to the current state). Everything else is held fixed: network architectures, training budgets, observation spaces, and reward definitions. Results are averaged over 50 runs per condition, and off-policy methods (SAC, TD3) use HER for the sparse reward setting.

Because network outputs are Euclidean and do not satisfy manifold constraints, the authors handle projections as a practical compromise: project the mean inside the network, sample in ambient Euclidean space, and project the sampled off-manifold action again inside the environment. This keeps training compatible with standard log-probability computations for PPO and SAC while guaranteeing feasible rotations at execution.

The paper is organized around five hypotheses — about smoothness/uniqueness, exploration, entropy regularization, unit-rotation centering, and action scaling — each stated, tested against the pure-rotation results, and explained through ablations. Finally, the authors transfer the most promising representations to three robot benchmarks, changing only the action representation of the policy in each case.

Why This Matters

Impact on research: The paper fills a gap between well-studied rotation representations in supervised learning and the under-explored question of action representations in deep RL. It moves beyond intuitions about smoothness and uniqueness to show that representation-induced geometry directly shapes exploration, entropy behavior, and optimization stability, and it supplies a mechanism-level explanation rather than a leaderboard.

Real-world applications:

• Robot manipulation: Policies that command full task-space poses, such as single-arm block lifting and two-arm peg-in-hole tasks in RoboSuite, where the action representation choice changed measured success rates.
• Aerial vehicles: Drone attitude regulation, figure-8 trajectory tracking, and autonomous drone racing through gates, where representation choice affected convergence speed and crash rates.
• Goal-conditioned arm control: Reaching and pick-and-place tasks where the gripper orientation is part of the goal, where tangent representations roughly doubled success over Euler angles.
• Any embodied system commanding orientations: The authors note the findings are relevant to computer graphics, state estimation, and control more broadly, since these tasks rely on policies with action spaces including rotations.

Industry relevance: The guidelines are implementation-ready, and the three algorithms studied (PPO, SAC, TD3) are extensively used in robotics research to train policies deployed on physical hardware. The code is publicly available at github.com/amacati/so3_primer, with a project webpage at amacati.github.io/so3_primer, lowering the barrier for practitioners to adopt the recommendations.

Future Directions

• Extend beyond state-based observations and small networks. The authors flag this as a limitation, arguing that observations do not affect the action space so the results likely still apply, but noting this requires empirical evidence.
• Study discrete action space algorithms, which open entirely new questions about discretization schemes and the required density of the SO(3) cover.
• Build better benchmarks for full SO(3) control. The authors note a lack of suitable benchmarks requiring control over the full SO(3) manifold, and offer their HER environment extension as a starting point for a standard task set.
• Investigate representation choices for diffusion policies. The authors suggest that diffusion's multi-modality and noise processes could lead to significantly different conclusions, since diffusion policies have found widespread adoption for imitation learning in robotics.

Target Audience

Robotics and reinforcement learning practitioners who need to design policies with orientation or full task-space pose actions, especially those deploying learned policies on physical hardware such as robot arms and drones. It is also useful for RL researchers studying action parameterization, exploration, and entropy regularization in continuous control, and for readers who want a clear, implementation-oriented entry point to SO(3) representations in learning systems — the paper explicitly prioritizes clarity, common pitfalls, and ease of implementation over an exhaustive mathematical treatment of manifold optimization.

Authors’ abstract

Many robotic control tasks require policies to act on orientations, yet the geometry of SO(3) makes this nontrivial. Because SO(3) admits no global, smooth, minimal parameterization, common representations such as Euler angles, quaternions, rotation matrices, and Lie algebra coordinates introduce distinct constraints and failure modes. While these trade-offs are well studied for supervised learning, their implications for actions in reinforcement learning remain unclear. We systematically evaluate SO(3) action representations across three standard continuous control algorithms, PPO, SAC, and TD3, under dense and sparse rewards. We compare how representations shape exploration, interact with entropy regularization, and affect training stability through empirical studies and analyze the implications of different projections for obtaining valid rotations from Euclidean network outputs. Across a suite of robotics benchmarks, we quantify the practical impact of these choices and distill simple, implementation-ready guidelines for selecting and using rotation actions. Our results highlight that representation-induced geometry strongly influences exploration and optimization and show that representing actions as tangent vectors in the local frame yields the most reliable results across algorithms. The project webpage and code are available at amacati.github.io/so3_primer.