The Open Ant: A Robot Platform for Reinforcement Learning Research

AI summary

Overview

Research area: Reinforcement learning (RL) and robotics — specifically, physical robot platforms designed for RL researchers who primarily work in simulation.

Technical level: Intermediate. The paper assumes familiarity with RL terminology (policy, action-value functions, SARSA, SAC, sim-to-real transfer) but explains the hardware and platform design in accessible terms.

Scope: The paper presents the Open Ant, an open-source quadruped robot and matching MuJoCo simulation modeled on the Gymnasium Ant benchmark, and reports on-hardware learning results, sim-to-real transfer results, and usability observations from multi-site deployments.

What This Paper Is About

RL research is dominated by simulation because physical robots are expensive, slow to set up, and hard to maintain, which makes it unclear whether simulation results transfer to reality. The authors build a physical robot that mirrors an existing, widely used simulated environment (the Gymnasium Ant), so that researchers can move between simulation and hardware without learning an entirely new platform. Their goal is to show that learning directly from a physical robot's experience is practical, repeatable across different RL algorithms, and achievable by researchers with little or no robotics background.

Key Contributions

1. A physical robot plus simulation pair. The authors developed the Open Ant, consisting of the Physical Ant (hardware) and the Simulated Ant (MuJoCo), inspired by the Gymnasium Ant but altered in size, sensing, actuation, and reward structure to be realizable as a physical research platform. Both hardware design and software are released open-source on GitHub.

2. Demonstrated compatibility with multiple RL methods. The platform supports both learning directly from hardware experience and policy transfer from simulation to reality, using two substantially different algorithms: SARSA(λ) with linear function approximation and Soft Actor-Critic (SAC).

3. Evidence of a nimble experimental ecosystem. The authors report rapid onboarding of researchers with limited robotics and RL experience, and rapid repair and revision of hardware enabled by 3D printing and commercial off-the-shelf components.

4. A non-episodic, low-intervention task design. The "back-and-forth" reward formulation lets the robot keep learning indefinitely without a human returning it to the start position when it reaches the arena boundary.

Main Findings

• One-hour learning on hardware. Competent walking policies were learned from scratch in approximately one hour directly from the physical robot's experience for both SARSA(λ) and SAC. Each algorithm was run on the robot for 80 minutes per trial, with five independent trials per algorithm.

• SARSA(λ) walked reliably across all trials. In every one of the five trials, the agent achieved slow but competent walking, with an average reward of 2–4 cm/s within approximately one hour of on-hardware learning. The five trials had a total of one interruption, caused by a leg failure, and the leg was repaired within 10 minutes.

• SAC learned locomotion directly on hardware. By the end of training, all five trials on both the Physical Ant and the Physical Ant Lite exhibited stable, visually plausible walking behavior. SAC required more interventions to clear cable entanglement than SARSA(λ) — the authors note the robot was also moving more under SAC policies.

• Three modifications mattered for SAC on hardware. Reducing random interaction steps before learning from 5000 to 2000 (about 4 minutes of on-hardware random interaction), adding Layer Normalization to the policy and critic networks, and scaling rewards by a larger factor (an initial scaling of 1.0 led to poor learning). The LayerNorm and reward-scaling effects were validated in simulation across 30 seeds.

• Simulation policies transferred to hardware, but rankings changed. Of 2,304 policies learned in simulation with SAC (400,000 timesteps, roughly 13 hours each, with no domain randomization), ten policies were sampled with uniformly distributed simulated average reward between 13 cm/s and 23 cm/s. All transferred with some success and walked in the correct direction, but the policy that performed best in simulation was only the sixth best on the Physical Ant.

• Platform built and used across multiple sites. The robot has been independently assembled and tested in Canada, USA, and Malaysia. Once components were purchased, assembly took between 2 to 5 hours; a local university robotics student team reported that after understanding the parts, assembly took less than two hours.

• New users succeeded quickly. At the Openmind Research Institute Winter School in Malaysia, five independent teams (20 participants total) with limited robotics and RL experience and no prior experience with this platform all demonstrated successful learning and sim-to-real transfer within three days of access. Participants used their own laptops running Linux, MacOS, and Windows.

• Design details supporting the platform's claims. The Physical Ant is one third the scale of the Gymnasium Ant, has a 24-dimensional observation space versus 105 for Gymnasium Ant, an 8-dimensional continuous action space using position commands versus Gymnasium Ant's torque control, and costs approximately USD 2200 in components (excluding 3D-printing filament). A lower-cost variant, the Physical Ant Lite, using plastic-geared motors costs approximately USD 500, but those actuators are more susceptible to wear and overheating.

Methodology in Plain English

The authors began from the Gymnasium Ant, a popular simulated benchmark with a four-legged body and continuous control, and built a physical robot that matches it closely enough that the same software interface applies. Because the simulated Ant's proportions imply an extremely low mass-to-size ratio that cannot be realized physically, they scaled the robot to one third size, giving a higher mass-to-length ratio that is easier to manufacture and more robust.

The Physical Ant has four 3D-printed legs, each with two Dynamixel actuators, a spherical torso housing an IMU and USB electronics, and a tether to a wall outlet and to an external computer over USB. An overhead webcam tracks fiducial markers on the torso and the ground to compute the robot's position and heading.

For the task, instead of learning to walk forward — which would require a human to reset the robot at the arena edge — they designed a non-episodic back-and-forth task. The instant reward is the projection of the robot's position displacement onto a reward direction vector; when the robot crosses a circle of radius R (0.3 m in the experiments), the reward direction flips so the robot is incentivized to walk back. Performance is measured as average reward per second over a 120-second window.

For SARSA(λ), they abstracted the continuous robot commands into a discrete set of long-duration motion primitives lasting 10 robot timesteps, with the agent operating at 0.5 s per step while the robot was commanded every 0.05 s. The action-value function was approximated with tile coding, and parameters were chosen beforehand using Optuna across 100 configurations and ten random seeds in simulation.

For SAC, they adapted the CleanRL implementation, tuned the action bounds, and ran learning directly on the robot. They also evaluated whether policies trained purely in simulation (2,304 SAC policies, no domain randomization) could run on the Physical Ant without further learning, measuring each for 5,000 steps (10 minutes) with no interventions.

Why This Matters

Impact on research. The paper argues that RL research has become a victim of its own success with simulators, with relatively few studies learning directly in physical reality. By making a physical robot that shares an interface and morphology with a standard simulated benchmark and that can be built and repaired without robotics expertise, the authors aim to lower the practical barrier to including hardware experiments in ordinary RL research. They frame the platform as something the community can revise over time rather than a fixed benchmark, similar to the Arcade Learning Environment.

Real-world applications:

• Continual and run-time learning, where algorithms adapt during deployment rather than being retrained offline — relevant to adaptive control settings such as ground vehicles, quadcopters, and remote-controlled aircraft cited in the paper.
• Physical domains where accurate simulators are unavailable or require human expertise to select the relevant dynamics, such as the fusion plasma control, balloon navigation, and gravitational wave detector examples the paper cites.
• Education and rapid research prototyping, demonstrated by the winter school where 20 participants across five teams ran learning and transfer experiments within three days.
• Benchmarking sim-to-real transfer methods, since the platform provides matched simulated and physical versions of the same task and performance metric.

Industry relevance. Commercial robot platforms are cited as carrying risks of product obsolescence, high purchase costs, long and costly repairs, and rigid software interfaces. The Open Ant's use of commercial off-the-shelf components, 3D-printed parts, an approximate USD 2200 build cost, external compute over USB, and an open-source release positions it as a lower-cost, repairable alternative for organizations that want robot experiments without dedicated robotics engineering staff.

Future Directions

• Vision-based learning. The Physical Ant includes an on-board camera described as support for future vision-based learning tasks, which the current experiments do not use.
• Studying run-time versus design-time questions. The authors motivate the platform with the distinction between design-time knowledge (observation and action spaces, initial behavior policies, body design) and run-time knowledge acquired only during deployment, including phenomena like overheating that are hard to model; the platform is intended to make these run-time conditions practically studiable.
• Reducing the need for manual intervention. Cable entanglement required experimenters to pause, disentangle, and reset the robot in both SARSA(λ) and SAC experiments. Solving this would strengthen the claim of fully non-episodic, minimal-intervention learning.
• Reliable policy ranking for sim-to-real selection. The reordering of policy performance between simulation and the Physical Ant is flagged as an important problem, because comparing policies is necessary for policy improvement; the paper does not report a solution.

Note: The provided paper content is truncated partway through Section 6.1. Sections after that point, including the remainder of the platform-suitability discussion, the referenced thermal analysis (Section 10.1) and electrical and mechanical improvements (Section 10.2), and any conclusion, are not present in the content supplied, so their details are not reported here.

Target Audience

RL researchers who typically work in simulation and want a straightforward path to physical robot experiments; researchers interested in continual, run-time, or non-episodic learning; students and instructors who need an affordable, repairable teaching platform; and engineers evaluating open-source quadruped platforms for benchmarking sim-to-real transfer. Because the paper emphasizes onboarding researchers without robotics backgrounds, it is also relevant to labs that lack dedicated robotics engineering support.

Authors’ abstract

Reinforcement learning (RL) research has demonstrated success in both physical and simulated domains; however, the predominant methodology remains rooted in simulations. The predominance of simulations makes translating research to physical reality uncertain for both algorithms and researchers. We propose a physical platform that is designed to simplify the transition. In this paper, we present the Open Ant: a physical variant of the commonly used Gymnasium Ant environment, along with a simulation. We demonstrate that competent walking policies can be learned from scratch in approximately one hour directly from the physical robot's experience for two substantially different RL algorithms: SARSA($λ$) and Soft Actor-Critic (SAC). Separately, we show policies that were learned in simulation transfer to reality. We also examine how well the platform supports a nimble experimental ecosystem. Specifically, we observe the speed with which new users from diverse backgrounds achieve their first success with the platform, and how easily the platform can be repaired and updated when hardware issues arise. Both the hardware design and software are available as open-source on GitHub for ease of customization. In summary, we advocate for the use of the Open Ant for RL researchers who frequently use simulated environments, so they can more easily include robot experiments in their evaluations.