MILER: Semantic Mid-Level Representation for Sim-to-Real Reinforcement Learning in Unstructured Autonomous Driving

AI summary

Overview

Research area: Reinforcement learning for autonomous driving, specifically sim-to-real transfer in unstructured (off-road) environments.

Technical level: Advanced (assumes familiarity with reinforcement learning, bird's-eye-view perception, and vehicle dynamics).

Scope: This paper introduces a framework that trains a driving policy entirely in a simplified semantic simulation and transfers it to two real vehicles without additional fine-tuning, achieving autonomous driving at speeds up to 33.6 km/h on a challenging off-road test track.

What This Paper Is About

Reinforcement learning has the potential to produce driving policies that outperform hand-engineered ones, but training on real vehicles is unsafe, and training in realistic simulators is expensive and still suffers from a gap between simulation and reality. This gap is especially severe in unstructured environments like off-road terrain, where the visual and physical diversity makes realistic simulation difficult. The paper's goal is to eliminate that gap by having both the simulator and the real vehicle share the same simplified semantic representation of the world, so a policy trained in simulation can be deployed directly on real hardware.

Key Contributions

1. A semantic mid-level representation (MLR) simulator for unstructured environments that sidesteps the need to model realistic textures, geometry, and sensor physics. Maps are generated procedurally with Perlin noise, and the vehicle is modelled with a simple bicycle model, reducing simulation development effort substantially.

2. MILER, a zero-shot sim-to-real framework that pairs the MLR simulator with a BEVFusion-based perception stack on the real vehicle, producing bird's-eye-view inputs that match the simulator's format. No retraining or vehicle-specific policy adaptation is required.

3. An extension of trajectory-alignment sim-to-real transfer from vehicle dynamics to perception. Instead of applying the policy's actions directly to the real car, a virtual vehicle runs the policy in the MLR simulator while longitudinal and lateral controllers make the real car track the virtual one.

4. An extensive real-world evaluation on two different vehicles, covering 17.3 km of driving on a 3.0 km track with obstacles, hairpin curves, off-road sections, and speeds up to 33.6 km/h, which the authors claim is the first demonstration above 30 km/h for a vehicle-sized RL-based autonomous driving system in unstructured terrain.

Main Findings

• Zero-shot transfer works on multiple vehicles: The same trained policy was deployed on two vehicles with different wheelbases (2.60 m and 2.86 m), masses (1.8 t vs. 2.5 t), and actuator limits, with no vehicle-specific retraining beyond the wheelbase. Aggregate metrics were comparable (mean speeds of 4.36 and 4.19 m/s; max speeds of 9.33 and 9.02 m/s).

• High-speed driving without oscillations: The policy reached 33.6 km/h on a straight section with no notable oscillations, a marked improvement over comparable off-road RL work such as TADPO (roughly 12.3 km/h mean).

• Trajectory alignment is essential: An ablation that removed the virtual-vehicle alignment mechanism and applied control outputs directly caused strong oscillations and delayed maneuvers; the vehicle failed to navigate the first curve of the track.

• Perception differences explain most of the residual gap: When both vehicles were given identical BEVs derived from an HD map, their behavior converged to a mean velocity difference of only 0.150 m/s. With the deployed BEVFusion BEVs, the difference was 0.317 m/s, indicating that perception variation is the dominant remaining discrepancy between vehicles.

• Terrain-sensitive behavior emerges from the reward: The agent slowed down automatically in tall-grass semi-drivable areas and next to obstacles, and accelerated again on drivable surfaces, matching the intended reward shaping.

• Limitations observed: The agent did not perform human-like swing-out maneuvers before hairpin curves, sometimes approached obstacles hastily, and its maximum velocity plateaued even when the target exceeded 40 km/h, likely due to the limited 60 m BEV range relative to required braking distances (21.4 m at 30 km/h).

Methodology in Plain English

The researchers built a deliberately simplified simulator. Rather than rendering realistic camera or LiDAR data, it represents the world as a grid of cells labeled with four semantic classes: drivable (pavement, gravel), semi-drivable (grass), non-drivable (obstacles, parked cars), and unknown. Maps are generated procedurally by overlaying noise patterns, then a connected graph of waypoints is created through drivable regions. The vehicle is a simple bicycle model, avoiding the need to simulate detailed dynamics, friction, or sensor physics.

Training uses Proximal Policy Optimization with several adaptations: the policy outputs discrete actions (jerk and steering-rate change, each discretized into 5 bins), curriculum learning ramps up penalties slowly, the discount factor is set to 1 to encourage patient, non-risky driving, and states are normalized. The reward centers on progress along target waypoints, modulated by target velocity, heading error, smoothness, and safety limits based on terrain and obstacle proximity. Training took roughly 68 hours on an A100 GPU for one billion samples.

On the real vehicle, camera and LiDAR data are fused by BEVFusion into a BEV that matches the simulator's format. The key trick is that the policy's actions are not sent to the real car's actuators. Instead, a virtual vehicle in the simulator executes those actions, and a Stanley controller plus longitudinal controller steer the real vehicle to follow the virtual one. This decouples the imperfect real dynamics from the policy's assumptions. The perception BEV is transformed into the virtual vehicle's frame and circularly cropped to avoid artifacts at the edges.

The whole stack runs on an NVIDIA Jetson AGX Orin, with the policy exported to ONNX and compiled to TensorRT, giving a mean inference time of 1.9 ms.

Why This Matters

Impact on research: The paper shows that high-fidelity simulators are not strictly necessary for sim-to-real transfer in unstructured driving. By choosing a shared abstract representation as the interface between simulation and reality, the authors reduce engineering effort and eliminate a large class of domain-gap problems. This reframes the sim-to-real challenge from "make simulation look real" to "make simulator and reality agree on a representation."

Real-world applications:

• Off-road autonomous navigation for military, agricultural, or construction vehicles operating on mixed terrain.
• Planetary rovers, where realistic simulation of unknown terrain is impractical and slow speeds permit simplified dynamics.
• Search-and-rescue vehicles that must traverse grass, gravel, and debris without detailed maps.
• Any domain where RL policies must transfer from simulation to hardware with minimal per-platform tuning, such as warehouse or logistics robots.

Industry relevance: The demonstration that a single policy runs on two structurally different vehicles with no per-vehicle retraining, and that it runs entirely on an embedded Jetson platform, points to a practical path for fleet deployment where one trained model serves multiple platforms. The reduced simulator-engineering cost also lowers the barrier to entry for companies that cannot afford large-scale photorealistic simulation pipelines.

Future Directions

• Improved exploration: Even after a billion samples, the agent fails to discover sophisticated maneuvers like swinging out before a hairpin. Better exploration techniques or human-demonstration data could unlock more expert-like behavior.

• Extending BEV range: The 60 m perception horizon limits maximum safe speed. Larger BEV coverage or hierarchical representations could enable higher velocities.

• Dynamic agents: The current system handles only static obstacles. Incorporating moving vehicles or pedestrians would require both perception and reward changes.

• Better freedom for the planner: The tension between following target waypoints and deviating to avoid obstacles suggests a need for planners that can reason about partially observable terrain without drifting into areas that later turn out to be impassable.

• Robustness under adverse conditions: Vehicle B stopped once during rain and darkness due to over-conservative obstacle prediction, indicating that perception robustness under poor lighting remains an open problem.

Target Audience

This paper is most valuable to robotics and autonomous driving researchers working on sim-to-real transfer, reinforcement learning for control, and off-road or unstructured navigation. It is also relevant to engineers building perception and planning stacks for non-highway vehicles, and to practitioners interested in embedded deployment of learned policies. Readers should have a working understanding of reinforcement learning, bird's-eye-view perception, and basic vehicle dynamics to fully appreciate the technical choices; the general strategy, however, is accessible to a broader technical audience.

Authors’ abstract

Reinforcement learning constitutes a promising approach owing to its potential for superhuman performance and self-learned policies. However, its application to real-world autonomous driving remains scarce, particularly in unstructured environments, because of the challenges associated with sim-to-real transfer for unstructured environments. In this work, we present MILER, an end-to-end policy framework with zero-shot sim-to-real transfer. During offline training, we employ a custom semantic mid-level representation (MLR) simulator and train the policy network using reinforcement learning, with its control outputs applied directly to a bicycle model. During deployment on the real vehicle, camera and LiDAR data are processed by BEVFusion to generate a semantic bird's-eye-view representation consistent with that of the MLR simulator. The actions generated by the policy network are not applied directly to the real vehicle. Instead, we employ a trajectory-alignment strategy that enables zero-shot sim-to-real transfer of both perception and control. We extensively evaluate the proposed framework on a diverse test track comprising numerous challenges, including various obstacles, hairpin curves, velocities of up to 33.6 km/h, and off-road sections. In total, we drove 17.3 km with two different vehicles on a 3.0 km test track without human intervention, thereby demonstrating the effectiveness of our approach. Furthermore, the entire software stack runs on a Jetson AGX Orin.