Energy-Aware Path Following: Comparative Analysis of Reinforcement Learning and NMPC for Electric Vehicles

AI summary

Overview

Research area: Robotics / autonomous driving control — specifically path-following control for electric vehicles (EVs), comparing optimization-based control (Nonlinear Model Predictive Control, NMPC) against reinforcement learning (Proximal Policy Optimization, PPO) under a shared electric-energy model.

Technical level: Advanced. The paper assumes familiarity with the Frenet frame, nonlinear vehicle kinematics, nonlinear programming, pole placement, and deep RL (PPO, GAE, curriculum learning).

Scope (one sentence): The paper benchmarks four path-following controllers — NMPC, PPO, a gain-scheduled Ackermann state-feedback controller (PID-SF), and a Stanley geometric baseline — on one shared Frenet-frame kinematic vehicle model with the VT-CPEM electric energy model, quantifying both tracking error and regenerative energy recovery.

Note on sourcing: The supplied paper content is truncated. Sections after Phase 1 of the PPO curriculum are cut off, so detailed per-controller result tables from Sections 5.x are not present in the content provided; where a quantity is not reported there, this summary says so explicitly rather than supplying a value.

What This Paper Is About

Path-following controllers for autonomous vehicles are usually judged only on how closely they track a reference path, and they typically exclude any energy model entirely. For EVs, whose driving range is limited, this leaves out a first-order design criterion: how much energy the vehicle actually consumes and how much it can recover through regenerative braking. The paper's core problem is that learned (RL) controllers are reactive rather than predictive, so they brake late and recover less regenerative energy than an NMPC that plans ahead. The goal is to close that regenerative-energy gap by giving the RL policy preview information about upcoming curvature and reference speed, plus an energy-aware reward, and then to measure how much of the gap this closes without sacrificing tracking accuracy.

Key Contributions

1. A unified benchmarking setup for the tracking–regeneration trade-off. Four controllers (NMPC, PPO, PID-SF, Stanley) are compared under one common Frenet frame-based kinematic vehicle model, one curvature-aware reference velocity profile, and one validated energy model (VT-CPEM) with explicit regenerative braking. The closed-loop plant is the same nonlinear Frenet model for every controller, so no controller has a model advantage.

2. Preview-augmented PPO with an energy-based reward. The base observation is an 8-dimensional normalized vector; the preview variant augments it with curvature and reference speed at three lookahead distances (5 m, 15 m, 30 m), yielding a 14-dimensional observation. A signed energy reward term replaces a simplified unsigned acceleration penalty, so braking earns a negative (rewarded) contribution instead of being discouraged.

3. A five-phase curriculum-learning training regime and a 2×2 ablation. Training moves from a straight track (Phase 0) to progressively more complex curved and speed-regulation tasks, with a dedicated reward-design study over base versus preview observations and simplified versus energy rewards.

4. Transfer and energy analysis. The unmodified policy is transferred to unseen tracks (ISO 3888-1 lane-change, chicane, randomly generated parameterized splines, and a ±3° graded road) and to a dynamic single-track plant with linear tires, with energy consumption and recovered energy reported alongside tracking error, plus a warm-starting study for the NMPC.

Main Findings

• Preview plus energy reward closes 76% of NMPC's energy gap. The paper's headline result is that the preview observation combined with the energy-based reward closes 76% of the energy gap relative to NMPC.

• The same change simultaneously improves tracking. Lateral deviation RMSE improved from 0.459 m to 0.314 m when the preview/energy modification was applied.

• Zero failures across 30 initial conditions. The study reports 0/30 failures over Latin Hypercube Sampling (LHS) initial conditions, with a Wilcoxon test giving p < 0.0001 and effect size d = 0.826.

• The base (reactive) PPO underperforms NMPC. With a strictly local observation, a step-level reward cannot anticipate a deceleration ramp several seconds ahead, so the base PPO fared significantly worse than the NMPC. The residual gap is described as consistent with a finite preview window of 30 m, equivalent to 3.75 s at v_max = 8 m/s, versus the NMPC's 5 s horizon, together with value-function approximation error.

• The terminal deceleration ramp dominates regenerative recovery. The smooth terminal stopping profile sqrt(2·a_lng·max(s_max − s, 0)) contributes the largest proportion of sustained negative power recovered by the energy model.

• Warm-starting speeds up NMPC. Initializing IPOPT with the shifted previous solution reduced average solve time from 124.4 ms to 90.6 ms, a 1.37× speedup.

• Policy transfer succeeds in simulation. The unmodified policy transfers to an ISO 3888-1 lane-change, a chicane, randomly generated parameterized splines, and a ±3° graded road, and transfers zero-shot to a dynamic single-track vehicle model with linear tires with acceptable initial performance that was optimized after brief fine-tuning.

• The "simplified" reward variant is not energy-aware. Because it penalizes |u₂| irrespective of sign, it discourages the very deceleration from which energy is recovered. With the Phase-4 weights, its effective weight on |u₂| is 0.45 rather than the nominal 0.15.

• No singularity is encountered on the benchmark S-curve. With max |d| = 4.9 m and κ_max = 0.1101 m⁻¹, the maximum of |d·κ(s)| is approximately 0.54, below 1.0, so no division-by-zero in the Frenet arclength equation occurs prior to episode termination.

• Results for individual controllers beyond the highlights are not reported in the supplied content. The truncated text ends within the curriculum description, so per-controller energy-consumption and recovered-energy tables, and exact solution times for controllers other than the NMPC warm-start study, are not available here.

Methodology in Plain English

Shared vehicle and energy model. The car is described in the Frenet frame — coordinates relative to the reference path — using five states: distance travelled along the path (s), lateral deviation (d), heading error (α), steering angle (δ), and speed (v). The two control inputs are steering rate and acceleration. The model is discretized with fourth-order Runge–Kutta integration at a 0.1 s sampling period. Energy comes from VT-CPEM: a road-load force combining inertial, aerodynamic drag, rolling resistance, and road-grade terms, converted to wheel power and then to battery power through separate traction and regeneration efficiencies. Battery power splits into traction energy, recovered energy, and a small controller (ECU) energy term. Regeneration is assigned based on the sign of mechanical wheel power, not acceleration sign, so gentle decelerations where drag and rolling resistance keep traction force positive are correctly excluded from recovery.

Reference speed. A curvature-aware speed profile takes the minimum of a maximum speed, a lateral-acceleration-limited speed, and a terminal stopping ramp based on the comfort deceleration limit, then a three-step moving-average filter suppresses spline-derivative noise.

The four controllers.

• NMPC: minimizes a normalized cost over lateral deviation, heading error, speed error, and control effort, over a 50-step (5 s) horizon, solved with IPOPT at a maximum of 20 iterations and JIT-compiled via CasADi for real time.
• PID-SF: linearizes the lateral dynamics at a constant 5 m/s, places poles at {−1.5, −2.5, −5.0}, gain-schedules the resulting gains with speed, adds a curvature feedforward term, and pairs it with a longitudinal PI controller.
• Stanley: the classic geometric controller, with the cross-track gain retuned to account for the Frenet frame measuring deviation at the rear axle rather than the front axle, sharing the same longitudinal PI.
• PPO: trained with curriculum learning over five phases starting from a straight track, using either the base 8-dimensional observation or the 14-dimensional preview observation, and either a simplified acceleration penalty or a signed energy reward.

Evaluation. Controllers are run on a dynamic single-track plant with linear tires and on standardized tracks, with an additional LQR baseline for reference. Failures are counted across LHS-sampled initial conditions, and a 2×2 ablation isolates the effect of preview observation and energy reward.

Why This Matters

Impact on research. Most path-following benchmarks report tracking error and omit energy entirely, which is a poor proxy for deployment value in EVs. This paper argues that tracking and energy recovery are not independent objectives — naive acceleration penalties actively suppress regenerative braking — and that predictive information can be moved from online optimization into offline learning through preview observations. That reframing gives future work a concrete baseline: how much of an NMPC's energy advantage can a learned policy recover, and how much of it must come from an explicit horizon?

Real-world applications:

• EV range management in autonomous driving stacks, where the local feedback control layer determines how much energy the terminal deceleration and corner-entry/exit maneuvers actually recover.
• Embedded deployment on vehicle compute. NMPC is reported to draw relatively higher ECU energy than quasi-instantaneous controllers, and warm-starting reduces solve time from 124.4 ms to 90.6 ms — directly relevant for meeting control-loop deadlines on automotive hardware.
• Sim-to-real transfer of learned controllers, since the policy transfers without modification to unseen tracks, a graded road, and a different (dynamic single-track) plant model.
• Urban and suburban path following, where the paper's numerical guard (ϵ_d = 10⁻⁶) and curvature corridor are described as typically sufficient for urban scenarios.

Industry relevance. The comparison is framed around a realistic EV parameter set (1500 kg, 4.0 m wheelbase, 0.30 drag coefficient, 2.2 m² frontal area, 0.65 regeneration efficiency, 5 W estimated ECU draw), and the benchmark includes a real-time constraint on the NMPC. That combination — validated energy model, real-time solver, and a learned controller evaluated on the same footing — is what an automotive controls team needs before choosing between optimization-based and learned path following.

Future Directions

1. Extend the preview window or learn a longer effective horizon. The residual energy gap is attributed partly to a 30 m preview (3.75 s at v_max = 8 m/s) versus a 5 s NMPC horizon, so longer or adaptive lookahead distances are an obvious next step, alongside reducing value-function approximation error.

2. Make regeneration efficiency deceleration-dependent. The authors deliberately treat η_r as a constant to reduce NMPC computational burden, and state that VT-CPEM defines it as deceleration-dependent. They amend this with a sensitivity analysis over a range of η_r values (in a section not included in the supplied content); a full deceleration-dependent formulation remains open.

3. Close the remaining 24% of the energy gap and validate on hardware. The paper reports only simulation experiments — transfers to unseen tracks, grade, and a tire-dynamics plant are all in simulation. Real-vehicle or hardware-in-the-loop validation is not reported.

4. Address global stability and robustness guarantees. The PID-SF stability result is explicitly local, derived from a constant-speed linearization, and not a proof of global stability for the nonlinear, curvature-varying, saturated system. Similarly, the reward-based approach relies on reward design choices rather than formal guarantees.

Target Audience

This paper is best suited to graduate students and researchers in vehicle control, autonomous driving, and robot learning who already know MPC and deep RL and want a rigorous head-to-head comparison on a shared model; to control engineers at EV and ADAS companies deciding between optimization-based and learned path-following controllers for embedded deployment; and to researchers specifically interested in energy-aware control, regenerative braking, or curriculum learning and transfer in RL-based vehicle control. Readers looking for a beginner-level introduction to path following will find the paper assumes substantial prior background, though the comparison structure is clear enough to follow for an intermediate reader willing to look up the Frenet frame and PPO.

Authors’ abstract

Path-following control strategies typically follow the bi-objective optimization dilemma: minimizing deviations from a reference path while maintaining smooth speed profiles. The latter objective is especially relevant for Electric Vehicles (EVs), since their limited driving range can be extended by recovering energy through regenerative braking, a feature that has not yet been sufficiently studied in the literature. In this work, we perform a comparative analysis of four controllers under one common Frenet frame-based kinematic vehicle model, utilizing a validated energy model (VT-CPEM) with explicit regenerative braking. Herein, we implement the following controllers: Nonlinear Model Predictive Control (NMPC), Proximal Policy Optimization (PPO), gain-scheduled Ackermann state-feedback baseline (PID-SF), and a Stanley geometric baseline. To satisfy real-time requirements, we implement the NMPC using JIT-compiled CasADi. Moreover, we train the PPO using traditional straight and S-curve tracks, after which we successfully transfer the unmodified policy to unseen tracks, including: an ISO 3888-1 lane-change, a chicane, randomly-generated parameterized-splines, and a $\pm3^\circ$ graded road. In addition, the policy transfers to a dynamic single-track vehicle model with linear tires, zero-shot with an acceptable initial performance, which was optimized after brief fine-tuning. Thereby, we demonstrate that our PPO is readily transferable to more comprehensive vehicle models. We conclude with a performance analysis of developed controllers and discuss ideas for future work.