Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States

AI summary

Overview

Research area: Deep reinforcement learning for continuous control, specifically action-smoothing methods for control policies.

Technical level: Intermediate. The paper combines reinforcement learning practice with Lipschitz-continuity theory; the underlying idea is intuitive, but the paper contains formal definitions, assumptions, a lemma and a theorem, and a multi-term loss.

Scope: The paper proposes a new loss-penalty action-smoothing method, ASAP, that defines "similar states" from the environment's own transition distribution (instead of synthetic or heuristic neighborhoods) and evaluates it on Gymnasium/MuJoCo and Isaac-Lab control benchmarks with PPO and SAC.

What This Paper Is About

Deep reinforcement learning policies for control tasks tend to produce high-frequency action oscillations, which make them hard to deploy on real hardware. Prior loss-based fixes reduce these oscillations by forcing actions to be consistent across "similar states," but they define those similar states using fixed distributions or heuristic perturbations that do not match the real system dynamics. This paper instead defines similar states directly from the transition distribution of the environment, and builds a training loss that aligns a policy's action with the action predicted from the preceding state.

Key Contributions

1. A new action-smoothing method, ASAP, which combines an action constraint based on transition-induced similar states with a penalty on second-order action variations.
2. A transition-based definition of similar states that uses the distribution of next states reached from the previous state, together with a theoretical proof (Lemma 1) that these states form a bounded neighborhood suitable for imposing a local Lipschitz constraint.
3. A composite Lipschitz formulation (Theorem 1) showing that the policy-transition composition is locally Lipschitz in the noise variable, which is then reduced to a concrete spatial loss term for the actor.
4. Extensive experiments across Gymnasium and Isaac-Lab tasks showing that ASAP suppresses action oscillations while preserving or improving policy performance relative to existing methods, and that it stacks with architectural and temporal smoothing methods.

Main Findings

• Predictor-based fine-tuning works in isolation. Fine-tuning a SAC policy with a predictor trained by supervised learning on a fixed replay buffer consistently reduced the smoothness score, with an average improvement of 15.1%. Hopper and Walker showed reductions of up to 16.9% and 14.4% respectively, while cumulative return stayed stable.

• PPO results (Gymnasium). ASAP achieved the highest cumulative return in 3 out of 6 environments and the best smoothness score in 5 environments. On average it reduced the smoothness score sm by 43.3% compared to the PPO baseline, with particularly large improvements in Hopper (89.5%) and Walker (80.4%). CAPS and L2C2 reduced sm in Hopper by 83.5% and 21.3%, respectively — both smaller reductions than ASAP, and both with larger drops in return.

• SAC results (Gymnasium). ASAP achieved the highest return in 3 out of 6 environments and the best sm in 4 environments, reducing sm by an average of 27.9% over the SAC baseline while maintaining a similar level of return. It consistently matched or outperformed Grad-CAPS on both metrics.

• Robotic scenarios (Isaac-Lab). On four Isaac-Lab tasks, ASAP improved or maintained cumulative return while lowering the smoothness score. In the Franka reach task it improved return and decreased sm by 31.3%. The mean end-effector-to-target error decreased by 30.3%, from 0.90 cm to 0.62 cm. In Lift-Cube, however, the variance of the return increased, which the authors attribute to oscillation suppression constraining exploration and causing inadequate learning in a subset of random seeds.

• Compatibility with architectural methods. Combined with LipsNet under SAC, ASAP reduced sm by 54.5% on Hopper and 47.0% on Walker relative to LipsNet. Compared to LipsNet+CAPS, it achieved 41.4% and 27.1% lower sm on Hopper and Walker, respectively.

• Ablation — spatial term. With the temporal term fixed to Grad-CAPS, the CAPS spatial term reduced sm by 15.1% but caused a significant return drop; the L2C2 term reduced sm by only 4.9%; the ASAP spatial term reduced sm by 30.9% while stably maintaining return.

• Ablation — temporal term. With the spatial term fixed to ASAP, adding the CAPS temporal regularization improved sm by 36.6% in Hopper and 55.3% in Walker with consistent return gains. Adding the Grad-CAPS temporal term improved sm by 25.7% in Hopper and 36.2% in Walker, with return also improving except for a marginal decline in Hopper.

• Stated limitation. Because ASAP defines similar states from observation noise, high noise levels may enlarge the resulting neighborhood and weaken the local Lipschitz constraint and spatial regularization; this can be mitigated by adjusting the spatial loss weight.

Methodology in Plain English

The core observation is that a control policy oscillates because it reacts too strongly to tiny differences in the state. The natural fix is to force the policy to output similar actions for states that are "close together." The problem is deciding what "close together" means.

The authors say: two states are similar if they could both be reached from the same previous state. Because the environment itself generates those next states — through its own transition dynamics and its noise — this definition automatically reflects what the system actually does, rather than an assumed Gaussian ball or an ad-hoc projection rule.

The authors then show mathematically that this definition has a useful property: if the transition function is locally Lipschitz in the noise (Assumption 1) and the noise is bounded (Assumption 2), then any two states drawn from the same predecessor are guaranteed to be within a distance of twice the local noise-sensitivity constant times the noise bound. That means the set of similar states is a genuinely bounded neighborhood, so imposing a local Lipschitz constraint there is justified. Extending this to the composition of the policy with the transition function gives a bound on how much the policy's output can change across similar states.

That theoretical bound translates into a simple training loss. The policy's action at the current state is compared against the action predicted by a separate predictor network from the preceding state, and the predictor is trained to match the policy's actual output — the two directions are separated and each is given its own weight, so the target values can be fixed with a stop-gradient. On top of this spatial term, a temporal term borrowed from Grad-CAPS penalizes the second-order difference between consecutive actions, which directly attacks the high-frequency part of the oscillation. All of this is added to the standard reinforcement learning actor loss, so it works with algorithms such as PPO and SAC without changing the actor's architecture at inference time. In implementation, the predictor is added as an extra prediction head alongside the action head in the actor's MLP.

Evaluation uses two metrics: cumulative return for task performance, and a smoothness score computed from the FFT frequency spectrum of the action signal, where higher values mean more high-frequency content.

Why This Matters

Impact on research. Existing loss-penalty smoothing methods rest on similar-state definitions that are heuristic or synthetic, which the authors argue undermines both theoretical guarantees and empirical performance. By tying the similar-state definition to the environment's own transition kernel, this work gives the smoothing objective a principled footing and shows it can be layered on top of both architectural methods (LipsNet) and temporal methods (CAPS, Grad-CAPS) without interference. This shifts the design question from "how should we invent a neighborhood?" to "how should we read the neighborhood off the data we already collect?"

Real-world applications:

• Industrial and service robot manipulators, where jerky joint commands accelerate component wear and shorten actuator lifespan.
• Legged locomotion and rough-terrain robots, such as the Anymal-Velocity-Rough task evaluated here, where smooth joint commands matter for stability and hardware longevity.
• Precision manipulation tasks like the Franka reach task, where the paper reports a reduction in end-effector targeting error from 0.90 cm to 0.62 cm.
• Human-facing autonomous systems, where non-smooth motion degrades user experience and can create safety concerns.

Industry relevance. Because ASAP is a loss-only modification, it keeps inference identical to the base policy — no extra computation at deployment. That makes it attractive for teams that have already trained PPO or SAC policies and want to reduce actuator wear or improve motion quality without paying an inference-time cost or redesigning their network.

Future Directions

• Robustness under high observation noise. The authors explicitly note that high noise levels can enlarge the transition-induced neighborhood and weaken the spatial regularization. Characterizing when this happens, and how to set the spatial loss weight automatically rather than by hand, is an open question.
• Exploration side effects. In Lift-Cube the return variance increased, attributed to smoothing constraints limiting exploration. Understanding and mitigating this trade-off between action smoothness and exploration is unresolved.
• Beyond the predictor's "moving target." The paper notes that predictor training faces a moving-target problem and suggests more parallel environments and a lower predictor loss weight as remedies. Whether soft updates or other stabilization schemes could work in on-policy settings like PPO is left open.
• Extending to other similar-state definitions and dynamics. Since the approach depends on Assumptions 1 and 2 (locally Lipschitz transitions, bounded noise), testing how far the theory holds under discontinuous contacts or heavily stochastic real-world dynamics would clarify its practical envelope.

Target Audience

Reinforcement learning researchers working on continuous control and policy regularization, robotics engineers deploying learned controllers on physical hardware, and practitioners using PPO or SAC who need smoother actuator commands without changing inference-time cost. The paper is also relevant to readers interested in Lipschitz-based regularization, since it connects a distributional definition of similar states to a local Lipschitz constraint. A working familiarity with reinforcement learning and basic Lipschitz continuity concepts is assumed.

Authors’ abstract

Deep reinforcement learning has proven to be a powerful approach to solving control tasks, but its characteristic high-frequency oscillations make it difficult to apply in real-world environments. While prior methods have addressed action oscillations via architectural or loss-based methods, the latter typically depend on heuristic or synthetic definitions of state similarity to promote action consistency, which often fail to accurately reflect the underlying system dynamics. In this paper, we propose a novel loss-based method by introducing a transition-induced similar state. The transition-induced similar state is defined as the distribution of next states transitioned from the previous state. Since it utilizes only environmental feedback and actually collected data, it better captures system dynamics. Building upon this foundation, we introduce Action Smoothing by Aligning Actions with Predictions from Preceding States (ASAP), an action smoothing method that effectively mitigates action oscillations. ASAP enforces action smoothness by aligning the actions with those taken in transition-induced similar states and by penalizing second-order differences to suppress high-frequency oscillations. Experiments in Gymnasium and Isaac-Lab environments demonstrate that ASAP yields smoother control and improved policy performance over existing methods.