Enhancing Control Policy Smoothness By Aligning Actions With Predictions From Preceding States

Abstract

Deep reinforcement learning has proven to be a powerful approach to solving control tasks, but its characteristic high-frequency oscillations make it difficult to apply in real-world environments. While prior methods have addressed action oscillations via architectural or loss-based methods, the latter typically depend on heuristic or synthetic definitions of state similarity to promote action consistency, which often fail to accurately reflect the underlying system dynamics. In this paper, we propose a novel loss-based method by introducing a transition-induced similar state. The transition-induced similar state is defined as the distribution of next states transitioned from the previous state. Since it utilizes only environmental feedback and actually collected data, it better captures system dynamics. Building upon this foundation, we introduce Action Smoothing by Aligning Actions with Predictions from Preceding States (ASAP), an action smoothing method that effectively mitigates ac

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykwak2026enhancing

Related papers

• Addressing Action Oscillations Through Learning Policy Inertia (2021)7.81
• Specialized Deep Residual Policy Safe Reinforcement Learning-based Controller For Complex And Continuous State-action Spaces (2023)4.52
• No Prior Mask: Eliminate Redundant Action For Deep Reinforcement Learning (2023)1.81
• Extremum-seeking Action Selection For Accelerating Policy Optimization (2024)0.00
• Dreamsmooth: Improving Model-based Reinforcement Learning Via Reward Smoothing (2023)0.00
• Action-depedent Control Variates For Policy Optimization Via Stein's Identity (2017)0.00
• Handling Cost And Constraints With Off-policy Deep Reinforcement Learning (2023)0.00
• A Unifying Framework For Action-conditional Self-predictive Reinforcement Learning (2024)0.00