Truly Proximal Policy Optimization
2019 · Yuhui Wang, Hao He, Chao Wen, et al.
Abstract
Proximal policy optimization (PPO) is one of the most successful deep reinforcement-learning methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, its optimization behavior is still far from being fully understood. In this paper, we show that PPO could neither strictly restrict the likelihood ratio as it attempts to do nor enforce a well-defined trust region constraint, which means that it may still suffer from the risk of performance instability. To address this issue, we present an enhanced PPO method, named Truly PPO. Two critical improvements are made in our method: 1) it adopts a new clipping function to support a rollback behavior to restrict the difference between the new policy and the old one; 2) the triggering condition for clipping is replaced with a trust region-based one, such that optimizing the resulted surrogate objective function provides guaranteed monotonic improvement of the ultimate policy performance. It seems, by adher
Authors
(none)
Tags
Stats
Related papers
- Simple Policy Optimization (2024)0.00
- Proximal Policy Optimization Via Enhanced Exploration Efficiency (2020)13.70
- Proximal Policy Optimization Algorithms (2017)0.00
- Policy Optimization With Penalized Point Probability Distance: An Alternative To Proximal Policy Optimization (2018)0.00
- Revisiting Design Choices In Proximal Policy Optimization (2020)0.00
- Cim-ppo:proximal Policy Optimization With Liu-correntropy Induced Metric (2021)0.00
- A Theoretical Analysis Of Optimistic Proximal Policy Optimization In Linear Markov Decision Processes (2023)0.00
- KIPPO: Koopman-inspired Proximal Policy Optimization (2025)0.00