Gradient Informed Proximal Policy Optimization

Abstract

We introduce a novel policy learning method that integrates analytical gradients from differentiable environments with the Proximal Policy Optimization (PPO) algorithm. To incorporate analytical gradients into the PPO framework, we introduce the concept of an \{\alpha\}-policy that stands as a locally superior policy. By adaptively modifying the \{\alpha\} value, we can effectively manage the influence of analytical policy gradients during learning. To this end, we suggest metrics for assessing the variance and bias of analytical gradients, reducing dependence on these gradients when high variance or bias is detected. Our proposed approach outperforms baseline algorithms in various scenarios, such as function optimization, physics simulations, and traffic control environments. Our code can be found online: https://github.com/SonSang/gippo.

Stats

• citations1
• S2 citations—
• github stars27
• HF likes0
• heat score5.15
• arxiv keyson2023gradient

Code

• SonSang/gippo—

Related papers

• Proximal Policy Optimization Algorithms (2017)0.00
• KIPPO: Koopman-inspired Proximal Policy Optimization (2025)0.00
• On Proximal Policy Optimization's Heavy-tailed Gradients (2021)0.00
• ISOPO: Proximal Policy Gradients Without Pi-old (2025)0.00
• Truly Proximal Policy Optimization (2019)0.00
• Revisiting Design Choices In Proximal Policy Optimization (2020)0.00
• Proximal Policy Optimization Via Enhanced Exploration Efficiency (2020)13.70
• A Parametric Class Of Approximate Gradient Updates For Policy Optimization (2022)0.00