Convergence And Optimality Of Policy Gradient Methods In Weakly Smooth Settings

Abstract

Policy gradient methods have been frequently applied to problems in control and reinforcement learning with great success, yet existing convergence analysis still relies on non-intuitive, impractical and often opaque conditions. In particular, existing rates are achieved in limited settings, under strict regularity conditions. In this work, we establish explicit convergence rates of policy gradient methods, extending the convergence regime to weakly smooth policy classes with \(L_2\) integrable gradient. We provide intuitive examples to illustrate the insight behind these new conditions. Notably, our analysis also shows that convergence rates are achievable for both the standard policy gradient and the natural policy gradient algorithms under these assumptions. Lastly we provide performance guarantees for the converged policies.

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyzhang2021convergence

Related papers

• Global Convergence Of Policy Gradient Methods In Reinforcement Learning, Games And Control (2023)0.00
• On The Theory Of Policy Gradient Methods: Optimality, Approximation, And Distribution Shift (2019)0.00
• On The Convergence Of Discounted Policy Gradient Methods (2022)0.00
• Linear Convergence Of A Policy Gradient Method For Some Finite Horizon Continuous Time Control Problems (2022)0.00
• Policy Gradient In Partially Observable Environments: Approximation And Convergence (2018)0.00
• Policy Gradient Using Weak Derivatives For Reinforcement Learning (2020)0.00
• Global Convergence Using Policy Gradient Methods For Model-free Markovian Jump Linear Quadratic Control (2021)0.00
• On The Linear Convergence Of Natural Policy Gradient Algorithm (2021)0.00