PAGE-PG: A Simple And Loopless Variance-reduced Policy Gradient Method With Probabilistic Gradient Estimation

Abstract

Despite their success, policy gradient methods suffer from high variance of the gradient estimate, which can result in unsatisfactory sample complexity. Recently, numerous variance-reduced extensions of policy gradient methods with provably better sample complexity and competitive numerical performance have been proposed. After a compact survey on some of the main variance-reduced REINFORCE-type methods, we propose ProbAbilistic Gradient Estimation for Policy Gradient (PAGE-PG), a novel loopless variance-reduced policy gradient method based on a probabilistic switch between two types of updates. Our method is inspired by the PAGE estimator for supervised learning and leverages importance sampling to obtain an unbiased gradient estimator. We show that PAGE-PG enjoys a \(\mathcal\{O\}\left( \epsilon^\{-3\} \right)\) average sample complexity to reach an \(\epsilon\)-stationary solution, which matches the sample complexity of its most competitive counterparts under the same setting. A num

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygargiani2022page

Related papers

• On The Convergence And Sample Efficiency Of Variance-reduced Policy Gradient Method (2021)0.00
• Sample Efficient Policy Gradient Methods With Recursive Variance Reduction (2019)0.00
• PC-PG: Policy Cover Directed Exploration For Provable Policy Gradient Learning (2020)0.00
• Efficiently Escaping Saddle Points For Policy Optimization (2023)0.00
• Stochastic Variance Reduction For Policy Gradient Estimation (2017)0.00
• Stochastic Policy Gradient Methods: Improved Sample Complexity For Fisher-non-degenerate Policies (2023)0.00
• MDPGT: Momentum-based Decentralized Policy Gradient Tracking (2021)0.00
• Settling The Variance Of Multi-agent Policy Gradients (2021)0.00