A Hybrid Stochastic Policy Gradient Algorithm For Reinforcement Learning

Abstract

We propose a novel hybrid stochastic policy gradient estimator by combining an unbiased policy gradient estimator, the REINFORCE estimator, with another biased one, an adapted SARAH estimator for policy optimization. The hybrid policy gradient estimator is shown to be biased, but has variance reduced property. Using this estimator, we develop a new Proximal Hybrid Stochastic Policy Gradient Algorithm (ProxHSPGA) to solve a composite policy optimization problem that allows us to handle constraints or regularizers on the policy parameters. We first propose a single-looped algorithm then introduce a more practical restarting variant. We prove that both algorithms can achieve the best-known trajectory complexity \(\mathcal\{O\}\left(\epsilon^\{-3\}\right)\) to attain a first-order stationary point for the composite problem which is better than existing REINFORCE/GPOMDP \(\mathcal\{O\}\left(\epsilon^\{-4\}\right)\) and SVRPG \(\mathcal\{O\}\left(\epsilon^\{-10/3\}\right)\) in the non-compos

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypham2020a

Related papers

• Proximal Policy Optimization Algorithms (2017)0.00
• Efficiently Escaping Saddle Points For Policy Optimization (2023)0.00
• Global Convergence Of Receding-horizon Policy Search In Learning Estimator Designs (2023)1.20
• Stochastic Variance Reduction For Policy Gradient Estimation (2017)0.00
• An Improved Convergence Analysis Of Stochastic Variance-reduced Policy Gradient (2019)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Variance Reduction Based Partial Trajectory Reuse To Accelerate Policy Gradient Optimization (2022)0.00
• Model-free Policy Learning With Reward Gradients (2021)0.00