Stochastic Variance Reduction For Policy Gradient Estimation

Abstract

Recent advances in policy gradient methods and deep learning have demonstrated their applicability for complex reinforcement learning problems. However, the variance of the performance gradient estimates obtained from the simulation is often excessive, leading to poor sample efficiency. In this paper, we apply the stochastic variance reduced gradient descent (SVRG) to model-free policy gradient to significantly improve the sample-efficiency. The SVRG estimation is incorporated into a trust-region Newton conjugate gradient framework for the policy optimization. On several Mujoco tasks, our method achieves significantly better performance compared to the state-of-the-art model-free policy gradient methods in robotic continuous control such as trust region policy optimization (TRPO)

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyxu2017stochastic

Related papers

• Sample Efficient Policy Gradient Methods With Recursive Variance Reduction (2019)0.00
• On The Convergence And Sample Efficiency Of Variance-reduced Policy Gradient Method (2021)0.00
• An Improved Convergence Analysis Of Stochastic Variance-reduced Policy Gradient (2019)0.00
• Variance Reduction For Policy-gradient Methods Via Empirical Variance Minimization (2022)0.00
• Variance Reduction Based Partial Trajectory Reuse To Accelerate Policy Gradient Optimization (2022)0.00
• Trajectory-wise Control Variates For Variance Reduction In Policy Gradient Methods (2019)0.00
• Variance Reduction For Policy Gradient With Action-dependent Factorized Baselines (2018)0.00
• Action-depedent Control Variates For Policy Optimization Via Stein's Identity (2017)0.00