Variance Reduction For Policy-gradient Methods Via Empirical Variance Minimization

Abstract

Policy-gradient methods in Reinforcement Learning(RL) are very universal and widely applied in practice but their performance suffers from the high variance of the gradient estimate. Several procedures were proposed to reduce it including actor-critic(AC) and advantage actor-critic(A2C) methods. Recently the approaches have got new perspective due to the introduction of Deep RL: both new control variates(CV) and new sub-sampling procedures became available in the setting of complex models like neural networks. The vital part of CV-based methods is the goal functional for the training of the CV, the most popular one is the least-squares criterion of A2C. Despite its practical success, the criterion is not the only one possible. In this paper we for the first time investigate the performance of the one called Empirical Variance(EV). We observe in the experiments that not only EV-criterion performs not worse than A2C but sometimes can be considerably better. Apart from that, we also prove

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykaledin2022variance

Related papers

• Trajectory-wise Control Variates For Variance Reduction In Policy Gradient Methods (2019)0.00
• Stochastic Variance Reduction For Policy Gradient Estimation (2017)0.00
• Action-depedent Control Variates For Policy Optimization Via Stein's Identity (2017)0.00
• An Empirical Analysis Of Measure-valued Derivatives For Policy Gradients (2021)0.00
• On The Convergence And Sample Efficiency Of Variance-reduced Policy Gradient Method (2021)0.00
• Sample Efficient Policy Gradient Methods With Recursive Variance Reduction (2019)0.00
• Variance Reduction For Policy Gradient With Action-dependent Factorized Baselines (2018)0.00
• An Analysis Of Measure-valued Derivatives For Policy Gradients (2022)2.26