Variance Reduction For Policy Gradient With Action-dependent Factorized Baselines

Abstract

Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent baseline for variance reduction which fully exploits the structural form of the stochastic policy itself and does not make any additional assumptions about the MDP. We demonstrate and quantify the benefit of the action-dependent baseline through both theoretical analysis as well as numerical results, including an analysis of the suboptimality of the optimal state-dependent baseline. The result is a computationally efficient policy gradient algorithm, which scales to high-dimensional control problems, as demonstrated by a synthetic 2000-dimensional target matching task. Our experimental results indicate that action-dependent baselines allow for faster learning on standard re

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywu2018variance

Related papers

• Action-depedent Control Variates For Policy Optimization Via Stein's Identity (2017)0.00
• Policy Gradient Methods For Reinforcement Learning With Function Approximation And Action-dependent Baselines (2017)0.00
• Trajectory-wise Control Variates For Variance Reduction In Policy Gradient Methods (2019)0.00
• Factored Policy Gradients: Leveraging Structure For Efficient Learning In Momdps (2021)0.00
• Stochastic Variance Reduction For Policy Gradient Estimation (2017)0.00
• Variance Reduction For Policy-gradient Methods Via Empirical Variance Minimization (2022)0.00
• On Many-actions Policy Gradient (2022)0.00
• Marginal Policy Gradients: A Unified Family Of Estimators For Bounded Action Spaces With Applications (2018)0.00