Deep Bayesian Quadrature Policy Optimization

Abstract

We study the problem of obtaining accurate policy gradient estimates using a finite number of samples. Monte-Carlo methods have been the default choice for policy gradient estimation, despite suffering from high variance in the gradient estimates. On the other hand, more sample efficient alternatives like Bayesian quadrature methods have received little attention due to their high computational complexity. In this work, we propose deep Bayesian quadrature policy gradient (DBQPG), a computationally efficient high-dimensional generalization of Bayesian quadrature, for policy gradient estimation. We show that DBQPG can substitute Monte-Carlo estimation in policy gradient methods, and demonstrate its effectiveness on a set of continuous control benchmarks. In comparison to Monte-Carlo estimation, DBQPG provides (i) more accurate gradient estimates with a significantly lower variance, (ii) a consistent improvement in the sample complexity and average return for several deep policy gradient

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytej2020deep

Related papers

• Bayesian Policy Gradients Via Alpha Divergence Dropout Inference (2017)0.00
• Alternating Optimisation And Quadrature For Robust Control (2016)7.16
• Parameter-free Reduction Of The Estimation Bias In Deep Reinforcement Learning For Deterministic Policy Gradients (2021)0.00
• Stochastic Variance Reduction For Policy Gradient Estimation (2017)0.00
• Q-prop: Sample-efficient Policy Gradient With An Off-policy Critic (2016)0.00
• Policy Gradient Optimization Of Thompson Sampling Policies (2020)0.00
• Quantile-based Deep Reinforcement Learning Using Two-timescale Policy Gradient Algorithms (2023)0.00
• Optimal Estimation Of Off-policy Policy Gradient Via Double Fitted Iteration (2022)0.00