Sample Complexity Of Estimating The Policy Gradient For Nearly Deterministic Dynamical Systems

Abstract

Reinforcement learning is a promising approach to learning robotics controllers. It has recently been shown that algorithms based on finite-difference estimates of the policy gradient are competitive with algorithms based on the policy gradient theorem. We propose a theoretical framework for understanding this phenomenon. Our key insight is that many dynamical systems (especially those of interest in robotics control tasks) are nearly deterministic -- i.e., they can be modeled as a deterministic system with a small stochastic perturbation. We show that for such systems, finite-difference estimates of the policy gradient can have substantially lower variance than estimates based on the policy gradient theorem. Finally, we empirically evaluate our insights in an experiment on the inverted pendulum.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybastani2019sample

Related papers

• An Analysis Of Measure-valued Derivatives For Policy Gradients (2022)2.26
• An Empirical Analysis Of Measure-valued Derivatives For Policy Gradients (2021)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Non-parametric Stochastic Policy Gradient With Strategic Retreat For Non-stationary Environment (2022)0.00
• Model-free Policy Learning With Reward Gradients (2021)0.00
• Deterministic Policy Gradient For Reinforcement Learning With Continuous Time And State (2025)0.00
• Stochastic Variance Reduction For Policy Gradient Estimation (2017)0.00
• Smoothing Policies And Safe Policy Gradients (2019)7.50