Distributional Policy Optimization: An Alternative Approach For Continuous Control

Abstract

We identify a fundamental problem in policy gradient-based methods in continuous control. As policy gradient methods require the agent's underlying probability distribution, they limit policy representation to parametric distribution classes. We show that optimizing over such sets results in local movement in the action space and thus convergence to sub-optimal solutions. We suggest a novel distributional framework, able to represent arbitrary distribution functions over the continuous action space. Using this framework, we construct a generative scheme, trained using an off-policy actor-critic paradigm, which we call the Generative Actor Critic (GAC). Compared to policy gradient methods, GAC does not require knowledge of the underlying probability distribution, thereby overcoming these limitations. Empirical evaluation shows that our approach is comparable and often surpasses current state-of-the-art baselines in continuous domains.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytessler2019distributional

Related papers

• Policy Search By Target Distribution Learning For Continuous Control (2019)3.58
• Policy Optimization In A Noisy Neighborhood: On Return Landscapes In Continuous Control (2023)0.00
• Proximal Policy Optimization With Continuous Bounded Action Space Via The Beta Distribution (2021)0.00
• Generative Actor-critic: An Off-policy Algorithm Using The Push-forward Model (2021)0.00
• Policy Gradient Algorithms Implicitly Optimize By Continuation (2023)0.00
• Off-policy Policy Gradient With State Distribution Correction (2019)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Beyond Distributions: Geometric Action Control For Continuous Reinforcement Learning (2025)0.00