Marginal Policy Gradients: A Unified Family Of Estimators For Bounded Action Spaces With Applications

Abstract

Many complex domains, such as robotics control and real-time strategy (RTS) games, require an agent to learn a continuous control. In the former, an agent learns a policy over \(\mathbb\{R\}^d\) and in the latter, over a discrete set of actions each of which is parametrized by a continuous parameter. Such problems are naturally solved using policy based reinforcement learning (RL) methods, but unfortunately these often suffer from high variance leading to instability and slow convergence. Unnecessary variance is introduced whenever policies over bounded action spaces are modeled using distributions with unbounded support by applying a transformation \(T\) to the sampled action before execution in the environment. Recently, the variance reduced clipped action policy gradient (CAPG) was introduced for actions in bounded intervals, but to date no variance reduced methods exist when the action is a direction, something often seen in RTS games. To this end we introduce the angular policy gr

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyeisenach2018marginal

Related papers

• Clipped Action Policy Gradient (2018)0.00
• Variance Reduction For Policy Gradient With Action-dependent Factorized Baselines (2018)0.00
• Beyond Distributions: Geometric Action Control For Continuous Reinforcement Learning (2025)0.00
• All-action Policy Gradient Methods: A Numerical Integration Approach (2019)0.00
• Proximal Policy Optimization With Continuous Bounded Action Space Via The Beta Distribution (2021)0.00
• Distributions As Actions: A Unified Framework For Diverse Action Spaces (2025)0.00
• Action-depedent Control Variates For Policy Optimization Via Stein's Identity (2017)0.00
• Direct Policy Gradients: Direct Optimization Of Policies In Discrete Action Spaces (2019)0.00