Discretizing Continuous Action Space With Unimodal Probability Distributions For On-policy Reinforcement Learning

Abstract

For on-policy reinforcement learning, discretizing action space for continuous control can easily express multiple modes and is straightforward to optimize. However, without considering the inherent ordering between the discrete atomic actions, the explosion in the number of discrete actions can possess undesired properties and induce a higher variance for the policy gradient estimator. In this paper, we introduce a straightforward architecture that addresses this issue by constraining the discrete policy to be unimodal using Poisson probability distributions. This unimodal architecture can better leverage the continuity in the underlying continuous action space using explicit unimodal probability distributions. We conduct extensive experiments to show that the discrete policy with the unimodal probability distribution provides significantly faster convergence and higher performance for on-policy reinforcement learning algorithms in challenging control tasks, especially in highly compl

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhu2024discretizing

Related papers

• Categorical Policies: Multimodal Policy Learning And Exploration In Continuous Control (2025)0.00
• Adaptive Discretization For Episodic Reinforcement Learning In Metric Spaces (2019)2.26
• Proximal Policy Optimization With Continuous Bounded Action Space Via The Beta Distribution (2021)0.00
• Accuracy Of Discretely Sampled Stochastic Policies In Continuous-time Reinforcement Learning (2025)0.00
• Marginal Policy Gradients: A Unified Family Of Estimators For Bounded Action Spaces With Applications (2018)0.00
• Beyond Distributions: Geometric Action Control For Continuous Reinforcement Learning (2025)0.00
• Policy Prediction Network: Model-free Behavior Policy With Model-based Learning In Continuous Action Space (2019)0.00
• Distributional Policy Optimization: An Alternative Approach For Continuous Control (2019)0.00