Categorical Policies: Multimodal Policy Learning And Exploration In Continuous Control

Abstract

A policy in deep reinforcement learning (RL), either deterministic or stochastic, is commonly parameterized as a Gaussian distribution alone, limiting the learned behavior to be unimodal. However, the nature of many practical decision-making problems favors a multimodal policy that facilitates robust exploration of the environment and thus to address learning challenges arising from sparse rewards, complex dynamics, or the need for strategic adaptation to varying contexts. This issue is exacerbated in continuous control domains where exploration usually takes place in the vicinity of the predicted optimal action, either through an additive Gaussian noise or the sampling process of a stochastic policy. In this paper, we introduce Categorical Policies to model multimodal behavior modes with an intermediate categorical distribution, and then generate output action that is conditioned on the sampled mode. We explore two sampling schemes that ensure differentiable discrete latent structure

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyislam2025categorical

Related papers

• Autoregressive Policies For Continuous Control Deep Reinforcement Learning (2019)7.50
• Beyond Distributions: Geometric Action Control For Continuous Reinforcement Learning (2025)0.00
• Discretizing Continuous Action Space With Unimodal Probability Distributions For On-policy Reinforcement Learning (2024)0.00
• On The Sample Complexity And Metastability Of Heavy-tailed Policy Search In Continuous Control (2021)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Policy Search By Target Distribution Learning For Continuous Control (2019)3.58
• Marginal Policy Gradients: A Unified Family Of Estimators For Bounded Action Spaces With Applications (2018)0.00
• Learning Deterministic Policies With Policy Gradients In Constrained Markov Decision Processes (2025)0.00