Distributions As Actions: A Unified Framework For Diverse Action Spaces

Abstract

We introduce a novel reinforcement learning (RL) framework that treats parameterized action distributions as actions, redefining the boundary between agent and environment. This reparameterization makes the new action space continuous, regardless of the original action type (discrete, continuous, hybrid, etc.). Under this new parameterization, we develop a generalized deterministic policy gradient estimator, Distributions-as-Actions Policy Gradient (DA-PG), which has lower variance than the gradient in the original action space. Although learning the critic over distribution parameters poses new challenges, we introduce Interpolated Critic Learning (ICL), a simple yet effective strategy to enhance learning, supported by insights from bandit settings. Building on TD3, a strong baseline for continuous control, we propose a practical actor-critic algorithm, Distributions-as-Actions Actor-Critic (DA-AC). Empirically, DA-AC achieves competitive performance in various settings across discret

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhe2025distributions

Related papers

• Beyond Distributions: Geometric Action Control For Continuous Reinforcement Learning (2025)0.00
• Marginal Policy Gradients: A Unified Family Of Estimators For Bounded Action Spaces With Applications (2018)0.00
• Rethinking Adversarial Attacks In Reinforcement Learning From Policy Distribution Perspective (2025)5.84
• Action Gaps And Advantages In Continuous-time Distributional Reinforcement Learning (2024)0.00
• Distributional Soft Actor-critic With Diffusion Policy (2025)0.00
• Continuous Control Reinforcement Learning: Distributed Distributional Drq Algorithms (2024)0.00
• DR-SAC: Distributionally Robust Soft Actor-critic For Reinforcement Learning Under Uncertainty (2025)0.00
• Distributional Policy Optimization: An Alternative Approach For Continuous Control (2019)0.00