Action Gaps And Advantages In Continuous-time Distributional Reinforcement Learning

Abstract

When decisions are made at high frequency, traditional reinforcement learning (RL) methods struggle to accurately estimate action values. In turn, their performance is inconsistent and often poor. Whether the performance of distributional RL (DRL) agents suffers similarly, however, is unknown. In this work, we establish that DRL agents are sensitive to the decision frequency. We prove that action-conditioned return distributions collapse to their underlying policy's return distribution as the decision frequency increases. We quantify the rate of collapse of these return distributions and exhibit that their statistics collapse at different rates. Moreover, we define distributional perspectives on action gaps and advantages. In particular, we introduce the superiority as a probabilistic generalization of the advantage -- the core object of approaches to mitigating performance issues in high-frequency value-based RL. In addition, we build a superiority-based DRL algorithm. Through simulat

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywiltzer2024action

Related papers

• Distributions As Actions: A Unified Framework For Diverse Action Spaces (2025)0.00
• Normality-guided Distributional Reinforcement Learning For Continuous Control (2022)0.00
• Continuous Control Reinforcement Learning: Distributed Distributional Drq Algorithms (2024)0.00
• A Comparative Analysis Of Expected And Distributional Reinforcement Learning (2019)9.76
• Deterministic Policy Gradient For Reinforcement Learning With Continuous Time And State (2025)0.00
• A Risk-sensitive Approach To Policy Optimization (2022)3.58
• Tractable Representations For Convergent Approximation Of Distributional HJB Equations (2025)0.00
• Invariance To Quantile Selection In Distributional Continuous Control (2022)0.00