Action Noise In Off-policy Deep Reinforcement Learning: Impact On Exploration And Performance

Abstract

Many Deep Reinforcement Learning (D-RL) algorithms rely on simple forms of exploration such as the additive action noise often used in continuous control domains. Typically, the scaling factor of this action noise is chosen as a hyper-parameter and is kept constant during training. In this paper, we focus on action noise in off-policy deep reinforcement learning for continuous control. We analyze how the learned policy is impacted by the noise type, noise scale, and impact scaling factor reduction schedule. We consider the two most prominent types of action noise, Gaussian and Ornstein-Uhlenbeck noise, and perform a vast experimental campaign by systematically varying the noise type and scale parameter, and by measuring variables of interest like the expected return of the policy and the state-space coverage during exploration. For the latter, we propose a novel state-space coverage measure \(\operatorname\{X\}_\{\mathcal\{U\}\text\{rel\}\}\) that is more robust to estimation artifacts

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhollenstein2022action

Related papers

• Parameter Space Noise For Exploration (2017)0.00
• Exploring More When It Needs In Deep Reinforcement Learning (2021)0.00
• Noisy Networks For Exploration (2017)0.00
• Adaptive Symmetric Reward Noising For Reinforcement Learning (2019)0.00
• Guided Exploration In Reinforcement Learning Via Monte Carlo Critic Optimization (2022)0.00
• Colored Noise In PPO: Improved Exploration And Performance Through Correlated Action Sampling (2023)4.52
• Enhancing Robustness In Deep Reinforcement Learning: A Lyapunov Exponent Approach (2024)0.00
• Policy Optimization In A Noisy Neighborhood: On Return Landscapes In Continuous Control (2023)0.00