Estimation Error Correction In Deep Reinforcement Learning For Deterministic Actor-critic Methods

Abstract

In value-based deep reinforcement learning methods, approximation of value functions induces overestimation bias and leads to suboptimal policies. We show that in deep actor-critic methods that aim to overcome the overestimation bias, if the reinforcement signals received by the agent have a high variance, a significant underestimation bias arises. To minimize the underestimation, we introduce a parameter-free, novel deep Q-learning variant. Our Q-value update rule combines the notions behind Clipped Double Q-learning and Maxmin Q-learning by computing the critic objective through the nested combination of maximum and minimum operators to bound the approximate value estimates. We evaluate our modification on the suite of several OpenAI Gym continuous control tasks, improving the state-of-the-art in every environment tested.

Stats

• citations8
• S2 citations—
• github stars0
• HF likes0
• heat score7.16
• arxiv keysaglam2021estimation

Related papers

• Parameter-free Reduction Of The Estimation Bias In Deep Reinforcement Learning For Deterministic Policy Gradients (2021)0.00
• Mitigating Off-policy Bias In Actor-critic Methods With One-step Q-learning: A Novel Correction Approach (2022)0.00
• Distributional Soft Actor-critic: Off-policy Reinforcement Learning For Addressing Value Estimation Errors (2020)17.77
• Adaptively Calibrated Critic Estimates For Deep Reinforcement Learning (2021)7.16
• WD3: Taming The Estimation Bias In Deep Reinforcement Learning (2020)10.21
• Exploiting Estimation Bias In Clipped Double Q-learning For Continous Control Reinforcement Learning Tasks (2024)0.00
• An Information-theoretic Optimality Principle For Deep Reinforcement Learning (2017)0.00
• Value Activation For Bias Alleviation: Generalized-activated Deep Double Deterministic Policy Gradients (2021)0.00