Automating Control Of Overestimation Bias For Reinforcement Learning

Abstract

Overestimation bias control techniques are used by the majority of high-performing off-policy reinforcement learning algorithms. However, most of these techniques rely on pre-defined bias correction policies that are either not flexible enough or require environment-specific tuning of hyperparameters. In this work, we present a general data-driven approach for the automatic selection of bias control hyperparameters. We demonstrate its effectiveness on three algorithms: Truncated Quantile Critics, Weighted Delayed DDPG, and Maxmin Q-learning. The proposed technique eliminates the need for an extensive hyperparameter search. We show that it leads to a significant reduction of the actual number of interactions while preserving the performance.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykuznetsov2021automating

Related papers

• Exploiting Estimation Bias In Clipped Double Q-learning For Continous Control Reinforcement Learning Tasks (2024)0.00
• Parameter-free Reduction Of The Estimation Bias In Deep Reinforcement Learning For Deterministic Policy Gradients (2021)0.00
• WD3: Taming The Estimation Bias In Deep Reinforcement Learning (2020)10.21
• Adaptively Calibrated Critic Estimates For Deep Reinforcement Learning (2021)7.16
• Addressing Maximization Bias In Reinforcement Learning With Two-sample Testing (2022)0.00
• Moderate Actor-critic Methods: Controlling Overestimation Bias Via Expectile Loss (2025)0.00
• Mitigating Estimation Bias With Representation Learning In TD Error-driven Regularization (2025)0.00
• Estimation Error Correction In Deep Reinforcement Learning For Deterministic Actor-critic Methods (2021)7.16