Addressing Maximization Bias In Reinforcement Learning With Two-sample Testing

Abstract

Value-based reinforcement-learning algorithms have shown strong results in games, robotics, and other real-world applications. Overestimation bias is a known threat to those algorithms and can sometimes lead to dramatic performance decreases or even complete algorithmic failure. We frame the bias problem statistically and consider it an instance of estimating the maximum expected value (MEV) of a set of random variables. We propose the \(T\)-Estimator (TE) based on two-sample testing for the mean, that flexibly interpolates between over- and underestimation by adjusting the significance level of the underlying hypothesis tests. We also introduce a generalization, termed \(K\)-Estimator (KE), that obeys the same bias and variance bounds as the TE and relies on a nearly arbitrary kernel function. We introduce modifications of \(Q\)-Learning and the Bootstrapped Deep \(Q\)-Network (BDQN) using the TE and the KE, and prove convergence in the tabular setting. Furthermore, we propose an adap

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywaltz2022addressing

Related papers

• Automating Control Of Overestimation Bias For Reinforcement Learning (2021)0.00
• On The Estimation Bias In Double Q-learning (2021)0.00
• WD3: Taming The Estimation Bias In Deep Reinforcement Learning (2020)10.21
• Parameter-free Reduction Of The Estimation Bias In Deep Reinforcement Learning For Deterministic Policy Gradients (2021)0.00
• Estimation Error Correction In Deep Reinforcement Learning For Deterministic Actor-critic Methods (2021)7.16
• Exploiting Estimation Bias In Clipped Double Q-learning For Continous Control Reinforcement Learning Tasks (2024)0.00
• Reducing Variance In Temporal-difference Value Estimation Via Ensemble Of Deep Networks (2022)0.00
• An Information-theoretic Optimality Principle For Deep Reinforcement Learning (2017)0.00