The Mean-squared Error Of Double Q-learning

Abstract

In this paper, we establish a theoretical comparison between the asymptotic mean-squared error of Double Q-learning and Q-learning. Our result builds upon an analysis for linear stochastic approximation based on Lyapunov equations and applies to both tabular setting and with linear function approximation, provided that the optimal policy is unique and the algorithms converge. We show that the asymptotic mean-squared error of Double Q-learning is exactly equal to that of Q-learning if Double Q-learning uses twice the learning rate of Q-learning and outputs the average of its two estimators. We also present some practical implications of this theoretical observation using simulations.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyweng2020the

Related papers

• Finite-time Analysis For Double Q-learning (2020)0.00
• Finite-time Analysis Of Simultaneous Double Q-learning (2024)0.00
• On The Estimation Bias In Double Q-learning (2021)0.00
• Deep Q-learning: Theoretical Insights From An Asymptotic Analysis (2020)10.35
• Q-learning As A Monotone Scheme (2024)0.00
• Smooth Q-learning: Accelerate Convergence Of Q-learning Using Similarity (2021)0.00
• Two-timescale Q-learning With Function Approximation In Zero-sum Stochastic Games (2023)0.00
• Regularized Q-learning (2022)0.00