Convergent And Efficient Deep Q Network Algorithm

Abstract

Despite the empirical success of the deep Q network (DQN) reinforcement learning algorithm and its variants, DQN is still not well understood and it does not guarantee convergence. In this work, we show that DQN can indeed diverge and cease to operate in realistic settings. Although there exist gradient-based convergent methods, we show that they actually have inherent problems in learning dynamics which cause them to fail even in simple tasks. To overcome these problems, we propose a convergent DQN algorithm (C-DQN) that is guaranteed to converge and can work with large discount factors (0.9998). It learns robustly in difficult settings and can learn several difficult games in the Atari 2600 benchmark that DQN fails to solve. Our codes have been publicly released and can be used to reproduce our results.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2021convergent

Related papers

• A Theoretical Analysis Of Deep Q-learning (2019)0.00
• On The Convergence And Sample Complexity Analysis Of Deep Q-networks With \(\epsilon\)-greedy Exploration (2023)3.58
• Deep Q-learning: Theoretical Insights From An Asymptotic Analysis (2020)10.35
• Elastic Step DQN: A Novel Multi-step Algorithm To Alleviate Overestimation In Deep Qnetworks (2022)10.85
• Does DQN Learn? (2022)0.00
• Generalization And Regularization In DQN (2018)0.00
• DQN With Model-based Exploration: Efficient Learning On Environments With Sparse Rewards (2019)0.00
• Transforming Game Play: A Comparative Study Of DCQN And DTQN Architectures In Reinforcement Learning (2024)0.00