A Theoretical Analysis Of Deep Q-learning

Abstract

Despite the great empirical success of deep reinforcement learning, its theoretical foundation is less well understood. In this work, we make the first attempt to theoretically understand the deep Q-network (DQN) algorithm (Mnih et al., 2015) from both algorithmic and statistical perspectives. In specific, we focus on a slight simplification of DQN that fully captures its key features. Under mild assumptions, we establish the algorithmic and statistical rates of convergence for the action-value functions of the iterative policy sequence obtained by DQN. In particular, the statistical error characterizes the bias and variance that arise from approximating the action-value function using deep neural network, while the algorithmic error converges to zero at a geometric rate. As a byproduct, our analysis provides justifications for the techniques of experience replay and target network, which are crucial to the empirical success of DQN. Furthermore, as a simple extension of DQN, we propose

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfan2019a

Related papers

• Deep Q-learning: Theoretical Insights From An Asymptotic Analysis (2020)10.35
• On The Convergence And Sample Complexity Analysis Of Deep Q-networks With \(\epsilon\)-greedy Exploration (2023)3.58
• Convergent And Efficient Deep Q Network Algorithm (2021)0.00
• Universal Approximation Theorem Of Deep Q-networks (2025)0.00
• Does DQN Learn? (2022)0.00
• A Finite-time Analysis Of Q-learning With Neural Network Function Approximation (2019)0.00
• Graying The Black Box: Understanding Dqns (2016)0.00
• Elastic Step DQN: A Novel Multi-step Algorithm To Alleviate Overestimation In Deep Qnetworks (2022)10.85