Deep Q-learning: Theoretical Insights From An Asymptotic Analysis

Abstract

Deep Q-Learning is an important reinforcement learning algorithm, which involves training a deep neural network, called Deep Q-Network (DQN), to approximate the well-known Q-function. Although wildly successful under laboratory conditions, serious gaps between theory and practice as well as a lack of formal guarantees prevent its use in the real world. Adopting a dynamical systems perspective, we provide a theoretical analysis of a popular version of Deep Q-Learning under realistic and verifiable assumptions. More specifically, we prove an important result on the convergence of the algorithm, characterizing the asymptotic behavior of the learning process. Our result sheds light on hitherto unexplained properties of the algorithm and helps understand empirical observations, such as performance inconsistencies even after training. Unlike previous theories, our analysis accommodates state Markov processes with multiple stationary distributions. In spite of the focus on Deep Q-Learning, we

Stats

• citations23
• S2 citations—
• github stars0
• HF likes0
• heat score10.35
• arxiv keyramaswamy2020deep

Related papers

• A Theoretical Analysis Of Deep Q-learning (2019)0.00
• Universal Approximation Theorem Of Deep Q-networks (2025)0.00
• Convergent And Efficient Deep Q Network Algorithm (2021)0.00
• On The Convergence And Sample Complexity Analysis Of Deep Q-networks With \(\epsilon\)-greedy Exploration (2023)3.58
• A Finite-time Analysis Of Q-learning With Neural Network Function Approximation (2019)0.00
• Finite-time Analysis Of Asynchronous Q-learning Under Diminishing Step-size From Control-theoretic View (2022)3.58
• Deep Q-learning: A Robust Control Approach (2022)9.23
• Finite-time Analysis For Double Q-learning (2020)0.00