Equivalence Between Policy Gradients And Soft Q-learning

Abstract

Two of the leading approaches for model-free reinforcement learning are policy gradient methods and \(Q\)-learning methods. \(Q\)-learning methods can be effective and sample-efficient when they work, however, it is not well-understood why they work, since empirically, the \(Q\)-values they estimate are very inaccurate. A partial explanation may be that \(Q\)-learning methods are secretly implementing policy gradient updates: we show that there is a precise equivalence between \(Q\)-learning and policy gradient methods in the setting of entropy-regularized reinforcement learning, that "soft" (entropy-regularized) \(Q\)-learning is exactly equivalent to a policy gradient method. We also point out a connection between \(Q\)-learning methods and natural policy gradient methods. Experimentally, we explore the entropy-regularized versions of \(Q\)-learning and policy gradients, and we find them to perform as well as (or slightly better than) the standard variants on the Atari benchmark. We

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyschulman2017equivalence

Related papers

• Beyond Exact Gradients: Convergence Of Stochastic Soft-max Policy Gradient Methods With Entropy Regularization (2021)2.26
• Residual Policy Gradient: A Reward View Of Kl-regularized Objective (2025)0.00
• Entropy-augmented Entropy-regularized Reinforcement Learning And A Continuous Path From Policy Gradient To Q-learning (2020)0.00
• Policy Gradient Using Weak Derivatives For Reinforcement Learning (2020)0.00
• Policy Optimization Reinforcement Learning With Entropy Regularization (2019)0.00
• Fast Global Convergence Of Natural Policy Gradient Methods With Entropy Regularization (2020)0.00
• Soft Policy Gradient Method For Maximum Entropy Deep Reinforcement Learning (2019)10.85
• Approximate Newton Policy Gradient Algorithms (2021)0.00