Deep Reinforcement Learning And The Deadly Triad

Abstract

We know from reinforcement learning theory that temporal difference learning can fail in certain cases. Sutton and Barto (2018) identify a deadly triad of function approximation, bootstrapping, and off-policy learning. When these three properties are combined, learning can diverge with the value estimates becoming unbounded. However, several algorithms successfully combine these three properties, which indicates that there is at least a partial gap in our understanding. In this work, we investigate the impact of the deadly triad in practice, in the context of a family of popular deep reinforcement learning models - deep Q-networks trained with experience replay - analysing how the components of this system play a role in the emergence of the deadly triad, and in the agent's performance

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvanhasselt2018deep

Related papers

• Breaking The Deadly Triad With A Target Network (2021)0.00
• Revisiting A Design Choice In Gradient Temporal Difference Learning (2023)0.00
• Learning Expected Emphatic Traces For Deep RL (2021)5.24
• Dissecting Deep RL With High Update Ratios: Combatting Value Divergence (2024)0.00
• The Nature Of Temporal Difference Errors In Multi-step Distributional Reinforcement Learning (2022)0.00
• Simplifying Deep Temporal Difference Learning (2024)0.00
• Combating Reinforcement Learning's Sisyphean Curse With Intrinsic Fear (2016)0.00
• Analysis Of Off-policy Multi-step Td-learning With Linear Function Approximation (2024)2.26