Convergence Results For Q-learning With Experience Replay

Abstract

A commonly used heuristic in RL is experience replay (e.g.~\citet\{lin1993reinforcement, mnih2015human\}), in which a learner stores and re-uses past trajectories as if they were sampled online. In this work, we initiate a rigorous study of this heuristic in the setting of tabular Q-learning. We provide a convergence rate guarantee, and discuss how it compares to the convergence of Q-learning depending on important parameters such as the frequency and number of replay iterations. We also provide theoretical evidence showing when we might expect this heuristic to strictly improve performance, by introducing and analyzing a simple class of MDPs. Finally, we provide some experiments to support our theoretical findings.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyszlak2021convergence

Related papers

• Replay For Safety (2021)0.00
• Online Target Q-learning With Reverse Experience Replay: Efficiently Finding The Optimal Policy For Linear Mdps (2021)0.00
• On The Convergence Of Experience Replay In Policy Optimization: Characterizing Bias, Variance, And Finite-time Convergence (2021)0.00
• A Deeper Look At Experience Replay (2017)0.00
• Introspective Experience Replay: Look Back When Surprised (2022)0.00
• Large Batch Experience Replay (2021)0.00
• Experience Replay Using Transition Sequences (2017)8.82
• Stabilising Experience Replay For Deep Multi-agent Reinforcement Learning (2017)0.00