Experience Replay Using Transition Sequences

Abstract

Experience replay is one of the most commonly used approaches to improve the sample efficiency of reinforcement learning algorithms. In this work, we propose an approach to select and replay sequences of transitions in order to accelerate the learning of a reinforcement learning agent in an off-policy setting. In addition to selecting appropriate sequences, we also artificially construct transition sequences using information gathered from previous agent-environment interactions. These sequences, when replayed, allow value function information to trickle down to larger sections of the state/state-action space, thereby making the most of the agent's experience. We demonstrate our approach on modified versions of standard reinforcement learning tasks such as the mountain car and puddle world problems and empirically show that it enables better learning of value functions as compared to other forms of experience replay. Further, we briefly discuss some of the possible extensions to this w

Stats

• citations14
• S2 citations—
• github stars0
• HF likes0
• heat score8.82
• arxiv keykarimpanal2017experience

Related papers

• Replay For Safety (2021)0.00
• CUER: Corrected Uniform Experience Replay For Off-policy Continuous Deep Reinforcement Learning Algorithms (2024)0.00
• Improving Experience Replay Through Modeling Of Similar Transitions' Sets (2021)0.00
• Improving Experience Replay With Successor Representation (2021)0.00
• Bootstrapping A DQN Replay Memory With Synthetic Experiences (2020)5.84
• Prioritized Sequence Experience Replay (2019)0.00
• Adaptive Experience Selection For Policy Gradient (2020)0.00
• Learning To Sample With Local And Global Contexts In Experience Replay Buffer (2020)0.00