Prioritizing Samples In Reinforcement Learning With Reducible Loss

Abstract

Most reinforcement learning algorithms take advantage of an experience replay buffer to repeatedly train on samples the agent has observed in the past. Not all samples carry the same amount of significance and simply assigning equal importance to each of the samples is a na\"ive strategy. In this paper, we propose a method to prioritize samples based on how much we can learn from a sample. We define the learn-ability of a sample as the steady decrease of the training loss associated with this sample over time. We develop an algorithm to prioritize samples with high learn-ability, while assigning lower priority to those that are hard-to-learn, typically caused by noise or stochasticity. We empirically show that our method is more robust than random sampling and also better than just prioritizing with respect to the training loss, i.e. the temporal difference loss, which is used in prioritized experience replay.

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keysujit2022prioritizing

Related papers

• Regret Minimization Experience Replay In Off-policy Reinforcement Learning (2021)0.00
• Large Batch Experience Replay (2021)0.00
• Prioritized Generative Replay (2024)0.00
• Improving Experience Replay With Successor Representation (2021)0.00
• Prioritized Level Replay (2020)0.00
• Learning To Sample With Local And Global Contexts In Experience Replay Buffer (2020)0.00
• Decoupled Prioritized Resampling For Offline RL (2023)5.84
• An Equivalence Between Loss Functions And Non-uniform Sampling In Experience Replay (2020)0.00