Prioritized Trajectory Replay: A Replay Memory For Data-driven Reinforcement Learning

Abstract

In recent years, data-driven reinforcement learning (RL), also known as offline RL, have gained significant attention. However, the role of data sampling techniques in offline RL has been overlooked despite its potential to enhance online RL performance. Recent research suggests applying sampling techniques directly to state-transitions does not consistently improve performance in offline RL. Therefore, in this study, we propose a memory technique, (Prioritized) Trajectory Replay (TR/PTR), which extends the sampling perspective to trajectories for more comprehensive information extraction from limited data. TR enhances learning efficiency by backward sampling of trajectories that optimizes the use of subsequent state information. Building on TR, we build the weighted critic target to avoid sampling unseen actions in offline training, and Prioritized Trajectory Replay (PTR) that enables more efficient trajectory sampling, prioritized by various trajectory priority metrics. We demonstrat

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2023prioritized

Related papers

• Harnessing Mixed Offline Reinforcement Learning Datasets Via Trajectory Weighting (2023)0.00
• PTR-PPO: Proximal Policy Optimization With Prioritized Trajectory Replay (2021)0.00
• Decoupled Prioritized Resampling For Offline RL (2023)5.84
• Snapshot Reinforcement Learning: Leveraging Prior Trajectories For Efficiency (2024)0.00
• Prioritized Generative Replay (2024)0.00
• Using Offline Data To Speed Up Reinforcement Learning In Procedurally Generated Environments (2023)6.77
• Provably Efficient Offline Reinforcement Learning With Trajectory-wise Reward (2022)0.00
• Trajdeleter: Enabling Trajectory Forgetting In Offline Reinforcement Learning Agents (2024)0.00