Rewarded Region Replay (R3) For Policy Learning With Discrete Action Space

Abstract

We introduce a new on-policy algorithm called Rewarded Region Replay (R3), which significantly improves on PPO in solving environments with discrete action spaces. R3 improves sample efficiency by using a replay buffer which contains past successful trajectories with reward above a certain threshold, which are used to update a PPO agent with importance sampling. Crucially, we discard the importance sampling factors which are above a certain ratio to reduce variance and stabilize training. We found that R3 significantly outperforms PPO in Minigrid environments with sparse rewards and discrete action space, such as DoorKeyEnv and CrossingEnv, and moreover we found that the improvement margin of our method versus baseline PPO increases with the complexity of the environment. We also benchmarked the performance of R3 against DDQN (Double Deep Q-Network), which is a standard baseline in off-policy methods for discrete actions, and found that R3 also outperforms DDQN agent in DoorKeyEnv. Las

Stats

• citations0
• S2 citations—
• github stars2
• HF likes0
• heat score0.95
• arxiv keyli2024rewarded

Related papers

• PTR-PPO: Proximal Policy Optimization With Prioritized Trajectory Replay (2021)0.00
• Relative Entropy Pathwise Policy Optimization (2025)0.00
• Continuous-action Reinforcement Learning For Playing Racing Games: Comparing SPG To PPO (2020)0.00
• Efficient Deep Reinforcement Learning With Predictive Processing Proximal Policy Optimization (2022)0.00
• Proximal Policy Optimization Algorithms (2017)0.00
• Colored Noise In PPO: Improved Exploration And Performance Through Correlated Action Sampling (2023)4.52
• Reward Scale Robustness For Proximal Policy Optimization Via Dreamerv3 Tricks (2023)0.00
• The Surprising Effectiveness Of PPO In Cooperative, Multi-agent Games (2021)0.00