Regret Minimization Experience Replay In Off-policy Reinforcement Learning

Abstract

In reinforcement learning, experience replay stores past samples for further reuse. Prioritized sampling is a promising technique to better utilize these samples. Previous criteria of prioritization include TD error, recentness and corrective feedback, which are mostly heuristically designed. In this work, we start from the regret minimization objective, and obtain an optimal prioritization strategy for Bellman update that can directly maximize the return of the policy. The theory suggests that data with higher hindsight TD error, better on-policiness and more accurate Q value should be assigned with higher weights during sampling. Thus most previous criteria only consider this strategy partially. We not only provide theoretical justifications for previous criteria, but also propose two new methods to compute the prioritization weight, namely ReMERN and ReMERT. ReMERN learns an error network, while ReMERT exploits the temporal ordering of states. Both methods outperform previous priori

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2021regret

Related papers

• Replay For Safety (2021)0.00
• Off-policy Correction For Deep Deterministic Policy Gradient Algorithms Via Batch Prioritized Experience Replay (2021)0.00
• Prioritizing Samples In Reinforcement Learning With Reducible Loss (2022)5.24
• Reward Prediction Error Prioritisation In Experience Replay: The RPE-PER Method (2025)0.00
• Large Batch Experience Replay (2021)0.00
• Improving Experience Replay With Successor Representation (2021)0.00
• CUER: Corrected Uniform Experience Replay For Off-policy Continuous Deep Reinforcement Learning Algorithms (2024)0.00
• MAC-PO: Multi-agent Experience Replay Via Collective Priority Optimization (2023)0.00