Seizing Serendipity: Exploiting The Value Of Past Success In Off-policy Actor-critic

Abstract

Learning high-quality \(Q\)-value functions plays a key role in the success of many modern off-policy deep reinforcement learning (RL) algorithms. Previous works primarily focus on addressing the value overestimation issue, an outcome of adopting function approximators and off-policy learning. Deviating from the common viewpoint, we observe that \(Q\)-values are often underestimated in the latter stage of the RL training process, potentially hindering policy learning and reducing sample efficiency. We find that such a long-neglected phenomenon is often related to the use of inferior actions from the current policy in Bellman updates as compared to the more optimal action samples in the replay buffer. To address this issue, our insight is to incorporate sufficient exploitation of past successes while maintaining exploration optimism. We propose the Blended Exploitation and Exploration (BEE) operator, a simple yet effective approach that updates \(Q\)-value using both historical best-per

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyji2023seizing

Related papers

• Imitating Past Successes Can Be Very Suboptimal (2022)0.00
• Optimistic Curiosity Exploration And Conservative Exploitation With Linear Reward Shaping (2022)0.00
• Learning Self-imitating Diverse Policies (2018)0.00
• Towards Adapting Reinforcement Learning Agents To New Tasks: Insights From Q-values (2024)0.00
• Mitigating Off-policy Bias In Actor-critic Methods With One-step Q-learning: A Novel Correction Approach (2022)0.00
• An Information-theoretic Optimality Principle For Deep Reinforcement Learning (2017)0.00
• Offline-boosted Actor-critic: Adaptively Blending Optimal Historical Behaviors In Deep Off-policy RL (2024)0.00
• Q-pensieve: Boosting Sample Efficiency Of Multi-objective RL Through Memory Sharing Of Q-snapshots (2022)0.00