Bias Resilient Multi-step Off-policy Goal-conditioned Reinforcement Learning

Abstract

In goal-conditioned reinforcement learning (GCRL), sparse rewards present significant challenges, often obstructing efficient learning. Although multi-step GCRL can boost this efficiency, it can also lead to off-policy biases in target values. This paper dives deep into these biases, categorizing them into two distinct categories: "shooting" and "shifting". Recognizing that certain behavior policies can hasten policy refinement, we present solutions designed to capitalize on the positive aspects of these biases while minimizing their drawbacks, enabling the use of larger step sizes to speed up GCRL. An empirical study demonstrates that our approach ensures a resilient and robust improvement, even in ten-step learning scenarios, leading to superior learning efficiency and performance that generally surpass the baseline and several state-of-the-art multi-step GCRL benchmarks.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywu2023bias

Related papers

• Breaking The Bias Barrier In Concave Multi-objective Reinforcement Learning (2026)0.00
• Bayesian Reparameterization Of Reward-conditioned Reinforcement Learning With Energy-based Models (2023)0.00
• Multi-step Greedy Reinforcement Learning Algorithms (2019)0.00
• Dense And Diverse Goal Coverage In Multi Goal Reinforcement Learning (2025)0.00
• DDPG++: Striving For Simplicity In Continuous-control Off-policy Reinforcement Learning (2020)0.00
• Learning To Explore When Mistakes Are Not Allowed (2025)0.00
• SMORE: Score Models For Offline Goal-conditioned Reinforcement Learning (2023)0.00
• Provably Efficient Offline Goal-conditioned Reinforcement Learning With General Function Approximation And Single-policy Concentrability (2023)0.00