Learning From Sparse Offline Datasets Via Conservative Density Estimation
2024 · Zhepeng Cen, Zuxin Liu, Zitong Wang, et al.
Abstract
Offline reinforcement learning (RL) offers a promising direction for learning policies from pre-collected datasets without requiring further interactions with the environment. However, existing methods struggle to handle out-of-distribution (OOD) extrapolation errors, especially in sparse reward or scarce data settings. In this paper, we propose a novel training algorithm called Conservative Density Estimation (CDE), which addresses this challenge by explicitly imposing constraints on the state-action occupancy stationary distribution. CDE overcomes the limitations of existing approaches, such as the stationary distribution correction method, by addressing the support mismatch issue in marginal importance sampling. Our method achieves state-of-the-art performance on the D4RL benchmark. Notably, CDE consistently outperforms baselines in challenging tasks with sparse rewards or insufficient data, demonstrating the advantages of our approach in addressing the extrapolation error problem i
Authors
(none)
Tags
Stats
Related papers
- CDSA: Conservative Denoising Score-based Algorithm For Offline Reinforcement Learning (2024)0.00
- Constrained Policy Optimization With Explicit Behavior Density For Offline Reinforcement Learning (2023)0.00
- Coptidice: Offline Constrained Reinforcement Learning Via Stationary Distribution Correction Estimation (2022)0.00
- Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage (2023)0.00
- Optidice: Offline Policy Optimization Via Stationary Distribution Correction Estimation (2021)0.00
- Offline Reinforcement Learning With Imbalanced Datasets (2023)0.00
- DOMAIN: Mildly Conservative Model-based Offline Reinforcement Learning (2023)0.00
- Regularizing A Model-based Policy Stationary Distribution To Stabilize Offline Reinforcement Learning (2022)0.00