When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning

Abstract

In offline reinforcement learning (RL), one detrimental issue to policy learning is the error accumulation of deep Q function in out-of-distribution (OOD) areas. Unfortunately, existing offline RL methods are often over-conservative, inevitably hurting generalization performance outside data distribution. In our study, one interesting observation is that deep Q functions approximate well inside the convex hull of training data. Inspired by this, we propose a new method, DOGE (Distance-sensitive Offline RL with better GEneralization). DOGE marries dataset geometry with deep function approximators in offline RL, and enables exploitation in generalizable OOD areas rather than strictly constraining policy within data distribution. Specifically, DOGE trains a state-conditioned distance function that can be readily plugged into standard actor-critic methods as a policy constraint. Simple yet elegant, our algorithm enjoys better generalization compared to state-of-the-art methods on D4RL benc

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyli2022when

Related papers

• Equivariant Data Augmentation For Generalization In Offline Reinforcement Learning (2023)0.00
• Distributionally Robust Offline Reinforcement Learning With Linear Function Approximation (2022)0.00
• Improving Zero-shot Generalization In Offline Reinforcement Learning Using Generalized Similarity Functions (2021)2.26
• Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage (2023)0.00
• Domain Generalization For Robust Model-based Offline Reinforcement Learning (2022)0.00
• An Optimistic Perspective On Offline Reinforcement Learning (2019)0.00
• Learning From Sparse Offline Datasets Via Conservative Density Estimation (2024)0.00
• Uncertainty-based Offline Reinforcement Learning With Diversified Q-ensemble (2021)0.00