Regularizing A Model-based Policy Stationary Distribution To Stabilize Offline Reinforcement Learning

Abstract

Offline reinforcement learning (RL) extends the paradigm of classical RL algorithms to purely learning from static datasets, without interacting with the underlying environment during the learning process. A key challenge of offline RL is the instability of policy training, caused by the mismatch between the distribution of the offline data and the undiscounted stationary state-action distribution of the learned policy. To avoid the detrimental impact of distribution mismatch, we regularize the undiscounted stationary distribution of the current policy towards the offline data during the policy optimization process. Further, we train a dynamics model to both implement this regularization and better estimate the stationary distribution of the current policy, reducing the error induced by distribution mismatch. On a wide range of continuous-control offline RL datasets, our method indicates competitive performance, which validates our algorithm. The code is publicly available.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyang2022regularizing

Related papers

• Iteratively Refined Behavior Regularization For Offline Reinforcement Learning (2023)2.26
• Optidice: Offline Policy Optimization Via Stationary Distribution Correction Estimation (2021)0.00
• Offline Policy Optimization In RL With Variance Regularizaton (2022)0.00
• Distributionally Robust Offline Reinforcement Learning With Linear Function Approximation (2022)0.00
• Comadice: Offline Cooperative Multi-agent Reinforcement Learning With Stationary Distribution Shift Regularization (2024)0.00
• A Behavior Regularized Implicit Policy For Offline Reinforcement Learning (2022)0.00
• Policy Regularization With Dataset Constraint For Offline Reinforcement Learning (2023)0.00
• Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage (2023)0.00