Overcoming Model Bias For Robust Offline Deep Reinforcement Learning

Abstract

State-of-the-art reinforcement learning algorithms mostly rely on being allowed to directly interact with their environment to collect millions of observations. This makes it hard to transfer their success to industrial control problems, where simulations are often very costly or do not exist, and exploring in the real environment can potentially lead to catastrophic events. Recently developed, model-free, offline RL algorithms, can learn from a single dataset (containing limited exploration) by mitigating extrapolation error in value functions. However, the robustness of the training process is still comparatively low, a problem known from methods using value functions. To improve robustness and stability of the learning process, we use dynamics models to assess policy performance instead of value functions, resulting in MOOSE (MOdel-based Offline policy Search with Ensembles), an algorithm which ensures low model bias by keeping the policy within the support of the data. We compare M

Stats

• citations34
• S2 citations—
• github stars0
• HF likes0
• heat score11.58
• arxiv keyswazinna2020overcoming

Related papers

• Morel : Model-based Offline Reinforcement Learning (2020)0.00
• Revisiting Design Choices In Offline Model-based Reinforcement Learning (2021)6.34
• Deployment-efficient Reinforcement Learning Via Model-based Offline Optimization (2020)0.00
• An Optimistic Perspective On Offline Reinforcement Learning (2019)0.00
• Model-based Offline Reinforcement Learning With Pessimism-modulated Dynamics Belief (2022)0.00
• Model-based Offline Reinforcement Learning With Adversarial Data Augmentation (2025)0.00
• Towards Data-driven Offline Simulations For Online Reinforcement Learning (2022)0.00
• Regularizing A Model-based Policy Stationary Distribution To Stabilize Offline Reinforcement Learning (2022)0.00