Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage

Abstract

The goal of an offline reinforcement learning (RL) algorithm is to learn optimal polices using historical (offline) data, without access to the environment for online exploration. One of the main challenges in offline RL is the distribution shift which refers to the difference between the state-action visitation distribution of the data generating policy and the learning policy. Many recent works have used the idea of pessimism for developing offline RL algorithms and characterizing their sample complexity under a relatively weak assumption of single policy concentrability. Different from the offline RL literature, the area of distributionally robust learning (DRL) offers a principled framework that uses a minimax formulation to tackle model mismatch between training and testing environments. In this work, we aim to bridge these two areas by showing that the DRL approach can be used to tackle the distributional shift problem in offline RL. In particular, we propose two offline RL algor

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypanaganti2023bridging

Related papers

• Distributionally Robust Offline Reinforcement Learning With Linear Function Approximation (2022)0.00
• Mitigating Distribution Shift In Model-based Offline RL Via Shifts-aware Reward Learning (2024)0.00
• Double Pessimism Is Provably Efficient For Distributionally Robust Offline Reinforcement Learning: Generic Algorithm And Robust Partial Coverage (2023)0.00
• Bridging Offline Reinforcement Learning And Imitation Learning: A Tale Of Pessimism (2021)0.00
• Comadice: Offline Cooperative Multi-agent Reinforcement Learning With Stationary Distribution Shift Regularization (2024)0.00
• Distributionally Robust Model-based Offline Reinforcement Learning With Near-optimal Sample Complexity (2022)0.00
• Optidice: Offline Policy Optimization Via Stationary Distribution Correction Estimation (2021)0.00
• Off-policy Policy Gradient Algorithms By Constraining The State Distribution Shift (2019)0.00