Near-optimal Provable Uniform Convergence In Offline Policy Evaluation For Reinforcement Learning
2020 · Ming Yin, Yu Bai, Yu-Xiang Wang
Abstract
The problem of Offline Policy Evaluation (OPE) in Reinforcement Learning (RL) is a critical step towards applying RL in real-life applications. Existing work on OPE mostly focus on evaluating a fixed target policy \(\pi\), which does not provide useful bounds for offline policy learning as \(\pi\) will then be data-dependent. We address this problem by simultaneously evaluating all policies in a policy class \(\Pi\) -- uniform convergence in OPE -- and obtain nearly optimal error bounds for a number of global / local policy classes. Our results imply that the model-based planning achieves an optimal episode complexity of \(\widetilde\{O\}(H^3/d_m\epsilon^2)\) in identifying an \(\epsilon\)-optimal policy under the time-inhomogeneous episodic MDP model (\(H\) is the planning horizon, \(d_m\) is a quantity that reflects the exploration of the logging policy \(\mu\)). To the best of our knowledge, this is the first time the optimal rate is shown to be possible for the offline RL setting a
Authors
(none)
Tags
Stats
Related papers
- Optimal Uniform OPE And Model-based Offline Reinforcement Learning In Time-homogeneous, Reward-free And Task-agnostic Settings (2021)0.00
- Offline Policy Evaluation For Reinforcement Learning With Adaptively Collected Data (2023)0.00
- Near-optimal Offline Reinforcement Learning Via Double Variance Reduction (2021)0.00
- Pessimism In The Face Of Confounders: Provably Efficient Offline Reinforcement Learning In Partially Observable Markov Decision Processes (2022)0.00
- Projected State-action Balancing Weights For Offline Reinforcement Learning (2021)0.00
- Off-policy Evaluation In Doubly Inhomogeneous Environments (2023)7.16
- Expert-supervised Reinforcement Learning For Offline Policy Learning And Evaluation (2020)0.00
- Towards Optimal Off-policy Evaluation For Reinforcement Learning With Marginalized Importance Sampling (2019)0.00