Doubly Robust Distributionally Robust Off-policy Evaluation And Learning
2022 · Nathan Kallus, Xiaojie Mao, Kaiwen Wang, et al.
Abstract
Off-policy evaluation and learning (OPE/L) use offline observational data to make better decisions, which is crucial in applications where online experimentation is limited. However, depending entirely on logged data, OPE/L is sensitive to environment distribution shifts -- discrepancies between the data-generating environment and that where policies are deployed. \citet\{si2020distributional\} proposed distributionally robust OPE/L (DROPE/L) to address this, but the proposal relies on inverse-propensity weighting, whose estimation error and regret will deteriorate if propensities are nonparametrically estimated and whose variance is suboptimal even if not. For standard, non-robust, OPE/L, this is solved by doubly robust (DR) methods, but they do not naturally extend to the more complex DROPE/L, which involves a worst-case expectation. In this paper, we propose the first DR algorithms for DROPE/L with KL-divergence uncertainty sets. For evaluation, we propose Localized Doubly Robust DR
Authors
(none)
Tags
Stats
Related papers
- Intrinsically Efficient, Stable, And Bounded Off-policy Evaluation For Reinforcement Learning (2019)0.00
- Double Reinforcement Learning For Efficient Off-policy Evaluation In Markov Decision Processes (2019)0.00
- Off-policy Evaluation In Markov Decision Processes Under Weak Distributional Overlap (2024)0.00
- Off-policy Evaluation In Doubly Inhomogeneous Environments (2023)7.16
- Double Pessimism Is Provably Efficient For Distributionally Robust Offline Reinforcement Learning: Generic Algorithm And Robust Partial Coverage (2023)0.00
- More Efficient Off-policy Evaluation Through Regularized Targeted Learning (2019)0.00
- Distributional Robustness And Regularization In Reinforcement Learning (2020)0.00
- Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage (2023)0.00