Off-policy Evaluation In Infinite-horizon Reinforcement Learning With Latent Confounders

Abstract

Off-policy evaluation (OPE) in reinforcement learning is an important problem in settings where experimentation is limited, such as education and healthcare. But, in these very same settings, observed actions are often confounded by unobserved variables making OPE even more difficult. We study an OPE problem in an infinite-horizon, ergodic Markov decision process with unobserved confounders, where states and actions can act as proxies for the unobserved confounders. We show how, given only a latent variable model for states and actions, policy value can be identified from off-policy data. Our method involves two stages. In the first, we show how to use proxies to estimate stationary distribution ratios, extending recent work on breaking the curse of horizon to the confounded setting. In the second, we show optimal balancing can be combined with such learned ratios to obtain policy value while avoiding direct modeling of reward functions. We establish theoretical guarantees of consisten

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybennett2020off

Related papers

• An Instrumental Variable Approach To Confounded Off-policy Evaluation (2022)0.00
• Efficiently Breaking The Curse Of Horizon In Off-policy Evaluation With Double Reinforcement Learning (2019)10.21
• A Minimax Learning Approach To Off-policy Evaluation In Confounded Partially Observable Markov Decision Processes (2021)0.00
• Confounding-robust Policy Evaluation In Infinite-horizon Reinforcement Learning (2020)0.00
• Off-policy Evaluation In Doubly Inhomogeneous Environments (2023)7.16
• Conformal Off-policy Evaluation In Markov Decision Processes (2023)7.16
• Intrinsically Efficient, Stable, And Bounded Off-policy Evaluation For Reinforcement Learning (2019)0.00
• Non-asymptotic Confidence Intervals Of Off-policy Evaluation: Primal And Dual Bounds (2021)0.00