Black-box Off-policy Estimation For Infinite-horizon Reinforcement Learning

Abstract

Off-policy estimation for long-horizon problems is important in many real-life applications such as healthcare and robotics, where high-fidelity simulators may not be available and on-policy evaluation is expensive or impossible. Recently, \cite\{liu18breaking\} proposed an approach that avoids the *curse of horizon* suffered by typical importance-sampling-based methods. While showing promising results, this approach is limited in practice as it requires data be drawn from the *stationary distribution* of a *known* behavior policy. In this work, we propose a novel approach that eliminates such limitations. In particular, we formulate the problem as solving for the fixed point of a certain operator. Using tools from Reproducing Kernel Hilbert Spaces (RKHSs), we develop a new estimator that computes importance ratios of stationary distributions, without knowledge of how the off-policy data are collected. We analyze its asymptotic consistency and finite-sample generalization. Experiments

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymousavi2020black

Related papers

• Off-policy Evaluation In Infinite-horizon Reinforcement Learning With Latent Confounders (2020)0.00
• Efficiently Breaking The Curse Of Horizon In Off-policy Evaluation With Double Reinforcement Learning (2019)10.21
• Asymptotically Efficient Off-policy Evaluation For Tabular Reinforcement Learning (2020)0.00
• Doubly Robust Off-policy Value And Gradient Estimation For Deterministic Policies (2020)0.00
• Infinite-horizon Off-policy Policy Evaluation With Multiple Behavior Policies (2019)0.00
• Confounding-robust Policy Evaluation In Infinite-horizon Reinforcement Learning (2020)0.00
• Statistically Efficient Advantage Learning For Offline Reinforcement Learning In Infinite Horizons (2022)0.00
• Towards Optimal Off-policy Evaluation For Reinforcement Learning With Marginalized Importance Sampling (2019)0.00