Infinite-horizon Off-policy Policy Evaluation With Multiple Behavior Policies

Abstract

We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies. Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation. We propose estimated mixture policy (EMP), a novel class of partially policy-agnostic methods to accurately estimate those quantities. With careful analysis, we show that EMP gives rise to estimates with reduced variance for estimating the state stationary distribution correction while it also offers a useful induction bias for estimating the state-action stationary distribution correction. In extensive experiments with both continuous and discrete environments, we demonstrate that our algorithm offers significantly improved accuracy compared to the state-of-the-art methods.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2019infinite

Related papers

• Data-efficient Policy Evaluation Through Behavior Policy Search (2017)0.00
• Optimal Mixture Weights For Off-policy Evaluation With Multiple Behavior Policies (2020)0.00
• Importance Sampling Policy Evaluation With An Estimated Behavior Policy (2018)0.00
• Black-box Off-policy Estimation For Infinite-horizon Reinforcement Learning (2020)0.00
• Off-policy Evaluation In Infinite-horizon Reinforcement Learning With Latent Confounders (2020)0.00
• Off-policy Policy Gradient With State Distribution Correction (2019)0.00
• Behaviour Policy Estimation In Off-policy Policy Evaluation: Calibration Matters (2018)0.00
• A Maximum-entropy Approach To Off-policy Evaluation In Average-reward Mdps (2020)0.00