Data-efficient Policy Evaluation Through Behavior Policy Search

Abstract

We consider the task of evaluating a policy for a Markov decision process (MDP). The standard unbiased technique for evaluating a policy is to deploy the policy and observe its performance. We show that the data collected from deploying a different policy, commonly called the behavior policy, can be used to produce unbiased estimates with lower mean squared error than this standard technique. We derive an analytic expression for the optimal behavior policy --- the behavior policy that minimizes the mean squared error of the resulting estimates. Because this expression depends on terms that are unknown in practice, we propose a novel policy evaluation sub-problem, behavior policy search: searching for a behavior policy that reduces mean squared error. We present a behavior policy search algorithm and empirically demonstrate its effectiveness in lowering the mean squared error of policy performance estimates.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhanna2017data

Related papers

• Importance Sampling Policy Evaluation With An Estimated Behavior Policy (2018)0.00
• Infinite-horizon Off-policy Policy Evaluation With Multiple Behavior Policies (2019)0.00
• Behaviour Policy Estimation In Off-policy Policy Evaluation: Calibration Matters (2018)0.00
• Efficient Policy Evaluation With Safety Constraint For Reinforcement Learning (2024)0.00
• Optimal Mixture Weights For Off-policy Evaluation With Multiple Behavior Policies (2020)0.00
• Doubly Optimal Policy Evaluation For Reinforcement Learning (2024)0.00
• Behaviour Policy Optimization: Provably Lower Variance Return Estimates For Off-policy Reinforcement Learning (2025)0.00
• Reinforcement Learning With Unbiased Policy Evaluation And Linear Function Approximation (2022)0.00