Efficient Policy Evaluation With Safety Constraint For Reinforcement Learning

Abstract

In reinforcement learning, classic on-policy evaluation methods often suffer from high variance and require massive online data to attain the desired accuracy. Previous studies attempt to reduce evaluation variance by searching for or designing proper behavior policies to collect data. However, these approaches ignore the safety of such behavior policies -- the designed behavior policies have no safety guarantee and may lead to severe damage during online executions. In this paper, to address the challenge of reducing variance while ensuring safety simultaneously, we propose an optimal variance-minimizing behavior policy under safety constraints. Theoretically, while ensuring safety constraints, our evaluation method is unbiased and has lower variance than on-policy evaluation. Empirically, our method is the only existing method to achieve both substantial variance reduction and safety constraint satisfaction. Furthermore, we show our method is even superior to previous methods in both

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2024efficient

Related papers

• Behaviour Policy Optimization: Provably Lower Variance Return Estimates For Off-policy Reinforcement Learning (2025)0.00
• Doubly Optimal Policy Evaluation For Reinforcement Learning (2024)0.00
• Conservative Exploration For Policy Optimization Via Off-policy Policy Evaluation (2023)0.00
• Evaluation-aware Reinforcement Learning (2025)0.00
• Statistically Efficient Variance Reduction With Double Policy Estimation For Off-policy Evaluation In Sequence-modeled Reinforcement Learning (2023)0.00
• Efficient Evaluation Of Natural Stochastic Policies In Offline Reinforcement Learning (2020)0.00
• Robust On-policy Sampling For Data-efficient Policy Evaluation In Reinforcement Learning (2021)0.00
• Low Variance Off-policy Evaluation With State-based Importance Sampling (2022)0.00