Doubly Optimal Policy Evaluation For Reinforcement Learning

Abstract

Policy evaluation estimates the performance of a policy by (1) collecting data from the environment and (2) processing raw data into a meaningful estimate. Due to the sequential nature of reinforcement learning, any improper data-collecting policy or data-processing method substantially deteriorates the variance of evaluation results over long time steps. Thus, policy evaluation often suffers from large variance and requires massive data to achieve the desired accuracy. In this work, we design an optimal combination of data-collecting policy and data-processing baseline. Theoretically, we prove our doubly optimal policy evaluation method is unbiased and guaranteed to have lower variance than previously best-performing methods. Empirically, compared with previous works, we show our method reduces variance substantially and achieves superior empirical performance.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2024doubly

Related papers

• Efficient Policy Evaluation With Safety Constraint For Reinforcement Learning (2024)0.00
• Doubly Robust Interval Estimation For Optimal Policy Evaluation In Online Learning (2021)0.00
• Behaviour Policy Optimization: Provably Lower Variance Return Estimates For Off-policy Reinforcement Learning (2025)0.00
• Statistically Efficient Variance Reduction With Double Policy Estimation For Off-policy Evaluation In Sequence-modeled Reinforcement Learning (2023)0.00
• Evaluation-aware Reinforcement Learning (2025)0.00
• Robust On-policy Sampling For Data-efficient Policy Evaluation In Reinforcement Learning (2021)0.00
• Intrinsically Efficient, Stable, And Bounded Off-policy Evaluation For Reinforcement Learning (2019)0.00
• Online Estimation And Inference For Robust Policy Evaluation In Reinforcement Learning (2023)2.26