Policy Optimization Through Approximate Importance Sampling

Abstract

Recent policy optimization approaches (Schulman et al., 2015a; 2017) have achieved substantial empirical successes by constructing new proxy optimization objectives. These proxy objectives allow stable and low variance policy learning, but require small policy updates to ensure that the proxy objective remains an accurate approximation of the target policy value. In this paper we derive an alternative objective that obtains the value of the target policy by applying importance sampling (IS). However, the basic importance sampled objective is not suitable for policy optimization, as it incurs too high variance in policy updates. We therefore introduce an approximation that allows us to directly trade-off the bias of approximation with the variance in policy updates. We show that our approximation unifies previously developed approaches and allows us to interpolate between them. We develop a practical algorithm by optimizing the introduced objective with proximal policy optimization tech

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytomczak2019policy

Related papers

• Cautiously Optimistic Policy Optimization And Exploration With Linear Function Approximation (2021)0.00
• Low Variance Off-policy Evaluation With State-based Importance Sampling (2022)0.00
• Sample Dropout: A Simple Yet Effective Variance Reduction Technique In Deep Policy Optimization (2023)0.00
• A Parametric Class Of Approximate Gradient Updates For Policy Optimization (2022)0.00
• Conservative Optimistic Policy Optimization Via Multiple Importance Sampling (2021)0.00
• Low-switching Policy Gradient With Exploration Via Online Sensitivity Sampling (2023)0.00
• Importance Sampling Policy Evaluation With An Estimated Behavior Policy (2018)0.00
• Proximal Policy Optimization Algorithms (2017)0.00