Sample Dropout: A Simple Yet Effective Variance Reduction Technique In Deep Policy Optimization

Abstract

Recent success in Deep Reinforcement Learning (DRL) methods has shown that policy optimization with respect to an off-policy distribution via importance sampling is effective for sample reuse. In this paper, we show that the use of importance sampling could introduce high variance in the objective estimate. Specifically, we show in a principled way that the variance of importance sampling estimate grows quadratically with importance ratios and the large ratios could consequently jeopardize the effectiveness of surrogate objective optimization. We then propose a technique called sample dropout to bound the estimation variance by dropping out samples when their ratio deviation is too high. We instantiate this sample dropout technique on representative policy optimization algorithms, including TRPO, PPO, and ESPO, and demonstrate that it consistently boosts the performance of those DRL algorithms on both continuous and discrete action controls, including MuJoCo, DMControl and Atari video

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylin2023sample

Related papers

• Dropout Strategy In Reinforcement Learning: Limiting The Surrogate Objective Variance In Policy Optimization Methods (2023)0.00
• Low Variance Off-policy Evaluation With State-based Importance Sampling (2022)0.00
• Stochastic Variance Reduction For Policy Gradient Estimation (2017)0.00
• Behaviour Policy Optimization: Provably Lower Variance Return Estimates For Off-policy Reinforcement Learning (2025)0.00
• Variance Reduction For Policy-gradient Methods Via Empirical Variance Minimization (2022)0.00
• Learning Sampling Policy For Faster Derivative Free Optimization (2021)0.00
• On The Reduction Of Variance And Overestimation Of Deep Q-learning (2019)0.00
• Policy Optimization Through Approximate Importance Sampling (2019)0.00