Merging Deterministic Policy Gradient Estimations With Varied Bias-variance Tradeoff For Effective Deep Reinforcement Learning

Abstract

Deep reinforcement learning (DRL) on Markov decision processes (MDPs) with continuous action spaces is often approached by directly training parametric policies along the direction of estimated policy gradients (PGs). Previous research revealed that the performance of these PG algorithms depends heavily on the bias-variance tradeoffs involved in estimating and using PGs. A notable approach towards balancing this tradeoff is to merge both on-policy and off-policy gradient estimations. However existing PG merging methods can be sample inefficient and are not suitable to train deterministic policies directly. To address these issues, this paper introduces elite PGs and strengthens their variance reduction effect by adopting elitism and policy consolidation techniques to regularize policy training based on policy behavioral knowledge extracted from elite trajectories. Meanwhile, we propose a two-step method to merge elite PGs and conventional PGs as a new extension of the conventional inte

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2019merging

Related papers

• Interpolated Policy Gradient: Merging On-policy And Off-policy Gradient Estimation For Deep Reinforcement Learning (2017)0.00
• Mixed Policy Gradient: Off-policy Reinforcement Learning Driven Jointly By Data And Model (2021)0.00
• Variance Reduction Based Partial Trajectory Reuse To Accelerate Policy Gradient Optimization (2022)0.00
• On The Convergence And Sample Efficiency Of Variance-reduced Policy Gradient Method (2021)0.00
• Revisiting Estimation Bias In Policy Gradients For Deep Reinforcement Learning (2023)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Stochastic Variance Reduction For Policy Gradient Estimation (2017)0.00
• Variance Reduction For Policy Gradient With Action-dependent Factorized Baselines (2018)0.00