Vmfer: Von Mises-fisher Experience Resampling Based On Uncertainty Of Gradient Directions For Policy Improvement

Abstract

Reinforcement Learning (RL) is a widely employed technique in decision-making problems, encompassing two fundamental operations -- policy evaluation and policy improvement. Enhancing learning efficiency remains a key challenge in RL, with many efforts focused on using ensemble critics to boost policy evaluation efficiency. However, when using multiple critics, the actor in the policy improvement process can obtain different gradients. Previous studies have combined these gradients without considering their disagreements. Therefore, optimizing the policy improvement process is crucial to enhance learning efficiency. This study focuses on investigating the impact of gradient disagreements caused by ensemble critics on policy improvement. We introduce the concept of uncertainty of gradient directions as a means to measure the disagreement among gradients utilized in the policy improvement process. Through measuring the disagreement among gradients, we find that transitions with lower unce

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhu2024vmfer

Related papers

• Accelerating Residual Reinforcement Learning With Uncertainty Estimation (2025)0.00
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00
• An Empirical Analysis Of Measure-valued Derivatives For Policy Gradients (2021)0.00
• Variance Reduction For Policy-gradient Methods Via Empirical Variance Minimization (2022)0.00
• An Analysis Of Measure-valued Derivatives For Policy Gradients (2022)2.26
• Stochastic Variance Reduction For Policy Gradient Estimation (2017)0.00
• Multilinear Tensor Low-rank Approximation For Policy-gradient Methods In Reinforcement Learning (2025)0.00
• Adaptive Multi-fidelity Reinforcement Learning For Variance Reduction In Engineering Design Optimization (2025)0.00