Stabilizing Policy Gradient Methods Via Reward Profiling

Abstract

Policy gradient methods, which have been extensively studied in the last decade, offer an effective and efficient framework for reinforcement learning problems. However, their performances can often be unsatisfactory, suffering from unreliable reward improvements and slow convergence, due to high variance in gradient estimations. In this paper, we propose a universal reward profiling framework that can be seamlessly integrated with any policy gradient algorithm, where we selectively update the policy based on high-confidence performance estimations. We theoretically justify that our technique will not slow down the convergence of the baseline policy gradient methods, but with high probability, will result in stable and monotonic improvements of their performance. Empirically, on eight continuous-control benchmarks (Box2D and MuJoCo/PyBullet), our profiling yields up to 1.5x faster convergence to near-optimal returns, up to 1.75x reduction in return variance on some setups. Our profilin

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyahmed2025stabilizing

Related papers

• Model-free Policy Learning With Reward Gradients (2021)0.00
• Smoothing Policies And Safe Policy Gradients (2019)7.50
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00
• Global Convergence Of Policy Gradient Methods In Reinforcement Learning, Games And Control (2023)0.00
• The Reinforce Policy Gradient Algorithm Revisited (2023)0.00
• Score-aware Policy-gradient And Performance Guarantees Using Local Lyapunov Stability (2023)0.00
• Policy Gradient From Demonstration And Curiosity (2020)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00