Never Worse, Mostly Better: Stable Policy Improvement In Deep Reinforcement Learning

Abstract

In recent years, there has been significant progress in applying deep reinforcement learning (RL) for solving challenging problems across a wide variety of domains. Nevertheless, convergence of various methods has been shown to suffer from inconsistencies, due to algorithmic instability and variance, as well as stochasticity in the benchmark environments. Particularly, despite the fact that the agent's performance may be improving on average, it may abruptly deteriorate at late stages of training. In this work, we study methods for enhancing the agent's learning process, by providing conservative updates with respect to either the obtained history or a reference benchmark policy. Our method, termed EVEREST, obtains high confidence improvements via confidence bounds of a reference policy. Through extensive empirical analysis we demonstrate the benefit of our approach in terms of both performance and stabilization, with significant improvements in continuous control and Atari benchmarks.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykhanna2019never

Related papers

• Moments Matter:stabilizing Policy Optimization Using Return Distributions (2026)0.00
• Deep Reinforcement Learning At The Edge Of The Statistical Precipice (2021)0.00
• Conservative Exploration For Policy Optimization Via Off-policy Policy Evaluation (2023)0.00
• Constrained Policy Improvement For Safe And Efficient Reinforcement Learning (2018)0.00
• Model-agnostic Solutions For Deep Reinforcement Learning In Non-ergodic Contexts (2026)0.00
• Value Enhancement Of Reinforcement Learning Via Efficient And Robust Trust Region Optimization (2023)0.00
• Regularization Matters In Policy Optimization (2019)2.68
• Reproducibility Of Benchmarked Deep Reinforcement Learning Tasks For Continuous Control (2017)0.00