Regularization Matters In Policy Optimization

Abstract

Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention thanks to its encouraging performance on a variety of control tasks. Yet, conventional regularization techniques in training neural networks (e.g., \(L_2\) regularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment, and because the deep RL community focuses more on high-level algorithm designs. In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks. Interestingly, we find conventional regularization techniques on the policy networks can often bring large improvement, especially on harder tasks. Our findings are shown to be robust against training hyperparameter variations. We also compare these techniques with the more widely used entropy regularization. In addition, we study regularizing different components and

Stats

• citations0
• S2 citations—
• github stars21
• HF likes0
• heat score2.68
• arxiv keyliu2019regularization

Related papers

• Iterative Amortized Policy Optimization (2020)0.00
• Optimal Scheduling Of Entropy Regulariser For Continuous-time Linear-quadratic Reinforcement Learning (2022)4.52
• REBEL: Reward Regularization-based Approach For Robotic Reinforcement Learning From Human Feedback (2023)0.00
• A Comparative Theoretical Analysis Of Entropy Control Methods In Reinforcement Learning (2026)0.00
• Overestimation, Overfitting, And Plasticity In Actor-critic: The Bitter Lesson Of Reinforcement Learning (2024)0.00
• What Matters In On-policy Reinforcement Learning? A Large-scale Empirical Study (2020)0.00
• Promoting Coordination Through Policy Regularization In Multi-agent Deep Reinforcement Learning (2019)0.00
• Decoupling Regularization From The Action Space (2024)0.00