What Matters In On-policy Reinforcement Learning? A Large-scale Empirical Study

Abstract

In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20]. As a step towards filling that gap, we implement >50 such ``choices'' in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyandrychowicz2020what

Related papers

• A Comprehensive Survey Of Reinforcement Learning: From Algorithms To Practical Challenges (2024)0.00
• An Empirical Investigation Of The Challenges Of Real-world Reinforcement Learning (2020)0.00
• Statistical Reinforcement Learning In The Real World: A Survey Of Challenges And Future Directions (2026)0.00
• Regularization Matters In Policy Optimization (2019)2.68
• Investigating The Impact Of Action Representations In Policy Gradient Algorithms (2023)0.00
• DDPG++: Striving For Simplicity In Continuous-control Off-policy Reinforcement Learning (2020)0.00
• Human-inspired Framework To Accelerate Reinforcement Learning (2023)0.00
• Reproducibility Of Benchmarked Deep Reinforcement Learning Tasks For Continuous Control (2017)0.00