Empirical Design In Reinforcement Learning

Abstract

Empirical design in reinforcement learning is no small task. Running good experiments requires attention to detail and at times significant computational resources. While compute resources available per dollar have continued to grow rapidly, so have the scale of typical experiments in reinforcement learning. It is now common to benchmark agents with millions of parameters against dozens of tasks, each using the equivalent of 30 days of experience. The scale of these experiments often conflict with the need for proper statistical evidence, especially when comparing algorithms. Recent studies have highlighted how popular algorithms are sensitive to hyper-parameter settings and implementation details, and that common empirical practice leads to weak statistical evidence (Machado et al., 2018; Henderson et al., 2018). Here we take this one step further. This manuscript represents both a call to action, and a comprehensive resource for how to do good experiments in reinforcement learning.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypatterson2023empirical

Related papers

• What Matters In On-policy Reinforcement Learning? A Large-scale Empirical Study (2020)0.00
• Performance Comparisons Of Reinforcement Learning Algorithms For Sequential Experimental Design (2025)0.00
• An Empirical Investigation Of The Challenges Of Real-world Reinforcement Learning (2020)0.00
• Statistically Efficient Bayesian Sequential Experiment Design Via Reinforcement Learning With Cross-entropy Estimators (2023)0.00
• An Experimental Design Perspective On Model-based Reinforcement Learning (2021)0.00
• Sequential Bayesian Experimental Designs Via Reinforcement Learning (2022)0.00
• A Comprehensive Survey Of Reinforcement Learning: From Algorithms To Practical Challenges (2024)0.00
• Empirical Policy Evaluation With Supergraphs (2020)0.00