Measuring Sample Efficiency And Generalization In Reinforcement Learning Benchmarks: Neurips 2020 Procgen Benchmark

Abstract

The NeurIPS 2020 Procgen Competition was designed as a centralized benchmark with clearly defined tasks for measuring Sample Efficiency and Generalization in Reinforcement Learning. Generalization remains one of the most fundamental challenges in deep reinforcement learning, and yet we do not have enough benchmarks to measure the progress of the community on Generalization in Reinforcement Learning. We present the design of a centralized benchmark for Reinforcement Learning which can help measure Sample Efficiency and Generalization in Reinforcement Learning by doing end to end evaluation of the training and rollout phases of thousands of user submitted code bases in a scalable way. We designed the benchmark on top of the already existing Procgen Benchmark by defining clear tasks and standardizing the end to end evaluation setups. The design aims to maximize the flexibility available for researchers who wish to design future iterations of such benchmarks, and yet imposes necessary prac

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymohanty2021measuring

Related papers

• Leveraging Procedural Generation To Benchmark Reinforcement Learning (2019)0.00
• Improving Generalization On The Procgen Benchmark With Simple Architectural Changes And Scale (2024)0.00
• Assessing Generalization In Deep Reinforcement Learning (2018)0.00
• The Principle Of Unchanged Optimality In Reinforcement Learning Generalization (2019)0.00
• Quantifying Generalization In Reinforcement Learning (2018)0.00
• Measuring And Characterizing Generalization In Deep Reinforcement Learning (2018)9.76
• Procedural Generalization By Planning With Self-supervised World Models (2021)0.00
• Benchmarking Robustness And Generalization In Multi-agent Systems: A Case Study On Neural MMO (2023)0.00