Measuring Progress In Deep Reinforcement Learning Sample Efficiency

Abstract

Sampled environment transitions are a critical input to deep reinforcement learning (DRL) algorithms. Current DRL benchmarks often allow for the cheap and easy generation of large amounts of samples such that perceived progress in DRL does not necessarily correspond to improved sample efficiency. As simulating real world processes is often prohibitively hard and collecting real world experience is costly, sample efficiency is an important indicator for economically relevant applications of DRL. We investigate progress in sample efficiency on Atari games and continuous control tasks by comparing the number of samples that a variety of algorithms need to reach a given performance level according to training curves in the corresponding publications. We find exponential progress in sample efficiency with estimated doubling times of around 10 to 18 months on Atari, 5 to 24 months on state-based continuous control and of around 4 to 9 months on pixel-based continuous control depending on the

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydorner2021measuring

Related papers

• Importance Of Using Appropriate Baselines For Evaluation Of Data-efficiency In Deep Reinforcement Learning For Atari (2020)0.00
• Off-policy RL Algorithms Can Be Sample-efficient For Continuous Control Via Sample Multiple Reuse (2023)0.00
• Efficientzero V2: Mastering Discrete And Continuous Control With Limited Data (2024)0.00
• Deep Reinforcement Learning At The Edge Of The Statistical Precipice (2021)0.00
• Is Deep Reinforcement Learning Really Superhuman On Atari? Leveling The Playing Field (2019)0.00
• Decoupled Exploration And Exploitation Policies For Sample-efficient Reinforcement Learning (2021)0.00
• Snapshot Reinforcement Learning: Leveraging Prior Trajectories For Efficiency (2024)0.00
• Human-inspired Framework To Accelerate Reinforcement Learning (2023)0.00