SEED RL: Scalable And Efficient Deep-rl With Accelerated Central Inference

Abstract

We present a modern scalable reinforcement learning agent called SEED (Scalable, Efficient Deep-RL). By effectively utilizing modern accelerators, we show that it is not only possible to train on millions of frames per second but also to lower the cost of experiments compared to current methods. We achieve this with a simple architecture that features centralized inference and an optimized communication layer. SEED adopts two state of the art distributed algorithms, IMPALA/V-trace (policy gradients) and R2D2 (Q-learning), and is evaluated on Atari-57, DeepMind Lab and Google Research Football. We improve the state of the art on Football and are able to reach state of the art on Atari-57 three times faster in wall-time. For the scenarios we consider, a 40% to 80% cost reduction for running experiments is achieved. The implementation along with experiments is open-sourced so results can be reproduced and novel ideas tried out.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyespeholt2019seed

Related papers

• Deep Reinforcement Learning At The Edge Of The Statistical Precipice (2021)0.00
• A3C-S: Automated Agent Accelerator Co-search Towards Efficient Deep Reinforcement Learning (2021)0.00
• Accelerated Methods For Deep Reinforcement Learning (2018)0.00
• Is Deep Reinforcement Learning Really Superhuman On Atari? Leveling The Playing Field (2019)0.00
• Google Research Football: A Novel Reinforcement Learning Environment (2019)17.22
• Importance Of Using Appropriate Baselines For Evaluation Of Data-efficiency In Deep Reinforcement Learning For Atari (2020)0.00
• Human-inspired Framework To Accelerate Reinforcement Learning (2023)0.00
• Cleanrl: High-quality Single-file Implementations Of Deep Reinforcement Learning Algorithms (2021)0.00