SRL: Scaling Distributed Reinforcement Learning To Over Ten Thousand Cores

Abstract

The ever-growing complexity of reinforcement learning (RL) tasks demands a distributed system to efficiently generate and process a massive amount of data. However, existing open-source libraries suffer from various limitations, which impede their practical use in challenging scenarios where large-scale training is necessary. In this paper, we present a novel abstraction on the dataflows of RL training, which unifies diverse RL training applications into a general framework. Following this abstraction, we develop a scalable, efficient, and extensible distributed RL system called ReaLlyScalableRL, which allows efficient and massively parallelized training and easy development of customized algorithms. Our evaluation shows that SRL outperforms existing academic libraries, reaching at most 21x higher training throughput in a distributed setting. On learning performance, beyond performing and scaling well on common RL benchmarks with different RL algorithms, SRL can reproduce the same solu

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymei2023srl

Related papers

• Rllib Flow: Distributed Reinforcement Learning Is A Dataflow Problem (2020)0.00
• Surreal-system: Fully-integrated Stack For Distributed Deep Reinforcement Learning (2019)0.00
• Rllib: Abstractions For Distributed Reinforcement Learning (2017)0.00
• Integrating Distributed Architectures In Highly Modular RL Libraries (2020)0.00
• MSRL: Distributed Reinforcement Learning With Dataflow Fragments (2022)0.00
• Distributed Deep Reinforcement Learning: An Overview (2020)0.00
• Computerrl: Scaling End-to-end Online Reinforcement Learning For Computer Use Agents (2025)0.00
• A Scalable And Reproducible System-on-chip Simulation For Reinforcement Learning (2021)0.00