Rllib Flow: Distributed Reinforcement Learning Is A Dataflow Problem

Abstract

Researchers and practitioners in the field of reinforcement learning (RL) frequently leverage parallel computation, which has led to a plethora of new algorithms and systems in the last few years. In this paper, we re-examine the challenges posed by distributed RL and try to view it through the lens of an old idea: distributed dataflow. We show that viewing RL as a dataflow problem leads to highly composable and performant implementations. We propose RLlib Flow, a hybrid actor-dataflow programming model for distributed RL, and validate its practicality by porting the full suite of algorithms in RLlib, a widely adopted distributed RL library. Concretely, RLlib Flow provides 2-9 code savings in real production code and enables the composition of multi-agent algorithms not possible by end users before. The open-source code is available as part of RLlib at https://github.com/ray-project/ray/tree/master/rllib.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliang2020rllib

Code

• ray-project/ray—

Related papers

• Rllib: Abstractions For Distributed Reinforcement Learning (2017)0.00
• MSRL: Distributed Reinforcement Learning With Dataflow Fragments (2022)0.00
• SRL: Scaling Distributed Reinforcement Learning To Over Ten Thousand Cores (2023)0.00
• Distributed Deep Reinforcement Learning: An Overview (2020)0.00
• Integrating Distributed Architectures In Highly Modular RL Libraries (2020)0.00
• Reverse Flow Matching: A Unified Framework For Online Reinforcement Learning With Diffusion And Flow Policies (2026)0.00
• Ray: A Distributed Framework For Emerging AI Applications (2017)0.00
• Composite Flow Matching For Reinforcement Learning With Shifted-dynamics Data (2025)0.00