Fully Asynchronous Policy Evaluation In Distributed Reinforcement Learning Over Networks

Abstract

This paper proposes a *fully asynchronous* scheme for the policy evaluation problem of distributed reinforcement learning (DisRL) over directed peer-to-peer networks. Without waiting for any other node of the network, each node can locally update its value function at any time by using (possibly delayed) information from its neighbors. This is in sharp contrast to the gossip-based scheme where a pair of nodes concurrently update. Though the fully asynchronous setting involves a difficult multi-timescale decision problem, we design a novel stochastic average gradient (SAG) based distributed algorithm and develop a push-pull augmented graph approach to prove its exact convergence at a linear rate of \(\mathcal\{O\}(c^k)\) where \(c\in(0,1)\) and \(k\) increases by one no matter on which node updates. Finally, numerical experiments validate that our method speeds up linearly with respect to the number of nodes, and is robust to straggler nodes.

Stats

• citations15
• S2 citations—
• github stars0
• HF likes0
• heat score9.03
• arxiv keysha2020fully

Related papers

• Asynchronous Policy Gradient Aggregation For Efficient Distributed Reinforcement Learning (2025)0.00
• Communication-efficient Policy Gradient Methods For Distributed Reinforcement Learning (2018)13.05
• Edge-compatible Reinforcement Learning For Recommendations (2021)0.00
• Scalable And Sample Efficient Distributed Policy Gradient Algorithms In Multi-agent Networked Systems (2022)0.00
• Scalable Reinforcement Learning For Multi-agent Networked Systems (2019)10.35
• Asynchronous Federated Reinforcement Learning With Policy Gradient Updates: Algorithm Design And Convergence Analysis (2024)0.00
• A Multi-agent Off-policy Actor-critic Algorithm For Distributed Reinforcement Learning (2019)11.39
• Global Convergence Guarantees For Federated Policy Gradient Methods With Adversaries (2024)0.00