Asynchronous Policy Gradient Aggregation For Efficient Distributed Reinforcement Learning

Abstract

We study distributed reinforcement learning (RL) with policy gradient methods under asynchronous and parallel computations and communications. While non-distributed methods are well understood theoretically and have achieved remarkable empirical success, their distributed counterparts remain less explored, particularly in the presence of heterogeneous asynchronous computations and communication bottlenecks. We introduce two new algorithms, Rennala NIGT and Malenia NIGT, which implement asynchronous policy gradient aggregation and achieve state-of-the-art efficiency. In the homogeneous setting, Rennala NIGT provably improves the total computational and communication complexity while supporting the AllReduce operation. In the heterogeneous setting, Malenia NIGT simultaneously handles asynchronous computations and heterogeneous environments with strictly better theoretical guarantees. Our results are further corroborated by experiments, showing that our methods significantly outperform pr

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytyurin2025asynchronous

Related papers

• Communication-efficient Policy Gradient Methods For Distributed Reinforcement Learning (2018)13.05
• Fully Asynchronous Policy Evaluation In Distributed Reinforcement Learning Over Networks (2020)9.03
• Asynchronous Federated Reinforcement Learning With Policy Gradient Updates: Algorithm Design And Convergence Analysis (2024)0.00
• Scalable And Sample Efficient Distributed Policy Gradient Algorithms In Multi-agent Networked Systems (2022)0.00
• Federated Natural Policy Gradient And Actor Critic Methods For Multi-task Reinforcement Learning (2023)0.00
• Distributed Policy Gradient With Variance Reduction In Multi-agent Reinforcement Learning (2021)0.00
• Descent-guided Policy Gradient For Scalable Cooperative Multi-agent Learning (2026)0.00
• Improved Communication Efficiency In Federated Natural Policy Gradient Via Admm-based Gradient Updates (2023)0.00