Asynchronous Federated Reinforcement Learning With Policy Gradient Updates: Algorithm Design And Convergence Analysis

Abstract

To improve the efficiency of reinforcement learning (RL), we propose a novel asynchronous federated reinforcement learning (FedRL) framework termed AFedPG, which constructs a global model through collaboration among \(N\) agents using policy gradient (PG) updates. To address the challenge of lagged policies in asynchronous settings, we design a delay-adaptive lookahead technique \textit\{specifically for FedRL\} that can effectively handle heterogeneous arrival times of policy gradients. We analyze the theoretical global convergence bound of AFedPG, and characterize the advantage of the proposed algorithm in terms of both the sample complexity and time complexity. Specifically, our AFedPG method achieves \(O(\frac\{\{\epsilon\}^\{-2.5\}\}\{N\})\) sample complexity for global convergence at each agent on average. Compared to the single agent setting with \(O(\epsilon^\{-2.5\})\) sample complexity, it enjoys a linear speedup with respect to the number of agents. Moreover, compared to syn

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylan2024asynchronous

Related papers

• Improved Communication Efficiency In Federated Natural Policy Gradient Via Admm-based Gradient Updates (2023)0.00
• Global Convergence Guarantees For Federated Policy Gradient Methods With Adversaries (2024)0.00
• Federated Natural Policy Gradient And Actor Critic Methods For Multi-task Reinforcement Learning (2023)0.00
• The Gradient Convergence Bound Of Federated Multi-agent Reinforcement Learning With Efficient Communication (2021)0.00
• Momentum For The Win: Collaborative Federated Reinforcement Learning Across Heterogeneous Environments (2024)0.00
• On The Linear Speedup Of Personalized Federated Reinforcement Learning With Shared Representations (2024)0.00
• Fedhpd: Heterogeneous Federated Reinforcement Learning Via Policy Distillation (2025)2.26
• Asynchronous Policy Gradient Aggregation For Efficient Distributed Reinforcement Learning (2025)0.00