Momentum-based Federated Reinforcement Learning With Interaction And Communication Efficiency

Abstract

Federated Reinforcement Learning (FRL) has garnered increasing attention recently. However, due to the intrinsic spatio-temporal non-stationarity of data distributions, the current approaches typically suffer from high interaction and communication costs. In this paper, we introduce a new FRL algorithm, named \(\texttt\{MFPO\}\), that utilizes momentum, importance sampling, and additional server-side adjustment to control the shift of stochastic policy gradients and enhance the efficiency of data utilization. We prove that by proper selection of momentum parameters and interaction frequency, \(\texttt\{MFPO\}\) can achieve \(\tilde\{\mathcal\{O\}\}(H N^\{-1\}\epsilon^\{-3/2\})\) and \(\tilde\{\mathcal\{O\}\}(\epsilon^\{-1\})\) interaction and communication complexities (\(N\) represents the number of agents), where the interaction complexity achieves linear speedup with the number of agents, and the communication complexity aligns the best achievable of existing first-order FL algorith

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keyyue2024momentum

Related papers

• Momentum For The Win: Collaborative Federated Reinforcement Learning Across Heterogeneous Environments (2024)0.00
• Finite-time Analysis Of On-policy Heterogeneous Federated Reinforcement Learning (2024)0.00
• Federated Offline Policy Optimization With Dual Regularization (2024)3.58
• Communication-efficient Consensus Mechanism For Federated Reinforcement Learning (2022)6.77
• The Gradient Convergence Bound Of Federated Multi-agent Reinforcement Learning With Efficient Communication (2021)0.00
• Global Convergence Guarantees For Federated Policy Gradient Methods With Adversaries (2024)0.00
• Fault-tolerant Federated Reinforcement Learning With Theoretical Guarantee (2021)0.00
• Federated Ensemble Model-based Reinforcement Learning In Edge Computing (2021)11.58