Momentum For The Win: Collaborative Federated Reinforcement Learning Across Heterogeneous Environments
2024 · Han Wang, Sihong He, Zhili Zhang, et al.
Abstract
We explore a Federated Reinforcement Learning (FRL) problem where \(N\) agents collaboratively learn a common policy without sharing their trajectory data. To date, existing FRL work has primarily focused on agents operating in the same or ``similar" environments. In contrast, our problem setup allows for arbitrarily large levels of environment heterogeneity. To obtain the optimal policy which maximizes the average performance across all potentially completely different environments, we propose two algorithms: FedSVRPG-M and FedHAPG-M. In contrast to existing results, we demonstrate that both FedSVRPG-M and FedHAPG-M, both of which leverage momentum mechanisms, can exactly converge to a stationary point of the average performance function, regardless of the magnitude of environment heterogeneity. Furthermore, by incorporating the benefits of variance-reduction techniques or Hessian approximation, both algorithms achieve state-of-the-art convergence results, characterized by a sample co
Authors
(none)
Tags
Stats
Related papers
- Momentum-based Federated Reinforcement Learning With Interaction And Communication Efficiency (2024)4.52
- Finite-time Analysis Of On-policy Heterogeneous Federated Reinforcement Learning (2024)0.00
- On The Linear Speedup Of Personalized Federated Reinforcement Learning With Shared Representations (2024)0.00
- Federated Reinforcement Learning With Constraint Heterogeneity (2024)0.00
- Global Convergence Guarantees For Federated Policy Gradient Methods With Adversaries (2024)0.00
- Fedhpd: Heterogeneous Federated Reinforcement Learning Via Policy Distillation (2025)2.26
- Federated Natural Policy Gradient And Actor Critic Methods For Multi-task Reinforcement Learning (2023)0.00
- Federated Stochastic Approximation Under Markov Noise And Heterogeneity: Applications In Reinforcement Learning (2022)0.00