Federated Stochastic Approximation Under Markov Noise And Heterogeneity: Applications In Reinforcement Learning

Abstract

Since reinforcement learning algorithms are notoriously data-intensive, the task of sampling observations from the environment is usually split across multiple agents. However, transferring these observations from the agents to a central location can be prohibitively expensive in terms of communication cost, and it can also compromise the privacy of each agent's local behavior policy. Federated reinforcement learning is a framework in which \(N\) agents collaboratively learn a global model, without sharing their individual data and policies. This global model is the unique fixed point of the average of \(N\) local operators, corresponding to the \(N\) agents. Each agent maintains a local copy of the global model and updates it using locally sampled data. In this paper, we show that by careful collaboration of the agents in solving this joint fixed point problem, we can find the global model \(N\) times faster, also known as linear speedup. We first propose a general framework for feder

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykhodadadian2022federated

Related papers

• Achieving Tighter Finite-time Rates For Heterogeneous Federated Stochastic Approximation Under Markovian Sampling (2025)0.00
• Local Stochastic Approximation: A Unified View Of Federated Learning And Distributed Multi-task Reinforcement Learning Algorithms (2020)0.00
• The Blessing Of Heterogeneity In Federated Q-learning: Linear Speedup And Beyond (2023)0.00
• Momentum For The Win: Collaborative Federated Reinforcement Learning Across Heterogeneous Environments (2024)0.00
• Finite-time Analysis Of On-policy Heterogeneous Federated Reinforcement Learning (2024)0.00
• Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysis (2025)0.00
• DSDF: An Approach To Handle Stochastic Agents In Collaborative Multi-agent Reinforcement Learning (2021)0.00
• Multi-agent Reinforcement Learning In Stochastic Networked Systems (2020)0.00