A Finite Time Analysis Of Distributed Q-learning

Abstract

Multi-agent reinforcement learning (MARL) has witnessed a remarkable surge in interest, fueled by the empirical success achieved in applications of single-agent reinforcement learning (RL). In this study, we consider a distributed Q-learning scenario, wherein a number of agents cooperatively solve a sequential decision making problem without access to the central reward function which is an average of the local rewards. In particular, we study finite-time analysis of a distributed Q-learning algorithm, and provide a new sample complexity result of \(\tilde\{\mathcal\{O\}\}\left( \min\left\\{\frac\{1\}\{\epsilon^2\}\frac\{t_\{\text\{mix\}\}\}\{(1-\gamma)^6 d_\{\min\}^4 \} ,\frac\{1\}\{\epsilon\}\frac\{\sqrt\{|\gS||\gA|\}\}\{(1-\sigma_2(\boldsymbol\{W\}))(1-\gamma)^4 d_\{\min\}^3\} \right\\}\right)\) under tabular lookup

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylim2024a

Related papers

• Provably Efficient Multi-agent Reinforcement Learning With Fully Decentralized Communication (2021)0.00
• A Law Of Iterated Logarithm For Multi-agent Reinforcement Learning (2021)0.00
• Sample Complexity Of Average-reward Q-learning: From Single-agent To Federated Reinforcement Learning (2026)0.00
• Unsynchronized Decentralized Q-learning: Two Timescale Analysis By Persistence (2023)2.26
• The Blessing Of Heterogeneity In Federated Q-learning: Linear Speedup And Beyond (2023)0.00
• Multi-agent Reinforcement Learning In Stochastic Networked Systems (2020)0.00
• Finite-sample Analysis For Decentralized Batch Multi-agent Reinforcement Learning With Networked Agents (2018)10.07
• V-learning -- A Simple, Efficient, Decentralized Algorithm For Multiagent RL (2021)0.00