Asynchronous Stochastic Approximations With Asymptotically Biased Errors And Deep Multi-agent Learning

Abstract

Asynchronous stochastic approximations (SAs) are an important class of model-free algorithms, tools and techniques that are popular in multi-agent and distributed control scenarios. To counter Bellman's curse of dimensionality, such algorithms are coupled with function approximations. Although the learning/ control problem becomes more tractable, function approximations affect stability and convergence. In this paper, we present verifiable sufficient conditions for stability and convergence of asynchronous SAs with biased approximation errors. The theory developed herein is used to analyze Policy Gradient methods and noisy Value Iteration schemes. Specifically, we analyze the asynchronous approximate counterparts of the policy gradient (A2PG) and value iteration (A2VI) schemes. It is shown that the stability of these algorithms is unaffected by biased approximation errors, provided they are asymptotically bounded. With respect to convergence (of A2VI and A2PG), a relationship between t

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyramaswamy2018asynchronous

Related papers

• Non-asymptotic Analysis Of Biased Stochastic Approximation Scheme (2019)0.00
• On The Convergence Of Consensus Algorithms With Markovian Noise And Gradient Bias (2020)0.00
• Multi-agent Policy Optimization With Approximatively Synchronous Advantage Estimation (2020)0.00
• Distributed Value Function Approximation For Collaborative Multi-agent Reinforcement Learning (2020)8.60
• DASA: Delay-adaptive Multi-agent Stochastic Approximation (2024)2.26
• Stochastic Approximation With Delayed Updates: Finite-time Rates Under Markovian Sampling (2024)0.00
• Sample Complexity Bounds For Two Timescale Value-based Reinforcement Learning Algorithms (2020)0.00
• Adaptive Approximate Policy Iteration (2020)0.00