Non-asymptotic Convergence Of Adam-type Reinforcement Learning Algorithms Under Markovian Sampling

Abstract

Despite the wide applications of Adam in reinforcement learning (RL), the theoretical convergence of Adam-type RL algorithms has not been established. This paper provides the first such convergence analysis for two fundamental RL algorithms of policy gradient (PG) and temporal difference (TD) learning that incorporate AMSGrad updates (a standard alternative of Adam in theoretical analysis), referred to as PG-AMSGrad and TD-AMSGrad, respectively. Moreover, our analysis focuses on Markovian sampling for both algorithms. We show that under general nonlinear function approximation, PG-AMSGrad with a constant stepsize converges to a neighborhood of a stationary point at the rate of \(\mathcal\{O\}(1/T)\) (where \(T\) denotes the number of iterations), and with a diminishing stepsize converges exactly to a stationary point at the rate of \(\mathcal\{O\}(log^2 T/\sqrt\{T\})\). Furthermore, under linear function approximation, TD-AMSGrad with a constant stepsize converges to a neighborhood of

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyxiong2020non

Related papers

• Adam On Local Time: Addressing Nonstationarity In RL With Relative Adam Timesteps (2024)0.00
• On The Linear Convergence Of Natural Policy Gradient Algorithm (2021)0.00
• Finite-sample Analysis Of Nonlinear Stochastic Approximation With Applications In Reinforcement Learning (2019)10.35
• On The Convergence Theory Of Debiased Model-agnostic Meta-reinforcement Learning (2020)0.00
• Finite Sample Analysis Of The GTD Policy Evaluation Algorithms In Markov Setting (2018)0.00
• Adversarially-robust TD Learning With Markovian Data: Finite-time Rates And Fundamental Limits (2025)0.00
• A Distributional Analysis Of Sampling-based Reinforcement Learning Algorithms (2020)0.00
• Finite Sample Analysis Of Two-timescale Stochastic Approximation With Applications To Reinforcement Learning (2017)0.00