A Finite Time Analysis Of Two Time-scale Actor Critic Methods

Abstract

Actor-critic (AC) methods have exhibited great empirical success compared with other reinforcement learning algorithms, where the actor uses the policy gradient to improve the learning policy and the critic uses temporal difference learning to estimate the policy gradient. Under the two time-scale learning rate schedule, the asymptotic convergence of AC has been well studied in the literature. However, the non-asymptotic convergence and finite sample complexity of actor-critic methods are largely open. In this work, we provide a non-asymptotic analysis for two time-scale actor-critic methods under non-i.i.d. setting. We prove that the actor-critic method is guaranteed to find a first-order stationary point (i.e., \(\|\nabla J(\boldsymbol\{\theta\})\|_2^2 \le \epsilon\)) of the non-concave performance function \(J(\boldsymbol\{\theta\})\), with \(\mathcal\{\tilde\{O\}\}(\epsilon^\{-2.5\})\) sample complexity. To the best of our knowledge, this is the first work providing finite-time ana

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywu2020a

Related papers

• Non-asymptotic Convergence Analysis Of Two Time-scale (natural) Actor-critic Algorithms (2020)0.00
• Finite-time Analysis Of Single-timescale Actor-critic (2022)0.00
• Finite-time Analysis Of Fully Decentralized Single-timescale Actor-critic (2022)0.00
• Non-asymptotic Analysis For Single-loop (natural) Actor-critic With Compatible Function Approximation (2024)0.00
• Finite Sample Analysis Of Two-time-scale Natural Actor-critic Algorithm (2021)7.50
• Global Convergence Of Two-timescale Actor-critic For Solving Linear Quadratic Regulator (2022)4.52
• Single-timescale Actor-critic Provably Finds Globally Optimal Policy (2020)0.00
• Improving Sample Complexity Bounds For (natural) Actor-critic Algorithms (2020)0.00