Global Convergence Of Two-timescale Actor-critic For Solving Linear Quadratic Regulator

Abstract

The actor-critic (AC) reinforcement learning algorithms have been the powerhouse behind many challenging applications. Nevertheless, its convergence is fragile in general. To study its instability, existing works mostly consider the uncommon double-loop variant or basic models with finite state and action space. We investigate the more practical single-sample two-timescale AC for solving the canonical linear quadratic regulator (LQR) problem, where the actor and the critic update only once with a single sample in each iteration on an unbounded continuous state and action space. Existing analysis cannot conclude the convergence for such a challenging case. We develop a new analysis framework that allows establishing the global convergence to an \(\epsilon\)-optimal solution with at most an \(\mathcal\{O\}(\epsilon^\{-2.5\})\) sample complexity. To our knowledge, this is the first finite-time convergence analysis for the single sample two-timescale AC for solving LQR with global optimali

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keychen2022global

Related papers

• Finite-time Analysis Of Fully Decentralized Single-timescale Actor-critic (2022)0.00
• Natural Actor-critic Converges Globally For Hierarchical Linear Quadratic Regulator (2019)0.00
• A Finite Time Analysis Of Two Time-scale Actor Critic Methods (2020)0.00
• Finite-time Analysis Of Single-timescale Actor-critic (2022)0.00
• Single-timescale Actor-critic Provably Finds Globally Optimal Policy (2020)0.00
• Non-asymptotic Convergence Analysis Of Two Time-scale (natural) Actor-critic Algorithms (2020)0.00
• Non-asymptotic Analysis For Single-loop (natural) Actor-critic With Compatible Function Approximation (2024)0.00
• Global Convergence Of The ODE Limit For Online Actor-critic Algorithms In Reinforcement Learning (2021)0.00