Finite-time Analysis Of Single-timescale Actor-critic

Abstract

Actor-critic methods have achieved significant success in many challenging applications. However, its finite-time convergence is still poorly understood in the most practical single-timescale form. Existing works on analyzing single-timescale actor-critic have been limited to i.i.d. sampling or tabular setting for simplicity. We investigate the more practical online single-timescale actor-critic algorithm on continuous state space, where the critic assumes linear function approximation and updates with a single Markovian sample per actor step. Previous analysis has been unable to establish the convergence for such a challenging scenario. We demonstrate that the online single-timescale actor-critic method provably finds an \(\epsilon\)-approximate stationary point with \(\widetilde\{\mathcal\{O\}\}(\epsilon^\{-2\})\) sample complexity under standard assumptions, which can be further improved to \(\mathcal\{O\}(\epsilon^\{-2\})\) under the i.i.d. sampling. Our novel framework systematica

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2022finite

Related papers

• A Finite Time Analysis Of Two Time-scale Actor Critic Methods (2020)0.00
• Finite Sample Analysis Of Two-time-scale Natural Actor-critic Algorithm (2021)7.50
• Single-timescale Actor-critic Provably Finds Globally Optimal Policy (2020)0.00
• Finite-time Analysis Of Fully Decentralized Single-timescale Actor-critic (2022)0.00
• Global Convergence Of The ODE Limit For Online Actor-critic Algorithms In Reinforcement Learning (2021)0.00
• Global Convergence Of Two-timescale Actor-critic For Solving Linear Quadratic Regulator (2022)4.52
• Analysis Of A Target-based Actor-critic Algorithm With Linear Function Approximation (2021)0.00
• Weak Convergence Analysis Of Online Neural Actor-critic Algorithms (2024)0.00