Global Convergence Of The ODE Limit For Online Actor-critic Algorithms In Reinforcement Learning

Abstract

Actor-critic algorithms are widely used in reinforcement learning, but are challenging to mathematically analyse due to the online arrival of non-i.i.d. data samples. The distribution of the data samples dynamically changes as the model is updated, introducing a complex feedback loop between the data distribution and the reinforcement learning algorithm. We prove that, under a time rescaling, the online actor-critic algorithm with tabular parametrization converges to an ordinary differential equation (ODE) as the number of updates becomes large. The proof first establishes the geometric ergodicity of the data samples under a fixed actor policy. Then, using a Poisson equation, we prove that the fluctuations of the data samples around a dynamic probability measure, which is a function of the evolving actor model, vanish as the number of updates become large. Once the ODE limit has been derived, we study its convergence properties using a two time-scale analysis which asymptotically de-co

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2021global

Related papers

• Weak Convergence Analysis Of Online Neural Actor-critic Algorithms (2024)0.00
• Finite-time Analysis Of Single-timescale Actor-critic (2022)0.00
• Single-timescale Actor-critic Provably Finds Globally Optimal Policy (2020)0.00
• Finite Sample Analysis Of Two-time-scale Natural Actor-critic Algorithm (2021)7.50
• Global Convergence Of Two-timescale Actor-critic For Solving Linear Quadratic Regulator (2022)4.52
• A Finite Time Analysis Of Two Time-scale Actor Critic Methods (2020)0.00
• Analysis Of A Target-based Actor-critic Algorithm With Linear Function Approximation (2021)0.00
• Finite-time Analysis Of Natural Actor-critic For Pomdps (2022)0.00