Weak Convergence Analysis Of Online Neural Actor-critic Algorithms

Abstract

We prove that a single-layer neural network trained with the online actor critic algorithm converges in distribution to a random ordinary differential equation (ODE) as the number of hidden units and the number of training steps \(\rightarrow \infty\). In the online actor-critic algorithm, the distribution of the data samples dynamically changes as the model is updated, which is a key challenge for any convergence analysis. We establish the geometric ergodicity of the data samples under a fixed actor policy. Then, using a Poisson equation, we prove that the fluctuations of the model updates around the limit distribution due to the randomly-arriving data samples vanish as the number of parameter updates \(\rightarrow \infty\). Using the Poisson equation and weak convergence techniques, we prove that the actor neural network and critic neural network converge to the solutions of a system of ODEs with random initial conditions. Analysis of the limit ODE shows that the limit critic network

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylam2024weak

Related papers

• Global Convergence Of The ODE Limit For Online Actor-critic Algorithms In Reinforcement Learning (2021)0.00
• Finite-time Analysis Of Single-timescale Actor-critic (2022)0.00
• Single-timescale Actor-critic Provably Finds Globally Optimal Policy (2020)0.00
• Finite Sample Analysis Of Two-time-scale Natural Actor-critic Algorithm (2021)7.50
• A Finite Time Analysis Of Two Time-scale Actor Critic Methods (2020)0.00
• Convergence Proof For Actor-critic Methods Applied To PPO And RUDDER (2020)11.67
• Neural Policy Gradient Methods: Global Optimality And Rates Of Convergence (2019)0.00
• Global Convergence Of Two-timescale Actor-critic For Solving Linear Quadratic Regulator (2022)4.52