A Small Gain Analysis of Single Timescale Actor Critic

Abstract

We consider a version of actor-critic which uses proportional step-sizes and only one critic update with a single sample from the stationary distribution per actor step. We provide an analysis of this method using the small-gain theorem. Specifically, we prove that this method can be used to find a stationary point, and that the resulting sample complexity improves the state of the art for actor-critic methods to $O\left({\mu }^{-2}{ϵ}^{-2}\right)$ to find an $ϵ$-approximate stationary point where $\mu$ is the condition number associated with the critic.