Non-asymptotic Analysis For Single-loop (natural) Actor-critic With Compatible Function Approximation

Abstract

Actor-critic (AC) is a powerful method for learning an optimal policy in reinforcement learning, where the critic uses algorithms, e.g., temporal difference (TD) learning with function approximation, to evaluate the current policy and the actor updates the policy along an approximate gradient direction using information from the critic. This paper provides the \textit\{tightest\} non-asymptotic convergence bounds for both the AC and natural AC (NAC) algorithms. Specifically, existing studies show that AC converges to an \(\epsilon+\epsilon_\{\text\{critic\}\}\) neighborhood of stationary points with the best known sample complexity of \(\mathcal\{O\}(\epsilon^\{-2\})\) (up to a log factor), and NAC converges to an \(\epsilon+\epsilon_\{\text\{critic\}\}+\sqrt\{\epsilon_\{\text\{actor\}\}\}\) neighborhood of the global optimum with the best known sample complexity of \(\mathcal\{O\}(\epsilon^\{-3\})\), where \(\epsilon_\{\text\{critic\}\}\) is the approximation error of the critic and \

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2024non

Related papers

• Non-asymptotic Convergence Analysis Of Two Time-scale (natural) Actor-critic Algorithms (2020)0.00
• Improving Sample Complexity Bounds For (natural) Actor-critic Algorithms (2020)0.00
• Finite-sample Analysis Of Off-policy Natural Actor-critic With Linear Function Approximation (2021)0.00
• A Finite Time Analysis Of Two Time-scale Actor Critic Methods (2020)0.00
• Decision-aware Actor-critic With Function Approximation And Theoretical Guarantees (2023)0.00
• Single-timescale Actor-critic Provably Finds Globally Optimal Policy (2020)0.00
• Finite Sample Analysis Of Two-time-scale Natural Actor-critic Algorithm (2021)7.50
• Finite-time Analysis Of Fully Decentralized Single-timescale Actor-critic (2022)0.00