When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence

Abstract

Generative adversarial imitation learning (GAIL) is a popular inverse reinforcement learning approach for jointly optimizing policy and reward from expert trajectories. A primary question about GAIL is whether applying a certain policy gradient algorithm to GAIL attains a global minimizer (i.e., yields the expert policy), for which existing understanding is very limited. Such global convergence has been shown only for the linear (or linear-type) MDP and linear (or linearizable) reward. In this paper, we study GAIL under general MDP and for nonlinear reward function classes (as long as the objective function is strongly concave with respect to the reward parameter). We characterize the global convergence with a sublinear rate for a broad range of commonly used policy gradient algorithms, all of which are implemented in an alternating manner with stochastic gradient ascent for reward update, including projected policy gradient (PPG)-GAIL, Frank-Wolfe policy gradient (FWPG)-GAIL, trust re

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyguan2020when

Related papers

• C-GAIL: Stabilizing Generative Adversarial Imitation Learning With Control Theory (2024)0.00
• Provably Efficient Generative Adversarial Imitation Learning For Online And Offline Setting With Linear Function Approximation (2021)0.00
• Non-adversarial Imitation Learning And Its Connections To Adversarial Methods (2020)0.00
• A Pragmatic Look At Deep Imitation Learning (2021)0.00
• Global Convergence Of Policy Gradient Methods In Reinforcement Learning, Games And Control (2023)0.00
• \(f\)-gail: Learning \(f\)-divergence For Generative Adversarial Imitation Learning (2020)0.00
• Augmenting GAIL With BC For Sample Efficient Imitation Learning (2020)0.00
• Last-iterate Global Convergence Of Policy Gradients For Constrained Reinforcement Learning (2024)0.00