Proximal Point Imitation Learning

Abstract

This work develops new algorithms with rigorous efficiency guarantees for infinite horizon imitation learning (IL) with linear function approximation without restrictive coherence assumptions. We begin with the minimax formulation of the problem and then outline how to leverage classical tools from optimization, in particular, the proximal-point method (PPM) and dual smoothing, for online and offline IL, respectively. Thanks to PPM, we avoid nested policy evaluation and cost updates for online IL appearing in the prior literature. In particular, we do away with the conventional alternating updates by the optimization of a single convex and smooth objective over both cost and Q-functions. When solved inexactly, we relate the optimization errors to the suboptimality of the recovered policy. As an added bonus, by re-interpreting PPM as dual smoothing with the expert policy as a center point, we also obtain an offline IL algorithm enjoying theoretical guarantees in terms of required expert

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyviano2022proximal

Related papers

• Imitation Learning In Discounted Linear Mdps Without Exploration Assumptions (2024)0.00
• Explaining Fast Improvement In Online Imitation Learning (2020)0.00
• Offline Imitation Learning By Controlling The Effective Planning Horizon (2024)0.00
• Mitigating Covariate Shift In Imitation Learning Via Offline Data Without Great Coverage (2021)0.00
• Provably Efficient Imitation Learning From Observation Alone (2019)0.00
• Minimax Optimal Online Imitation Learning Via Replay Estimation (2022)0.00
• Toward The Fundamental Limits Of Imitation Learning (2020)0.00
• Accelerating Imitation Learning With Predictive Models (2018)0.00