Explaining Fast Improvement In Online Imitation Learning

Abstract

Online imitation learning (IL) is an algorithmic framework that leverages interactions with expert policies for efficient policy optimization. Here policies are optimized by performing online learning on a sequence of loss functions that encourage the learner to mimic expert actions, and if the online learning has no regret, the agent can provably learn an expert-like policy. Online IL has demonstrated empirical successes in many applications and interestingly, its policy improvement speed observed in practice is usually much faster than existing theory suggests. In this work, we provide an explanation of this phenomenon. Let \(\xi\) denote the policy class bias and assume the online IL loss functions are convex, smooth, and non-negative. We prove that, after \(N\) rounds of online IL with stochastic feedback, the policy improves in \(\tilde\{O\}(1/N + \sqrt\{\xi/N\})\) in both expectation and high probability. In other words, we show that adopting a sufficiently expressive policy clas

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyan2020explaining

Related papers

• Accelerating Imitation Learning With Predictive Models (2018)0.00
• Proximal Point Imitation Learning (2022)0.00
• Minimax Optimal Online Imitation Learning Via Replay Estimation (2022)0.00
• Mitigating Covariate Shift In Imitation Learning Via Offline Data Without Great Coverage (2021)0.00
• Residual Q-learning: Offline And Online Policy Customization Without Value (2023)4.52
• Online Adaptation For Enhancing Imitation Learning Policies (2024)0.00
• Toward The Fundamental Limits Of Imitation Learning (2020)0.00
• Offline Imitation Learning By Controlling The Effective Planning Horizon (2024)0.00