On-policy Robot Imitation Learning From A Converging Supervisor

Abstract

Existing on-policy imitation learning algorithms, such as DAgger, assume access to a fixed supervisor. However, there are many settings where the supervisor may evolve during policy learning, such as a human performing a novel task or an improving algorithmic controller. We formalize imitation learning from a "converging supervisor" and provide sublinear static and dynamic regret guarantees against the best policy in hindsight with labels from the converged supervisor, even when labels during learning are only from intermediate supervisors. We then show that this framework is closely connected to a class of reinforcement learning (RL) algorithms known as dual policy iteration (DPI), which alternate between training a reactive learner with imitation learning and a model-based supervisor with data from the learner. Experiments suggest that when this framework is applied with the state-of-the-art deep model-based RL algorithm PETS as an improving supervisor, it outperforms deep RL baselin

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybalakrishna2019on

Related papers

• Learning To Reach Goals Via Iterated Supervised Learning (2019)0.00
• RLIF: Interactive Imitation Learning As Reinforcement Learning (2023)0.00
• Learning Self-imitating Diverse Policies (2018)0.00
• Blending Imitation And Reinforcement Learning For Robust Policy Improvement (2023)0.00
• Imitating Opponent To Win: Adversarial Policy Imitation Learning In Two-player Competitive Games (2022)0.00
• Policy Learning Using Weak Supervision (2020)0.00
• Dual RL: Unification And New Methods For Reinforcement And Imitation Learning (2023)0.00
• Dual Policy Iteration (2018)0.00