In-trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminates

Abstract

Inverse reinforcement learning (IRL) aims to learn a reward function and a corresponding policy that best fit the demonstrated trajectories of an expert. However, current IRL works cannot learn incrementally from an ongoing trajectory because they have to wait to collect at least one complete trajectory to learn. To bridge the gap, this paper considers the problem of learning a reward function and a corresponding policy while observing the initial state-action pair of an ongoing trajectory and keeping updating the learned reward and policy when new state-action pairs of the ongoing trajectory are observed. We formulate this problem as an online bi-level optimization problem where the upper level dynamically adjusts the learned reward according to the newly observed state-action pairs with the help of a meta-regularization term, and the lower level learns the corresponding policy. We propose a novel algorithm to solve this problem and guarantee that the algorithm achieves sub-linear loc

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keyliu2024in

Related papers

• Maximum-likelihood Inverse Reinforcement Learning With Finite-time Guarantees (2022)0.00
• Inverse Reinforcement Learning With Simultaneous Estimation Of Rewards And Dynamics (2016)0.00
• Inverse Reinforcement Learning With Missing Data (2019)0.00
• On The Effective Horizon Of Inverse Reinforcement Learning (2023)0.00
• Inverse Reinforcement Learning Without Reinforcement Learning (2023)0.00
• Offline Inverse RL: New Solution Concepts And Provably Efficient Algorithms (2024)0.00
• Towards Theoretical Understanding Of Inverse Reinforcement Learning (2023)0.00
• Inverse Reinforcement Learning From Non-stationary Learning Agents (2024)0.00