A Model-based Approach For Improving Reinforcement Learning Efficiency Leveraging Expert Observations

Abstract

This paper investigates how to incorporate expert observations (without explicit information on expert actions) into a deep reinforcement learning setting to improve sample efficiency. First, we formulate an augmented policy loss combining a maximum entropy reinforcement learning objective with a behavioral cloning loss that leverages a forward dynamics model. Then, we propose an algorithm that automatically adjusts the weights of each component in the augmented loss function. Experiments on a variety of continuous control tasks demonstrate that the proposed algorithm outperforms various benchmarks by effectively utilizing available expert observations.

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keyozcan2024a

Related papers

• Bayesian Residual Policy Optimization: Scalable Bayesian Reinforcement Learning With Clairvoyant Experts (2020)0.00
• Sample Efficient Reinforcement Learning Via Model-ensemble Exploration And Exploitation (2021)0.00
• Continuous Action Reinforcement Learning From A Mixture Of Interpretable Experts (2020)0.00
• Learning Off-policy With Model-based Intrinsic Motivation For Active Online Exploration (2024)0.00
• The Pitfalls Of Imitation Learning When Actions Are Continuous (2025)0.00
• Simplifying Model-based RL: Learning Representations, Latent-space Models, And Policies With One Objective (2022)0.00
• Beyond-expert Performance With Limited Demonstrations: Efficient Imitation Learning With Double Exploration (2025)0.00
• Online Matching Via Reinforcement Learning: An Expert Policy Orchestration Strategy (2025)0.00