Enhancing Online Reinforcement Learning With Meta-learned Objective From Offline Data

Abstract

A major challenge in Reinforcement Learning (RL) is the difficulty of learning an optimal policy from sparse rewards. Prior works enhance online RL with conventional Imitation Learning (IL) via a handcrafted auxiliary objective, at the cost of restricting the RL policy to be sub-optimal when the offline data is generated by a non-expert policy. Instead, to better leverage valuable information in offline data, we develop Generalized Imitation Learning from Demonstration (GILD), which meta-learns an objective that distills knowledge from offline data and instills intrinsic motivation towards the optimal policy. Distinct from prior works that are exclusive to a specific RL algorithm, GILD is a flexible module intended for diverse vanilla off-policy RL algorithms. In addition, GILD introduces no domain-specific hyperparameter and minimal increase in computational cost. In four challenging MuJoCo tasks with sparse rewards, we show that three RL algorithms enhanced with GILD significantly ou

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydeng2025enhancing

Related papers

• Meta-gradient Reinforcement Learning With An Objective Discovered Online (2020)0.00
• Improving Generalization In Meta Reinforcement Learning Using Learned Objectives (2019)0.00
• Offline Meta-reinforcement Learning With Online Self-supervision (2021)0.00
• Using Offline Data To Speed Up Reinforcement Learning In Procedurally Generated Environments (2023)6.77
• Reinforcement Learning With Sparse Rewards Using Guidance From Offline Demonstration (2022)0.00
• Optimality Inductive Biases And Agnostic Guidelines For Offline Reinforcement Learning (2021)0.00
• Hundreds Guide Millions: Adaptive Offline Reinforcement Learning With Expert Guidance (2023)7.50
• Morel : Model-based Offline Reinforcement Learning (2020)0.00