Learning To Explore With Meta-policy Gradient

Abstract

The performance of off-policy learning, including deep Q-learning and deep deterministic policy gradient (DDPG), critically depends on the choice of the exploration policy. Existing exploration methods are mostly based on adding noise to the on-going actor policy and can only explore *local* regions close to what the actor policy dictates. In this work, we develop a simple meta-policy gradient algorithm that allows us to adaptively learn the exploration policy in DDPG. Our algorithm allows us to train flexible exploration behaviors that are independent of the actor policy, yielding a *global exploration* that significantly speeds up the learning process. With an extensive study, we show that our method significantly improves the sample-efficiency of DDPG on a variety of reinforcement learning tasks.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyxu2018learning

Related papers

• Improved Exploration Through Latent Trajectory Optimization In Deep Deterministic Policy Gradient (2019)0.00
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00
• PC-PG: Policy Cover Directed Exploration For Provable Policy Gradient Learning (2020)0.00
• Behind The Myth Of Exploration In Policy Gradients (2024)0.00
• Exploring More When It Needs In Deep Reinforcement Learning (2021)0.00
• Diverse Exploration Via Conjugate Policies For Policy Gradient Methods (2019)4.52
• Meta Reinforcement Learning With Distribution Of Exploration Parameters Learned By Evolution Strategies (2018)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00