Meta Reinforcement Learning With Distribution Of Exploration Parameters Learned By Evolution Strategies

Abstract

In this paper, we propose a novel meta-learning method in a reinforcement learning setting, based on evolution strategies (ES), exploration in parameter space and deterministic policy gradients. ES methods are easy to parallelize, which is desirable for modern training architectures; however, such methods typically require a huge number of samples for effective training. We use deterministic policy gradients during adaptation and other techniques to compensate for the sample-efficiency problem while maintaining the inherent scalability of ES methods. We demonstrate that our method achieves good results compared to gradient-based meta-learning in high-dimensional control tasks in the MuJoCo simulator. In addition, because of gradient-free methods in the meta-training phase, which do not need information about gradients and policies in adaptation training, we predict and confirm our algorithm performs better in tasks that need multi-step adaptation.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyshen2018meta

Related papers

• Learning To Explore With Meta-policy Gradient (2018)0.00
• Accelerating Reinforcement Learning With A Directional-gaussian-smoothing Evolution Strategy (2020)6.77
• Learning Efficient And Effective Exploration Policies With Counterfactual Meta Policy (2019)0.00
• An Efficient Asynchronous Method For Integrating Evolutionary And Gradient-based Policy Search (2020)0.00
• Variance Reduction For Evolution Strategies Via Structured Control Variates (2019)0.00
• Improving Exploration In Evolution Strategies For Deep Reinforcement Learning Via A Population Of Novelty-seeking Agents (2017)0.00
• Distributionally Adaptive Meta Reinforcement Learning (2022)2.26
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00