Learning To Score Behaviors For Guided Policy Optimization

Abstract

We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space. We show that by utilizing the dual formulation of the WD, we can learn score functions over policy behaviors that can in turn be used to lead policy optimization towards (or away from) (un)desired behaviors. Combined with smoothed WDs, the dual formulation allows us to devise efficient algorithms that take stochastic gradient descent steps through WD regularizers. We incorporate these regularizers into two novel on-policy algorithms, Behavior-Guided Policy Gradient and Behavior-Guided Evolution Strategies, which we demonstrate can outperform existing methods in a variety of challenging environments. We also provide an open source demo.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypacchiano2019learning

Related papers

• Efficient Wasserstein Natural Gradients For Reinforcement Learning (2020)0.00
• Reinforcement Learning With Wasserstein Distance Regularisation, With Applications To Multipolicy Learning (2018)0.00
• Wd1: Weighted Policy Optimization For Reasoning In Diffusion Language Models (2025)0.00
• Diversity-inducing Policy Gradient: Using Maximum Mean Discrepancy To Find A Set Of Diverse Policies (2019)8.35
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00
• Improved Exploration Through Latent Trajectory Optimization In Deep Deterministic Policy Gradient (2019)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Relative Entropy Pathwise Policy Optimization (2025)0.00