Direct Policy Gradients: Direct Optimization Of Policies In Discrete Action Spaces

Abstract

Direct optimization is an appealing framework that replaces integration with optimization of a random objective for approximating gradients in models with discrete random variables. A\(^\star\) sampling is a framework for optimizing such random objectives over large spaces. We show how to combine these techniques to yield a reinforcement learning algorithm that approximates a policy gradient by finding trajectories that optimize a random objective. We call the resulting algorithms "direct policy gradient" (DirPG) algorithms. A main benefit of DirPG algorithms is that they allow the insertion of domain knowledge in the form of upper bounds on return-to-go at training time, like is used in heuristic search, while still directly computing a policy gradient. We further analyze their properties, showing there are cases where DirPG has an exponentially larger probability of sampling informative gradients compared to REINFORCE. We also show that there is a built-in variance reduction techniqu

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylorberbom2019direct

Related papers

• PC-PG: Policy Cover Directed Exploration For Provable Policy Gradient Learning (2020)0.00
• Policy Gradient Algorithms Implicitly Optimize By Continuation (2023)0.00
• Optimistic Natural Policy Gradient: A Simple Efficient Policy Optimization Framework For Online RL (2023)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Proximal Policy Optimization Algorithms (2017)0.00
• Marginal Policy Gradients: A Unified Family Of Estimators For Bounded Action Spaces With Applications (2018)0.00
• Zeroth-order Deterministic Policy Gradient (2020)0.00
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00