Greedification Operators For Policy Optimization: Investigating Forward And Reverse KL Divergences

Abstract

Approximate Policy Iteration (API) algorithms alternate between (approximate) policy evaluation and (approximate) greedification. Many different approaches have been explored for approximate policy evaluation, but less is understood about approximate greedification and what choices guarantee policy improvement. In this work, we investigate approximate greedification when reducing the KL divergence between the parameterized policy and the Boltzmann distribution over action values. In particular, we investigate the difference between the forward and reverse KL divergences, with varying degrees of entropy regularization. We show that the reverse KL has stronger policy improvement guarantees, but that reducing the forward KL can result in a worse policy. We also demonstrate, however, that a large enough reduction of the forward KL can induce improvement under additional assumptions. Empirically, we show on simple continuous-action environments that the forward KL can induce more exploratio

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychan2021greedification

Related papers

• Understanding The Pathologies Of Approximate Policy Evaluation When Combined With Greedification In Reinforcement Learning (2020)0.00
• Beyond KL Divergence: Policy Optimization With Flexible Bregman Divergences For LLM Reasoning (2026)0.00
• Generalized Munchausen Reinforcement Learning Using Tsallis KL Divergence (2023)0.00
• Dual Policy Iteration (2018)0.00
• Dichotomous Diffusion Policy Optimization (2025)0.00
• On The Theory Of Policy Gradient Methods: Optimality, Approximation, And Distribution Shift (2019)0.00
• F-divergence Constrained Policy Improvement (2017)0.00
• Divergence-augmented Policy Optimization (2025)0.00