F-divergence Constrained Policy Improvement

Abstract

To ensure stability of learning, state-of-the-art generalized policy iteration algorithms augment the policy improvement step with a trust region constraint bounding the information loss. The size of the trust region is commonly determined by the Kullback-Leibler (KL) divergence, which not only captures the notion of distance well but also yields closed-form solutions. In this paper, we consider a more general class of f-divergences and derive the corresponding policy update rules. The generic solution is expressed through the derivative of the convex conjugate function to f and includes the KL solution as a special case. Within the class of f-divergences, we further focus on a one-parameter family of \(\alpha\)-divergences to study effects of the choice of divergence on policy improvement. Previously known as well as new policy updates emerge for different values of \(\alpha\). We show that every type of policy update comes with a compatible policy evaluation resulting from the chosen

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybelousov2017f

Related papers

• \(f\)-policy Gradients: A General Framework For Goal Conditioned RL Using \(f\)-divergences (2023)0.00
• Entropic Regularization Of Markov Decision Processes (2019)6.77
• The \(f\)-divergence Reinforcement Learning Framework (2021)0.00
• An Analytical Update Rule For General Policy Optimization (2021)0.00
• Greedification Operators For Policy Optimization: Investigating Forward And Reverse KL Divergences (2021)0.00
• Divergence-augmented Policy Optimization (2025)0.00
• Generalized Munchausen Reinforcement Learning Using Tsallis KL Divergence (2023)0.00
• Easy Monotonic Policy Iteration (2016)0.00