The \(f\)-divergence Reinforcement Learning Framework

Abstract

The framework of deep reinforcement learning (DRL) provides a powerful and widely applicable mathematical formalization for sequential decision-making. This paper present a novel DRL framework, termed *\(f\)-Divergence Reinforcement Learning (FRL)*. In FRL, the policy evaluation and policy improvement phases are simultaneously performed by minimizing the \(f\)-divergence between the learning policy and sampling policy, which is distinct from conventional DRL algorithms that aim to maximize the expected cumulative rewards. We theoretically prove that minimizing such \(f\)-divergence can make the learning policy converge to the optimal policy. Besides, we convert the process of training agents in FRL framework to a saddle-point optimization problem with a specific \(f\) function through Fenchel conjugate, which forms new methods for policy evaluation and policy improvement. Through mathematical proofs and empirical evaluation, we demonstrate that the FRL framework has two advantages: (1)

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygong2021the

Related papers

• \(f\)-policy Gradients: A General Framework For Goal Conditioned RL Using \(f\)-divergences (2023)0.00
• Global Convergence Guarantees For Federated Policy Gradient Methods With Adversaries (2024)0.00
• F-divergence Constrained Policy Improvement (2017)0.00
• A Unified Framework For Rethinking Policy Divergence Measures In GRPO (2026)0.00
• Fault-tolerant Federated Reinforcement Learning With Theoretical Guarantee (2021)0.00
• Diffpo: Training Diffusion Llms To Reason Fast And Furious Via Reinforcement Learning (2025)0.00
• Momentum For The Win: Collaborative Federated Reinforcement Learning Across Heterogeneous Environments (2024)0.00
• Vmfer: Von Mises-fisher Experience Resampling Based On Uncertainty Of Gradient Directions For Policy Improvement (2024)0.00