Entropic Regularization Of Markov Decision Processes

Abstract

An optimal feedback controller for a given Markov decision process (MDP) can in principle be synthesized by value or policy iteration. However, if the system dynamics and the reward function are unknown, a learning agent must discover an optimal controller via direct interaction with the environment. Such interactive data gathering commonly leads to divergence towards dangerous or uninformative regions of the state space unless additional regularization measures are taken. Prior works proposed bounding the information loss measured by the Kullback-Leibler (KL) divergence at every policy improvement step to eliminate instability in the learning dynamics. In this paper, we consider a broader family of \(f\)-divergences, and more concretely \(\alpha\)-divergences, which inherit the beneficial property of providing the policy improvement step in closed form at the same time yielding a corresponding dual objective for policy evaluation. Such entropic proximal policy optimization view gives

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keybelousov2019entropic

Related papers

• A Regularized Approach To Sparse Optimal Policy In Reinforcement Learning (2019)0.00
• Solving Non-rectangular Reward-robust Mdps Via Frequency Regularization (2023)0.00
• Mutual-information Regularization In Markov Decision Processes And Actor-critic Learning (2019)0.00
• F-divergence Constrained Policy Improvement (2017)0.00
• Narrowing The Gap Between Adversarial And Stochastic Mdps Via Policy Optimization (2024)0.00
• Efficient Learning For Entropy-regularized Markov Decision Processes Via Multilevel Monte Carlo (2025)0.00
• Temporal Regularization In Markov Decision Process (2018)0.00
• Parameterized Mdps And Reinforcement Learning Problems -- A Maximum Entropy Principle Based Framework (2020)8.60