Entropy Regularized Reinforcement Learning Using Large Deviation Theory

Abstract

Reinforcement learning (RL) is an important field of research in machine learning that is increasingly being applied to complex optimization problems in physics. In parallel, concepts from physics have contributed to important advances in RL with developments such as entropy-regularized RL. While these developments have led to advances in both fields, obtaining analytical solutions for optimization in entropy-regularized RL is currently an open problem. In this paper, we establish a mapping between entropy-regularized RL and research in non-equilibrium statistical mechanics focusing on Markovian processes conditioned on rare events. In the long-time limit, we apply approaches from large deviation theory to derive exact analytical results for the optimal policy and optimal dynamics in Markov Decision Process (MDP) models of reinforcement learning. The results obtained lead to a novel analytical and computational framework for entropy-regularized RL which is validated by simulations. The

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keyarriojas2021entropy

Related papers

• Optimal Scheduling Of Entropy Regulariser For Continuous-time Linear-quadratic Reinforcement Learning (2022)4.52
• A Comparative Theoretical Analysis Of Entropy Control Methods In Reinforcement Learning (2026)0.00
• A Regularized Approach To Sparse Optimal Policy In Reinforcement Learning (2019)0.00
• Mutual-information Regularization In Markov Decision Processes And Actor-critic Learning (2019)0.00
• Entropy Regularization For Mean Field Games With Learning (2020)0.00
• Regularization Matters In Policy Optimization (2019)2.68
• Marginalized State Distribution Entropy Regularization In Policy Optimization (2019)0.00
• Maximum Entropy RL (provably) Solves Some Robust RL Problems (2021)0.00