A Theoretical Connection Between Statistical Physics And Reinforcement Learning

Abstract

Sequential decision making in the presence of uncertainty and stochastic dynamics gives rise to distributions over state/action trajectories in reinforcement learning (RL) and optimal control problems. This observation has led to a variety of connections between RL and inference in probabilistic graphical models (PGMs). Here we explore a different dimension to this relationship, examining reinforcement learning using the tools and abstractions of statistical physics. The central object in the statistical physics abstraction is the idea of a partition function \(\mathcal\{Z\}\), and here we construct a partition function from the ensemble of possible trajectories that an agent might take in a Markov decision process. Although value functions and \(Q\)-functions can be derived from this partition function and interpreted via average energies, the \(\mathcal\{Z\}\)-function provides an object with its own Bellman equation that can form the basis of alternative dynamic programming approach

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyrahme2019a

Related papers

• Physics-informed RL For Maximal Safety Probability Estimation (2024)5.24
• Entropy Regularized Reinforcement Learning Using Large Deviation Theory (2021)6.34
• Reinforcement Learning And Control As Probabilistic Inference: Tutorial And Review (2018)0.00
• Correlation Priors For Reinforcement Learning (2019)0.00
• Dynamic Mean Field Programming (2022)0.00
• Distributional Hamilton-jacobi-bellman Equations For Continuous-time Reinforcement Learning (2022)0.00
• Parameterized Mdps And Reinforcement Learning Problems -- A Maximum Entropy Principle Based Framework (2020)8.60
• Computationally Efficient RL Under Linear Bellman Completeness For Deterministic Dynamics (2024)0.00