VIREL: A Variational Inference Framework For Reinforcement Learning

Abstract

Applying probabilistic models to reinforcement learning (RL) enables the application of powerful optimisation tools such as variational inference to RL. However, existing inference frameworks and their algorithms pose significant challenges for learning optimal policies, e.g., the absence of mode capturing behaviour in pseudo-likelihood methods and difficulties learning deterministic policies in maximum entropy RL based approaches. We propose VIREL, a novel, theoretically grounded probabilistic inference framework for RL that utilises a parametrised action-value function to summarise future dynamics of the underlying MDP. This gives VIREL a mode-seeking form of KL divergence, the ability to learn deterministic optimal polices naturally from inference and the ability to optimise value functions and policies in separate, iterative steps. In applying variational expectation-maximisation to VIREL we thus show that the actor-critic algorithm can be reduced to expectation-maximisation, with

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfellows2018virel

Related papers

• Variational Inference For Model-free And Model-based Reinforcement Learning (2022)0.00
• VIME: Variational Information Maximizing Exploration (2016)0.00
• Efficient And Robust Reinforcement Learning With Uncertainty-based Value Expansion (2019)0.00
• Variational Empowerment As Representation Learning For Goal-based Reinforcement Learning (2021)0.00
• UVIP: Model-free Approach To Evaluate Reinforcement Learning Algorithms (2021)0.00
• The Value-improvement Path: Towards Better Representations For Reinforcement Learning (2020)6.77
• Policy Improvement Reinforcement Learning (2026)0.00
• Efficient Inference For Inverse Reinforcement Learning And Dynamic Discrete Choice Models (2025)0.00