Variational Inference For Model-free And Model-based Reinforcement Learning

Abstract

Variational inference (VI) is a specific type of approximate Bayesian inference that approximates an intractable posterior distribution with a tractable one. VI casts the inference problem as an optimization problem, more specifically, the goal is to maximize a lower bound of the logarithm of the marginal likelihood with respect to the parameters of the approximate posterior. Reinforcement learning (RL) on the other hand deals with autonomous agents and how to make them act optimally such as to maximize some notion of expected future cumulative reward. In the non-sequential setting where agents' actions do not have an impact on future states of the environment, RL is covered by contextual bandits and Bayesian optimization. In a proper sequential scenario, however, where agents' actions affect future states, instantaneous rewards need to be carefully traded off against potential long-term rewards. This manuscript shows how the apparently different subjects of VI and RL are linked in two

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyleibfried2022variational

Related papers

• VIREL: A Variational Inference Framework For Reinforcement Learning (2018)0.00
• Learning Multimodal Transition Dynamics For Model-based Reinforcement Learning (2017)0.00
• Simple And Optimal Methods For Stochastic Variational Inequalities, II: Markovian Noise And Policy Evaluation In Reinforcement Learning (2020)8.60
• Variational Intrinsic Control Revisited (2020)0.00
• VIME: Variational Information Maximizing Exploration (2016)0.00
• Reinforcement Learning And Control As Probabilistic Inference: Tutorial And Review (2018)0.00
• Variational Empowerment As Representation Learning For Goal-based Reinforcement Learning (2021)0.00
• MAVRL: Learning Reward Functions From Multiple Feedback Types With Amortized Variational Inference (2026)0.00