Variational Intrinsic Control Revisited

Abstract

In this paper, we revisit variational intrinsic control (VIC), an unsupervised reinforcement learning method for finding the largest set of intrinsic options available to an agent. In the original work by Gregor et al. (2016), two VIC algorithms were proposed: one that represents the options explicitly, and the other that does it implicitly. We show that the intrinsic reward used in the latter is subject to bias in stochastic environments, causing convergence to suboptimal solutions. To correct this behavior and achieve the maximal empowerment, we propose two methods respectively based on the transitional probability model and Gaussian mixture model. We substantiate our claims through rigorous mathematical derivations and experimental analyses.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykwon2020variational

Related papers

• Relative Variational Intrinsic Control (2020)0.00
• Variational Inference For Model-free And Model-based Reinforcement Learning (2022)0.00
• Generative Intrinsic Optimization: Intrinsic Control With Model Learning (2023)0.00
• Simple And Optimal Methods For Stochastic Variational Inequalities, II: Markovian Noise And Policy Evaluation In Reinforcement Learning (2020)8.60
• VIME: Variational Information Maximizing Exploration (2016)0.00
• Never Explore Repeatedly In Multi-agent Reinforcement Learning (2023)0.00
• VIREL: A Variational Inference Framework For Reinforcement Learning (2018)0.00
• Deep Intrinsically Motivated Exploration In Continuous Control (2022)0.00