Variational Empowerment As Representation Learning For Goal-based Reinforcement Learning

Abstract

Learning to reach goal states and learning diverse skills through mutual information (MI) maximization have been proposed as principled frameworks for self-supervised reinforcement learning, allowing agents to acquire broadly applicable multitask policies with minimal reward engineering. Starting from a simple observation that the standard goal-conditioned RL (GCRL) is encapsulated by the optimization objective of variational empowerment, we discuss how GCRL and MI-based RL can be generalized into a single family of methods, which we name variational GCRL (VGCRL), interpreting variational MI maximization, or variational empowerment, as representation learning methods that acquire functionally-aware state representations for goal reaching. This novel perspective allows us to: (1) derive simple but unexplored variants of GCRL to study how adding small representation capacity can already expand its capabilities; (2) investigate how discriminator function capacity and smoothness determine

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychoi2021variational

Related papers

• VIREL: A Variational Inference Framework For Reinforcement Learning (2018)0.00
• Variational Policy Gradient Method For Reinforcement Learning With General Utilities (2020)0.00
• A Unified Bellman Optimality Principle Combining Reward Maximization And Empowerment (2019)0.00
• Unsupervised Real-time Control Through Variational Empowerment (2017)3.58
• Stein Variational Goal Generation For Adaptive Exploration In Multi-goal Reinforcement Learning (2022)0.00
• Maximum Entropy-regularized Multi-goal Reinforcement Learning (2019)0.00
• VIME: Variational Information Maximizing Exploration (2016)0.00
• Learning With Amigo: Adversarially Motivated Intrinsic Goals (2020)0.00