Surprise-adaptive Intrinsic Motivation For Unsupervised Reinforcement Learning

Abstract

Both entropy-minimizing and entropy-maximizing (curiosity) objectives for unsupervised reinforcement learning (RL) have been shown to be effective in different environments, depending on the environment's level of natural entropy. However, neither method alone results in an agent that will consistently learn intelligent behavior across environments. In an effort to find a single entropy-based method that will encourage emergent behaviors in any environment, we propose an agent that can adapt its objective online, depending on the entropy conditions by framing the choice as a multi-armed bandit problem. We devise a novel intrinsic feedback signal for the bandit, which captures the agent's ability to control the entropy in its environment. We demonstrate that such agents can learn to control entropy and exhibit emergent behaviors in both high- and low-entropy regimes and can learn skillful behaviors in benchmark tasks. Videos of the trained agents and summarized findings can be found on

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhugessen2024surprise

Related papers

• Predictable Reinforcement Learning Dynamics Through Entropy Rate Minimization (2023)0.00
• Intrinsic Rewards For Exploration Without Harm From Observational Noise: A Simulation Study Based On The Free Energy Principle (2024)0.00
• R\'enyi State Entropy For Exploration Acceleration In Reinforcement Learning (2022)0.00
• Evaluating Agents Without Rewards (2020)0.00
• Fast Rates For Maximum Entropy Exploration (2023)0.00
• Behavioral Entropy-guided Dataset Generation For Offline Reinforcement Learning (2025)0.00
• Do You Need The Entropy Reward (in Practice)? (2022)0.00
• Maximum Entropy-regularized Multi-goal Reinforcement Learning (2019)0.00