Beyond Surprise: Improving Exploration Through Surprise Novelty

Abstract

We present a new computing model for intrinsic rewards in reinforcement learning that addresses the limitations of existing surprise-driven explorations. The reward is the novelty of the surprise rather than the surprise norm. We estimate the surprise novelty as retrieval errors of a memory network wherein the memory stores and reconstructs surprises. Our surprise memory (SM) augments the capability of surprise-based intrinsic motivators, maintaining the agent's interest in exciting exploration while reducing unwanted attraction to unpredictable or noisy observations. Our experiments demonstrate that the SM combined with various surprise predictors exhibits efficient exploring behaviors and significantly boosts the final performance in sparse reward environments, including Noisy-TV, navigation and challenging Atari games.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyle2023beyond

Related papers

• Curiosity-driven Exploration Via Latent Bayesian Surprise (2021)0.00
• Curiosity-driven Multi-agent Exploration With Mixed Objectives (2022)0.00
• Nuclear Norm Maximization Based Curiosity-driven Learning (2022)0.00
• VASE: Variational Assorted Surprise Exploration For Reinforcement Learning (2019)5.84
• Information Content Exploration (2023)0.00
• Self-supervised Exploration Via Temporal Inconsistency In Reinforcement Learning (2022)3.58
• Wonder Wins Ways: Curiosity-driven Exploration Through Multi-agent Contextual Calibration (2025)0.00
• Intrinsic Rewards From Self-organizing Feature Maps For Exploration In Reinforcement Learning (2023)0.00