Learning In Volatile Environments With The Bayes Factor Surprise

Abstract

Surprise-based learning allows agents to rapidly adapt to non-stationary stochastic environments characterized by sudden changes. We show that exact Bayesian inference in a hierarchical model gives rise to a surprise-modulated trade-off between forgetting old observations and integrating them with the new ones. The modulation depends on a probability ratio, which we call "Bayes Factor Surprise", that tests the prior belief against the current belief. We demonstrate that in several existing approximate algorithms the Bayes Factor Surprise modulates the rate of adaptation to new observations. We derive three novel surprised-based algorithms, one in the family of particle filters, one in the family of variational learning, and the other in the family of message passing, that have constant scaling in observation sequence length and particularly simple update dynamics for any distribution in the exponential family. Empirical results show that these surprise-based algorithms estimate paramet

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliakoni2019learning

Related papers

• Curiosity-driven Exploration Via Latent Bayesian Surprise (2021)0.00
• Beyond Surprise: Improving Exploration Through Surprise Novelty (2023)0.00
• VASE: Variational Assorted Surprise Exploration For Reinforcement Learning (2019)5.84
• The Hierarchical Adaptive Forgetting Variational Filter (2018)0.00
• MIME: Mutual Information Minimisation Exploration (2020)0.00
• Surprise-adaptive Intrinsic Motivation For Unsupervised Reinforcement Learning (2024)0.00
• Deep Interactive Bayesian Reinforcement Learning Via Meta-learning (2021)5.24
• Meta-trained Agents Implement Bayes-optimal Agents (2020)0.00