Deep Interactive Bayesian Reinforcement Learning Via Meta-learning

Abstract

Agents that interact with other agents often do not know a priori what the other agents' strategies are, but have to maximise their own online return while interacting with and learning about others. The optimal adaptive behaviour under uncertainty over the other agents' strategies w.r.t. some prior can in principle be computed using the Interactive Bayesian Reinforcement Learning framework. Unfortunately, doing so is intractable in most settings, and existing approximation methods are restricted to small tasks. To overcome this, we propose to meta-learn approximate belief inference and Bayes-optimal behaviour for a given prior. To model beliefs over other agents, we combine sequential and hierarchical Variational Auto-Encoders, and meta-train this inference model alongside the policy. We show empirically that our approach outperforms existing methods that use a model-free approach, sample from the approximate posterior, maintain memory-free models of others, or do not fully utilise th

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keyzintgraf2021deep

Related papers

• Varibad: A Very Good Method For Bayes-adaptive Deep RL Via Meta-learning (2019)0.00
• Meta-trained Agents Implement Bayes-optimal Agents (2020)0.00
• Offline Meta Learning Of Exploration (2020)0.00
• Varibased: Variational Bayes-adaptive Sequential Monte-carlo Planning For Deep Reinforcement Learning (2026)0.00
• Bayesian Action Decoder For Deep Multi-agent Reinforcement Learning (2018)0.00
• Meta-value Learning: A General Framework For Learning With Learning Awareness (2023)0.00
• Efficient Off-policy Meta-reinforcement Learning Via Probabilistic Context Variables (2019)0.00
• Meta-gradient Reinforcement Learning With An Objective Discovered Online (2020)0.00