Learning Multimodal Transition Dynamics For Model-based Reinforcement Learning

Abstract

In this paper we study how to learn stochastic, multimodal transition dynamics in reinforcement learning (RL) tasks. We focus on evaluating transition function estimation, while we defer planning over this model to future work. Stochasticity is a fundamental property of many task environments. However, discriminative function approximators have difficulty estimating multimodal stochasticity. In contrast, deep generative models do capture complex high-dimensional outcome distributions. First we discuss why, amongst such models, conditional variational inference (VI) is theoretically most appealing for model-based RL. Subsequently, we compare different VI models on their ability to learn complex stochasticity on simulated functions, as well as on a typical RL gridworld with multimodal dynamics. Results show VI successfully predicts multimodal outcomes, but also robustly ignores these for deterministic parts of the transition dynamics. In summary, we show a robust method to learn multimod

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymoerland2017learning

Related papers

• Variational Inference For Model-free And Model-based Reinforcement Learning (2022)0.00
• Model-based Reinforcement Learning With Multinomial Logistic Function Approximation (2022)2.26
• Learning To Reweight Imaginary Transitions For Model-based Reinforcement Learning (2021)0.00
• Towards A Simple Approach To Multi-step Model-based Reinforcement Learning (2018)0.00
• Trajectory-wise Multiple Choice Learning For Dynamics Generalization In Reinforcement Learning (2020)6.77
• Demystifying Reinforcement Learning In Time-varying Systems (2022)0.00
• Live In The Moment: Learning Dynamics Model Adapted To Evolving Policy (2022)0.00
• Distributionally Robust Model-based Reinforcement Learning With Large State Spaces (2023)0.00