Reinforcement Learning In The Wild With Maximum Likelihood-based Model Transfer

Abstract

In this paper, we study the problem of transferring the available Markov Decision Process (MDP) models to learn and plan efficiently in an unknown but similar MDP. We refer to it as \textit\{Model Transfer Reinforcement Learning (MTRL)\} problem. First, we formulate MTRL for discrete MDPs and Linear Quadratic Regulators (LQRs) with continuous state actions. Then, we propose a generic two-stage algorithm, MLEMTRL, to address the MTRL problem in discrete and continuous settings. In the first stage, MLEMTRL uses a \textit\{constrained Maximum Likelihood Estimation (MLE)\}-based approach to estimate the target MDP model using a set of known MDP models. In the second stage, using the estimated target MDP model, MLEMTRL deploys a model-based planning algorithm appropriate for the MDP class. Theoretically, we prove worst-case regret bounds for MLEMTRL both in realisable and non-realisable settings. We empirically demonstrate that MLEMTRL allows faster learning in new MDPs than learning from s

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyeriksson2023reinforcement

Related papers

• Provably Efficient Multi-task Reinforcement Learning With Model Transfer (2021)0.00
• Transfer Q-learning (2022)0.00
• Lipschitz Lifelong Reinforcement Learning (2020)8.35
• Model-based Reinforcement Learning With Multinomial Logistic Function Approximation (2022)2.26
• Value-biased Maximum Likelihood Estimation For Model-based Reinforcement Learning In Discounted Linear Mdps (2023)0.00
• TOM: Learning Policy-aware Models For Model-based Reinforcement Learning Via Transition Occupancy Matching (2023)0.00
• Provable Multi-task Reinforcement Learning: A Representation Learning Framework With Low Rank Rewards (2026)0.00
• Learning And Planning For Time-varying Mdps Using Maximum Likelihood Estimation (2019)0.00