How Does An Approximate Model Help In Reinforcement Learning?

Abstract

One of the key approaches to save samples in reinforcement learning (RL) is to use knowledge from an approximate model such as its simulator. However, how much does an approximate model help to learn a near-optimal policy of the true unknown model? Despite numerous empirical studies of transfer reinforcement learning, an answer to this question is still elusive. In this paper, we study the sample complexity of RL while an approximate model of the environment is provided. For an unknown Markov decision process (MDP), we show that the approximate model can effectively reduce the complexity by eliminating sub-optimal actions from the policy searching space. In particular, we provide an algorithm that uses \(\widetilde\{O\}(N/(1-\gamma)^3/\epsilon^2)\) samples in a generative model to learn an \(\epsilon\)-optimal policy, where \(\gamma\) is the discount factor and \(N\) is the number of near-optimal actions in the approximate model. This can be much smaller than the learning-from-scratch

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfeng2019how

Related papers

• Breaking The Sample Size Barrier In Model-based Reinforcement Learning With A Generative Model (2020)9.03
• Model-free Reinforcement Learning: From Clipped Pseudo-regret To Sample Complexity (2020)0.00
• Sample-efficient Reinforcement Learning For Linearly-parameterized Mdps With A Generative Model (2021)0.00
• Deciding What To Model: Value-equivalent Sampling For Reinforcement Learning (2022)0.00
• On The Model-misspecification In Reinforcement Learning (2023)0.00
• Sample Complexity Of Offline Reinforcement Learning With Deep Relu Networks (2021)0.00
• Online Model Selection For Reinforcement Learning With Function Approximation (2020)0.00
• Sample Complexity Of Reinforcement Learning Using Linearly Combined Model Ensembles (2019)0.00