Sample Complexity Of Reinforcement Learning Using Linearly Combined Model Ensembles

Abstract

Reinforcement learning (RL) methods have been shown to be capable of learning intelligent behavior in rich domains. However, this has largely been done in simulated domains without adequate focus on the process of building the simulator. In this paper, we consider a setting where we have access to an ensemble of pre-trained and possibly inaccurate simulators (models). We approximate the real environment using a state-dependent linear combination of the ensemble, where the coefficients are determined by the given state features and some unknown parameters. Our proposed algorithm provably learns a near-optimal policy with a sample complexity polynomial in the number of unknown parameters, and incurs no dependence on the size of the state (or action) space. As an extension, we also consider the more challenging problem of model selection, where the state features are unknown and can be chosen from a large candidate set. We provide exponential lower bounds that illustrate the fundamental h

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymodi2019sample

Related papers

• SEERL: Sample Efficient Ensemble Reinforcement Learning (2020)2.26
• Is Model Ensemble Necessary? Model-based RL Via A Single Model With Lipschitz Regularized Value Function (2023)0.00
• On The Sample Complexity Of Reinforcement Learning With Policy Space Generalization (2020)0.00
• Towards Applicable Reinforcement Learning: Improving The Generalization And Sample Efficiency With Policy Ensemble (2022)9.23
• How Does An Approximate Model Help In Reinforcement Learning? (2019)0.00
• Sample-efficient Reinforcement Learning For Linearly-parameterized Mdps With A Generative Model (2021)0.00
• Sample Complexity Of Offline Reinforcement Learning With Deep Relu Networks (2021)0.00
• Sample Efficient Reinforcement Learning Via Model-ensemble Exploration And Exploitation (2021)0.00