An Analysis Of Model-based Reinforcement Learning From Abstracted Observations

Abstract

Many methods for Model-based Reinforcement learning (MBRL) in Markov decision processes (MDPs) provide guarantees for both the accuracy of the model they can deliver and the learning efficiency. At the same time, state abstraction techniques allow for a reduction of the size of an MDP while maintaining a bounded loss with respect to the original problem. Therefore, it may come as a surprise that no such guarantees are available when combining both techniques, i.e., where MBRL merely observes abstract states. Our theoretical analysis shows that abstraction can introduce a dependence between samples collected online (e.g., in the real world). That means that, without taking this dependence into account, results for MBRL do not directly extend to this setting. Our result shows that we can use concentration inequalities for martingales to overcome this problem. This result makes it possible to extend the guarantees of existing MBRL algorithms to the setting with abstraction. We illustrate

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keystarre2022an

Related papers

• Learning Markov State Abstractions For Deep Reinforcement Learning (2021)0.00
• Model-invariant State Abstractions For Model-based Reinforcement Learning (2021)0.00
• When To Update Your Model: Constrained Model-based Reinforcement Learning (2022)2.26
• Model-based Exploration In Monitored Markov Decision Processes (2025)0.00
• Self-correcting Models For Model-based Reinforcement Learning (2016)0.00
• Plan To Predict: Learning An Uncertainty-foreseeing Model For Model-based Reinforcement Learning (2023)0.00
• Algorithmic Framework For Model-based Deep Reinforcement Learning With Theoretical Guarantees (2018)0.00
• Planning With Exploration: Addressing Dynamics Bottleneck In Model-based Reinforcement Learning (2020)0.00