Efficient Exploration In Continuous-time Model-based Reinforcement Learning

Abstract

Reinforcement learning algorithms typically consider discrete-time dynamics, even though the underlying systems are often continuous in time. In this paper, we introduce a model-based reinforcement learning algorithm that represents continuous-time dynamics using nonlinear ordinary differential equations (ODEs). We capture epistemic uncertainty using well-calibrated probabilistic models, and use the optimistic principle for exploration. Our regret bounds surface the importance of the measurement selection strategy(MSS), since in continuous time we not only must decide how to explore, but also when to observe the underlying system. Our analysis demonstrates that the regret is sublinear when modeling ODEs with Gaussian Processes (GP) for common choices of MSS, such as equidistant sampling. Additionally, we propose an adaptive, data-dependent, practical MSS that, when combined with GP dynamics, also achieves sublinear regret with significantly fewer samples. We showcase the benefits of co

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytreven2023efficient

Related papers

• Model-based Reinforcement Learning For Semi-markov Decision Processes With Neural Odes (2020)0.00
• Optimistic Active Exploration Of Dynamical Systems (2023)0.00
• Model-based Reinforcement Learning For Control Under Time-varying Dynamics (2026)0.00
• Distributionally Robust Model-based Reinforcement Learning With Large State Spaces (2023)0.00
• Model-based Exploration In Monitored Markov Decision Processes (2025)0.00
• Model-based Reinforcement Learning For Continuous Control With Posterior Sampling (2020)0.00
• Efficient Model-based Reinforcement Learning Through Optimistic Policy Search And Planning (2020)0.00
• Exploration Versus Exploitation In Reinforcement Learning: A Stochastic Control Approach (2018)9.76