Active Exploration In Markov Decision Processes

Abstract

We introduce the active exploration problem in Markov decision processes (MDPs). Each state of the MDP is characterized by a random value and the learner should gather samples to estimate the mean value of each state as accurately as possible. Similarly to active exploration in multi-armed bandit (MAB), states may have different levels of noise, so that the higher the noise, the more samples are needed. As the noise level is initially unknown, we need to trade off the exploration of the environment to estimate the noise and the exploitation of these estimates to compute a policy maximizing the accuracy of the mean predictions. We introduce a novel learning algorithm to solve this problem showing that active exploration in MDPs may be significantly more difficult than in MAB. We also derive a heuristic procedure to mitigate the negative effect of slowly mixing policies. Finally, we validate our findings on simple numerical simulations.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytarbouriech2019active

Related papers

• Model-based Exploration In Monitored Markov Decision Processes (2025)0.00
• Efficient Reinforcement Learning Via Initial Pure Exploration (2017)0.00
• Improved Bounds For Reward-agnostic And Reward-free Exploration (2026)0.00
• Provably Efficient Maximum Entropy Exploration (2018)0.00
• Conservative Exploration In Reinforcement Learning (2020)0.00
• Model-based Active Exploration (2018)0.00
• Learn A Flexible Exploration Model For Parameterized Action Markov Decision Processes (2025)0.00
• Exploration Conscious Reinforcement Learning Revisited (2018)0.00