Model-based Reinforcement Learning With Multinomial Logistic Function Approximation

Abstract

We study model-based reinforcement learning (RL) for episodic Markov decision processes (MDP) whose transition probability is parametrized by an unknown transition core with features of state and action. Despite much recent progress in analyzing algorithms in the linear MDP setting, the understanding of more general transition models is very restrictive. In this paper, we establish a provably efficient RL algorithm for the MDP whose state transition is given by a multinomial logistic model. To balance the exploration-exploitation trade-off, we propose an upper confidence bound-based algorithm. We show that our proposed algorithm achieves \(\tilde\{O\}(d \sqrt\{H^3 T\})\) regret bound where \(d\) is the dimension of the transition core, \(H\) is the horizon, and \(T\) is the total number of steps. To the best of our knowledge, this is the first model-based RL algorithm with multinomial logistic function approximation with provable guarantees. We also comprehensively evaluate our propose

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyhwang2022model

Related papers

• Randomized Exploration For Reinforcement Learning With Multinomial Logistic Function Approximation (2024)0.00
• Nearly Minimax Optimal Reinforcement Learning For Linear Markov Decision Processes (2022)0.00
• Provably Efficient Reinforcement Learning With Multinomial Logit Function Approximation (2024)0.00
• Reward-free Model-based Reinforcement Learning With Linear Function Approximation (2021)0.00
• Online Reinforcement Learning In Markov Decision Process Using Linear Programming (2023)3.58
• Nonstationary Reinforcement Learning With Linear Function Approximation (2020)0.00
• Prior-dependent Analysis Of Posterior Sampling Reinforcement Learning With Function Approximation (2024)0.00
• Value-biased Maximum Likelihood Estimation For Model-based Reinforcement Learning In Discounted Linear Mdps (2023)0.00