Distributionally Robust Model-based Reinforcement Learning With Large State Spaces

Abstract

Three major challenges in reinforcement learning are the complex dynamical systems with large state spaces, the costly data acquisition processes, and the deviation of real-world dynamics from the training environment deployment. To overcome these issues, we study distributionally robust Markov decision processes with continuous state spaces under the widely used Kullback-Leibler, chi-square, and total variation uncertainty sets. We propose a model-based approach that utilizes Gaussian Processes and the maximum variance reduction algorithm to efficiently learn multi-output nominal transition dynamics, leveraging access to a generative model (i.e., simulator). We further demonstrate the statistical sample complexity of the proposed method for different uncertainty sets. These complexity bounds are independent of the number of states and extend beyond linear dynamics, ensuring the effectiveness of our approach in identifying near-optimal distributionally-robust policies. The proposed met

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyramesh2023distributionally

Related papers

• The Curious Price Of Distributional Robustness In Reinforcement Learning With A Generative Model (2023)0.00
• Sample Complexity Of Robust Reinforcement Learning With A Generative Model (2021)0.00
• Distributional Method For Risk Averse Reinforcement Learning (2023)0.00
• Sample-efficient Robust Multi-agent Reinforcement Learning In The Face Of Environmental Uncertainty (2024)0.00
• Distributionally Robust Model-based Offline Reinforcement Learning With Near-optimal Sample Complexity (2022)0.00
• Distributionally Robust Online Markov Game With Linear Function Approximation (2025)0.00
• Improving Robustness Via Risk Averse Distributional Reinforcement Learning (2020)0.00
• Sample Complexity Of Variance-reduced Distributionally Robust Q-learning (2023)0.00