Solving Robust Mdps Through No-regret Dynamics

Abstract

Reinforcement Learning is a powerful framework for training agents to navigate different situations, but it is susceptible to changes in environmental dynamics. However, solving Markov Decision Processes that are robust to changes is difficult due to nonconvexity and size of action or state spaces. While most works have analyzed this problem by taking different assumptions on the problem, a general and efficient theoretical analysis is still missing. However, we generate a simple framework for improving robustness by solving a minimax iterative optimization problem where a policy player and an environmental dynamics player are playing against each other. Leveraging recent results in online nonconvex learning and techniques from improving policy gradient methods, we yield an algorithm that maximizes the robustness of the Value Function on the order of \(\mathcal\{O\}\left(\frac\{1\}\{T^\{\frac\{1\}\{2\}\}\}\right)\) where \(T\) is the number of iterations of the algorithm.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyguha2023solving

Related papers

• Efficient Policy Optimization In Robust Constrained Mdps With Iteration Complexity Guarantees (2025)0.00
• Bring Your Own (non-robust) Algorithm To Solve Robust Mdps By Estimating The Worst Kernel (2023)0.00
• Solving Non-rectangular Reward-robust Mdps Via Frequency Regularization (2023)0.00
• Sample Complexity Of Robust Reinforcement Learning With A Generative Model (2021)0.00
• Robust Reinforcement Learning Using Least Squares Policy Iteration With Provable Performance Guarantees (2020)0.00
• Robust Lagrangian And Adversarial Policy Gradient For Robust Constrained Markov Decision Processes (2023)2.26
• Robust Anytime Learning Of Markov Decision Processes (2022)0.00
• Policy Learning For Robust Markov Decision Process With A Mismatched Generative Model (2022)0.00