Mirror Descent Policy Optimisation For Robust Constrained Markov Decision Processes

Abstract

Safety is an essential requirement for reinforcement learning systems. The newly emerging framework of robust constrained Markov decision processes allows learning policies that satisfy long-term constraints while providing guarantees under epistemic uncertainty. This paper presents mirror descent policy optimisation for robust constrained Markov decision processes, making use of policy gradient techniques to optimise both the policy (as a maximiser) and the transition kernel (as an adversarial minimiser) on the Lagrangian representing a constrained Markov decision process. Our proposed algorithm obtains an \(\tilde\{\mathcal\{O\}\}\left(1/T^\{1/3\}\right)\) convergence rate in the sample-based robust constrained Markov decision process setting. The paper also contributes an algorithm for approximate gradient descent in the space of transition kernels, which is of independent interest for designing adversarial environments in general Markov decision processes. Experiments confirm the b

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybossens2025mirror

Related papers

• Policy Gradient For Robust Markov Decision Processes (2024)0.00
• Policy Mirror Descent With Temporal Difference Learning: Sample Complexity Under Online Markov Data (2025)0.00
• Policy Optimization For Constrained Mdps With Provable Fast Global Convergence (2021)0.00
• Policy Optimization With Stochastic Mirror Descent (2019)7.50
• Optimal Convergence Rate For Exact Policy Mirror Descent In Discounted Markov Decision Processes (2023)0.00
• Mirror Learning: A Unifying Framework Of Policy Optimisation (2022)0.00
• A Novel Framework For Policy Mirror Descent With General Parameterization And Linear Convergence (2023)2.26
• Robust Lagrangian And Adversarial Policy Gradient For Robust Constrained Markov Decision Processes (2023)2.26