Robust Risk-sensitive Reinforcement Learning With Conditional Value-at-risk

Abstract

Robust Markov Decision Processes (RMDPs) have received significant research interest, offering an alternative to standard Markov Decision Processes (MDPs) that often assume fixed transition probabilities. RMDPs address this by optimizing for the worst-case scenarios within ambiguity sets. While earlier studies on RMDPs have largely centered on risk-neutral reinforcement learning (RL), with the goal of minimizing expected total discounted costs, in this paper, we analyze the robustness of CVaR-based risk-sensitive RL under RMDP. Firstly, we consider predetermined ambiguity sets. Based on the coherency of CVaR, we establish a connection between robustness and risk sensitivity, thus, techniques in risk-sensitive RL can be adopted to solve the proposed problem. Furthermore, motivated by the existence of decision-dependent uncertainty in real-world problems, we study problems with state-action-dependent ambiguity sets. To solve this, we define a new risk measure named NCVaR and build the eq

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyni2024robust

Related papers

• Tight Bayesian Ambiguity Sets For Robust Mdps (2018)0.00
• A Bayesian Approach To Robust Reinforcement Learning (2019)0.00
• Improving Robustness Via Risk Averse Distributional Reinforcement Learning (2020)0.00
• Robust Lagrangian And Adversarial Policy Gradient For Robust Constrained Markov Decision Processes (2023)2.26
• Sample Complexity Of Robust Reinforcement Learning With A Generative Model (2021)0.00
• Towards Safe Reinforcement Learning Via Constraining Conditional Value-at-risk (2022)0.00
• Lyapunov Robust Constrained-mdps: Soft-constrained Robustly Stable Policy Optimization Under Model Uncertainty (2021)0.00
• The Curious Price Of Distributional Robustness In Reinforcement Learning With A Generative Model (2023)0.00