Robust Bayesian Dynamic Programming For On-policy Risk-sensitive Reinforcement Learning

Abstract

We propose a novel framework for risk-sensitive reinforcement learning (RSRL) that incorporates robustness against transition uncertainty. We define two distinct yet coupled risk measures: an inner risk measure addressing state and cost randomness and an outer risk measure capturing transition dynamics uncertainty. Our framework unifies and generalizes most existing RL frameworks by permitting general coherent risk measures for both inner and outer risk measures. Within this framework, we construct a risk-sensitive robust Markov decision process (RSRMDP), derive its Bellman equation, and provide error analysis under a given posterior distribution. We further develop a Bayesian Dynamic Programming (Bayesian DP) algorithm that alternates between posterior updates and value iteration. The approach employs an estimator for the risk-based Bellman operator that combines Monte Carlo sampling with convex optimization, for which we prove strong consistency guarantees. Furthermore, we demonstrat

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhan2025robust

Related papers

• A Bayesian Approach To Robust Reinforcement Learning (2019)0.00
• Online Bayesian Risk-averse Reinforcement Learning (2025)0.00
• On The Foundation Of Distributionally Robust Reinforcement Learning (2023)0.00
• Epistemic Risk-sensitive Reinforcement Learning (2019)0.00
• Conditionally Elicitable Dynamic Risk Measures For Deep Reinforcement Learning (2022)0.00
• Robust Risk-sensitive Reinforcement Learning With Conditional Value-at-risk (2024)5.84
• Non-stationary Risk-sensitive Reinforcement Learning: Near-optimal Dynamic Regret, Adaptive Detection, And Separation Design (2022)3.58
• Robust Lagrangian And Adversarial Policy Gradient For Robust Constrained Markov Decision Processes (2023)2.26