Exponential Bellman Equation And Improved Regret Bounds For Risk-sensitive Reinforcement Learning

Abstract

We study risk-sensitive reinforcement learning (RL) based on the entropic risk measure. Although existing works have established non-asymptotic regret guarantees for this problem, they leave open an exponential gap between the upper and lower bounds. We identify the deficiencies in existing algorithms and their analysis that result in such a gap. To remedy these deficiencies, we investigate a simple transformation of the risk-sensitive Bellman equations, which we call the exponential Bellman equation. The exponential Bellman equation inspires us to develop a novel analysis of Bellman backup procedures in risk-sensitive RL algorithms, and further motivates the design of a novel exploration mechanism. We show that these analytic and algorithmic innovations together lead to improved regret upper bounds over existing ones.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfei2021exponential

Related papers

• Bridging Distributional And Risk-sensitive Reinforcement Learning With Provable Regret Bounds (2022)0.00
• Non-stationary Risk-sensitive Reinforcement Learning: Near-optimal Dynamic Regret, Adaptive Detection, And Separation Design (2022)3.58
• Taming Equilibrium Bias In Risk-sensitive Multi-agent Reinforcement Learning (2024)0.00
• Risk-sensitive Reinforcement Learning With Exponential Criteria (2022)0.00
• Continuous-time Risk-sensitive Reinforcement Learning Via Quadratic Variation Penalty (2024)0.00
• Risk Bounds And Rademacher Complexity In Batch Reinforcement Learning (2021)0.00
• Provably Efficient Iterated Cvar Reinforcement Learning With Function Approximation And Human Feedback (2023)0.00
• Regret Bounds For Episodic Risk-sensitive Linear Quadratic Regulator (2024)0.00