Provably Efficient Iterated Cvar Reinforcement Learning With Function Approximation And Human Feedback

Abstract

Risk-sensitive reinforcement learning (RL) aims to optimize policies that balance the expected reward and risk. In this paper, we present a novel risk-sensitive RL framework that employs an Iterated Conditional Value-at-Risk (CVaR) objective under both linear and general function approximations, enriched by human feedback. These new formulations provide a principled way to guarantee safety in each decision making step throughout the control process. Moreover, integrating human feedback into risk-sensitive RL framework bridges the gap between algorithmic decision-making and human participation, allowing us to also guarantee safety for human-in-the-loop systems. We propose provably sample-efficient algorithms for this Iterated CVaR RL and provide rigorous theoretical analysis. Furthermore, we establish a matching lower bound to corroborate the optimality of our algorithms in a linear context.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2023provably

Related papers

• Provably Efficient Reinforcement Learning With Linear Function Approximation (2019)11.76
• Acrel: Adversarial Conditional Value-at-risk Reinforcement Learning (2021)0.00
• Towards Safe Reinforcement Learning Via Constraining Conditional Value-at-risk (2022)0.00
• Posterior Sampling With Delayed Feedback For Reinforcement Learning With Linear Function Approximation (2023)0.00
• Uniform-pac Bounds For Reinforcement Learning With Linear Function Approximation (2021)0.00
• A Risk-sensitive Approach To Policy Optimization (2022)3.58
• Conditionally Elicitable Dynamic Risk Measures For Deep Reinforcement Learning (2022)0.00
• A Nearly Optimal And Low-switching Algorithm For Reinforcement Learning With General Function Approximation (2023)0.00