Maximum Entropy RL (provably) Solves Some Robust RL Problems

Abstract

Many potential applications of reinforcement learning (RL) require guarantees that the agent will perform well in the face of disturbances to the dynamics or reward function. In this paper, we prove theoretically that maximum entropy (MaxEnt) RL maximizes a lower bound on a robust RL objective, and thus can be used to learn policies that are robust to some disturbances in the dynamics and the reward function. While this capability of MaxEnt RL has been observed empirically in prior work, to the best of our knowledge our work provides the first rigorous proof and theoretical characterization of the MaxEnt RL robust set. While a number of prior robust RL algorithms have been designed to handle similar disturbances to the reward function or dynamics, these methods typically require additional moving parts and hyperparameters on top of a base RL algorithm. In contrast, our results suggest that MaxEnt RL by itself is robust to certain disturbances, without requiring any additional modificat

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyeysenbach2021maximum

Related papers

• Do You Need The Entropy Reward (in Practice)? (2022)0.00
• Robust Adversarial Reinforcement Learning Via Bounded Rationality Curricula (2023)0.00
• Entropy Regularized Reinforcement Learning Using Large Deviation Theory (2021)6.34
• Off-policy Maximum Entropy RL With Future State And Action Visitation Measures (2024)0.00
• Reinforcement Learning With Non-ergodic Reward Increments: Robustness Via Ergodicity Transformations (2023)0.00
• Predictable Reinforcement Learning Dynamics Through Entropy Rate Minimization (2023)0.00
• Maximum Entropy-regularized Multi-goal Reinforcement Learning (2019)0.00
• Soft Actor-critic: Off-policy Maximum Entropy Deep Reinforcement Learning With A Stochastic Actor (2018)0.00