Robust Model-free Reinforcement Learning With Multi-objective Bayesian Optimization

Abstract

In reinforcement learning (RL), an autonomous agent learns to perform complex tasks by maximizing an exogenous reward signal while interacting with its environment. In real-world applications, test conditions may differ substantially from the training scenario and, therefore, focusing on pure reward maximization during training may lead to poor results at test time. In these cases, it is important to trade-off between performance and robustness while learning a policy. While several results exist for robust, model-based RL, the model-free case has not been widely investigated. In this paper, we cast the robust, model-free RL problem as a multi-objective optimization problem. To quantify the robustness of a policy, we use delay margin and gain margin, two robustness indicators that are common in control theory. We show how these metrics can be estimated from data in the model-free setting. We use multi-objective Bayesian optimization (MOBO) to solve efficiently this expensive-to-evaluat

Stats

• citations29
• S2 citations—
• github stars0
• HF likes0
• heat score11.08
• arxiv keyturchetta2019robust

Related papers

• Robust Bayesian Optimization With Reinforcement Learned Acquisition Functions (2022)0.00
• Robust Model-based Reinforcement Learning With An Adversarial Auxiliary Model (2024)0.00
• Safe Reinforcement Learning With Dual Robustness (2023)8.60
• Robust Cooperative Multi-agent Reinforcement Learning:a Mean-field Type Game Perspective (2024)0.00
• Online Robust Reinforcement Learning With Model Uncertainty (2021)0.00
• Combining Pessimism With Optimism For Robust And Efficient Model-based Deep Reinforcement Learning (2021)0.00
• Sample-efficient Robust Multi-agent Reinforcement Learning In The Face Of Environmental Uncertainty (2024)0.00
• Bayesian Robust Optimization For Imitation Learning (2020)0.00