Exploring The Training Robustness Of Distributional Reinforcement Learning Against Noisy State Observations

Abstract

In real scenarios, state observations that an agent observes may contain measurement errors or adversarial noises, misleading the agent to take suboptimal actions or even collapse while training. In this paper, we study the training robustness of distributional Reinforcement Learning (RL), a class of state-of-the-art methods that estimate the whole distribution, as opposed to only the expectation, of the total return. Firstly, we validate the contraction of distributional Bellman operators in the State-Noisy Markov Decision Process (SN-MDP), a typical tabular case that incorporates both random and adversarial state observation noises. In the noisy setting with function approximation, we then analyze the vulnerability of least squared loss in expectation-based RL with either linear or nonlinear function approximation. By contrast, we theoretically characterize the bounded gradient norm of distributional RL loss based on the categorical parameterization equipped with the KL divergence. T

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysun2021exploring

Related papers

• Robust Deep Reinforcement Learning Against Adversarial Perturbations On State Observations (2020)0.00
• The Curious Price Of Distributional Robustness In Reinforcement Learning With A Generative Model (2023)0.00
• Robust Reinforcement Learning On State Observations With Learned Optimal Adversary (2021)0.00
• Noise Distribution Decomposition Based Multi-agent Distributional Reinforcement Learning (2023)0.00
• Distributionally Robust Offline Reinforcement Learning With Linear Function Approximation (2022)0.00
• A Comparative Analysis Of Expected And Distributional Reinforcement Learning (2019)9.76
• Improving Robustness Via Risk Averse Distributional Reinforcement Learning (2020)0.00
• Distributionally Robust Model-based Reinforcement Learning With Large State Spaces (2023)0.00