The Distributional Reward Critic Framework For Reinforcement Learning Under Perturbed Rewards

Abstract

The reward signal plays a central role in defining the desired behaviors of agents in reinforcement learning (RL). Rewards collected from realistic environments could be perturbed, corrupted, or noisy due to an adversary, sensor error, or because they come from subjective human feedback. Thus, it is important to construct agents that can learn under such rewards. Existing methodologies for this problem make strong assumptions, including that the perturbation is known in advance, clean rewards are accessible, or that the perturbation preserves the optimal policy. We study a new, more general, class of unknown perturbations, and introduce a distributional reward critic framework for estimating reward distributions and perturbations during training. Our proposed methods are compatible with any RL algorithm. Despite their increased generality, we show that they achieve comparable or better rewards than existing methods in a variety of environments, including those with clean rewards. Under

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2024the

Related papers

• Reinforcement Learning With Perturbed Rewards (2018)13.74
• Disturbing Reinforcement Learning Agents With Corrupted Rewards (2021)0.00
• REBEL: Reward Regularization-based Approach For Robotic Reinforcement Learning From Human Feedback (2023)0.00
• Exploring The Training Robustness Of Distributional Reinforcement Learning Against Noisy State Observations (2021)0.00
• Noise Distribution Decomposition Based Multi-agent Distributional Reinforcement Learning (2023)0.00
• Distributional Reward Estimation For Effective Multi-agent Deep Reinforcement Learning (2022)0.00
• Assessing The Impact Of Distribution Shift On Reinforcement Learning Performance (2024)0.00
• Improving Robustness Via Risk Averse Distributional Reinforcement Learning (2020)0.00