Your Policy Regularizer Is Secretly An Adversary

Abstract

Policy regularization methods such as maximum entropy regularization are widely used in reinforcement learning to improve the robustness of a learned policy. In this paper, we show how this robustness arises from hedging against worst-case perturbations of the reward function, which are chosen from a limited set by an imagined adversary. Using convex duality, we characterize this robust set of adversarial reward perturbations under KL and alpha-divergence regularization, which includes Shannon and Tsallis entropy regularization as special cases. Importantly, generalization guarantees can be given within this robust set. We provide detailed discussion of the worst-case reward perturbations, and present intuitive empirical examples to illustrate this robustness and its relationship with generalization. Finally, we discuss how our analysis complements and extends previous results on adversarial reward robustness and path consistency optimality conditions.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybrekelmans2022your

Related papers

• Understanding The Impact Of Entropy On Policy Optimization (2018)0.00
• Evidence On The Regularisation Properties Of Maximum-entropy Reinforcement Learning (2025)0.00
• Complexity-regularized Proximal Policy Optimization (2025)0.00
• Policy Optimization Reinforcement Learning With Entropy Regularization (2019)0.00
• Marginalized State Distribution Entropy Regularization In Policy Optimization (2019)0.00
• An Entropy Regularization Free Mechanism For Policy-based Reinforcement Learning (2021)0.00
• Do You Need The Entropy Reward (in Practice)? (2022)0.00
• Increasing Entropy To Boost Policy Gradient Performance On Personalization Tasks (2023)0.00