Leverage The Average: An Analysis Of KL Regularization In RL

Abstract

Recent Reinforcement Learning (RL) algorithms making use of Kullback-Leibler (KL) regularization as a core component have shown outstanding performance. Yet, only little is understood theoretically about why KL regularization helps, so far. We study KL regularization within an approximate value iteration scheme and show that it implicitly averages q-values. Leveraging this insight, we provide a very strong performance bound, the very first to combine two desirable aspects: a linear dependency to the horizon (instead of quadratic) and an error propagation term involving an averaging effect of the estimation errors (instead of an accumulation effect). We also study the more general case of an additional entropy regularizer. The resulting abstract scheme encompasses many existing RL algorithms. Some of our assumptions do not hold with neural networks, so we complement this theoretical analysis with an extensive empirical study.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvieillard2020leverage

Related papers

• Rethinking KL Regularization In RLHF: From Value Estimation To Gradient Optimization (2025)0.00
• Sharp Analysis For Kl-regularized Contextual Bandits And RLHF (2024)0.00
• Information Asymmetry In Kl-regularized RL (2019)0.00
• Regularization Matters In Policy Optimization (2019)2.68
• Exploiting Hierarchy For Learning And Transfer In Kl-regularized RL (2019)0.00
• An \(L^2\) Analysis Of Reinforcement Learning In High Dimensions With Kernel And Neural Network Approximation (2021)0.00
• Averaged-dqn: Variance Reduction And Stabilization For Deep Reinforcement Learning (2016)0.00
• The Effective Horizon Explains Deep RL Performance In Stochastic Environments (2023)3.42