Kl-regularization Itself Is Differentially Private In Bandits And RLHF

Abstract

Differential Privacy (DP) provides a rigorous framework for privacy, ensuring the outputs of data-driven algorithms remain statistically indistinguishable across datasets that differ in a single entry. While guaranteeing DP generally requires explicitly injecting noise either to the algorithm itself or to its outputs, the intrinsic randomness of existing algorithms presents an opportunity to achieve DP ``for free''. In this work, we explore the role of regularization in achieving DP across three different decision-making problems: multi-armed bandits, linear contextual bandits, and reinforcement learning from human feedback (RLHF), in offline data settings. We show that adding KL-regularization to the learning objective (a common approach in optimization algorithms) makes the action sampled from the resulting stochastic policy itself differentially private. This offers a new route to privacy guarantees without additional noise injection, while also preserving the inherent advantage of

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2025kl

Related papers

• Offline Reinforcement Learning With Differential Privacy (2022)0.00
• Sharp Analysis For Kl-regularized Contextual Bandits And RLHF (2024)0.00
• Efficient Differentially Private Fine-tuning Of Llms Via Reinforcement Learning (2025)0.00
• Local Differential Privacy For Regret Minimization In Reinforcement Learning (2020)0.00
• Near-optimal Differentially Private Reinforcement Learning (2022)0.00
• Privacy Preserving Reinforcement Learning For Population Processes (2024)0.00
• Privacy-preserving Reinforcement Learning From Human Feedback Via Decoupled Reward Modeling (2026)0.00
• Locally Private Distributed Reinforcement Learning (2020)0.00