Privorl: Differentially Private Synthetic Dataset For Offline Reinforcement Learning

Abstract

Recently, offline reinforcement learning (RL) has become a popular RL paradigm. In offline RL, data providers share pre-collected datasets -- either as individual transitions or sequences of transitions forming trajectories -- to enable the training of RL models (also called agents) without direct interaction with the environments. Offline RL saves interactions with environments compared to traditional RL, and has been effective in critical areas, such as navigation tasks. Meanwhile, concerns about privacy leakage from offline RL datasets have emerged. To safeguard private information in offline RL datasets, we propose the first differential privacy (DP) offline dataset synthesis method, PrivORL, which leverages a diffusion model and diffusion transformer to synthesize transitions and trajectories, respectively, under DP. The synthetic dataset can then be securely released for downstream analysis and research. PrivORL adopts the popular approach of pre-training a synthesizer on publi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygong2025privorl

Related papers

• Offline Reinforcement Learning With Differential Privacy (2022)0.00
• Preserving Expert-level Privacy In Offline Reinforcement Learning (2024)0.00
• Don't Change The Algorithm, Change The Data: Exploratory Data For Offline Reinforcement Learning (2022)0.00
• D4RL: Datasets For Deep Data-driven Reinforcement Learning (2020)0.00
• Data Valuation For Offline Reinforcement Learning (2022)0.00
• Fewer May Be Better: Enhancing Offline Reinforcement Learning With Reduced Dataset (2025)0.00
• Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage (2023)0.00
• Popri: Private Federated Learning Using Preference-optimized Synthetic Data (2025)0.00