Federated Offline Policy Optimization With Dual Regularization

Abstract

Federated Reinforcement Learning (FRL) has been deemed as a promising solution for intelligent decision-making in the era of Artificial Internet of Things. However, existing FRL approaches often entail repeated interactions with the environment during local updating, which can be prohibitively expensive or even infeasible in many real-world domains. To overcome this challenge, this paper proposes a novel offline federated policy optimization algorithm, named \(\texttt\{DRPO\}\), which enables distributed agents to collaboratively learn a decision policy only from private and static data without further environmental interactions. \(\texttt\{DRPO\}\) leverages dual regularization, incorporating both the local behavioral policy and the global aggregated policy, to judiciously cope with the intrinsic two-tier distributional shifts in offline FRL. Theoretical analysis characterizes the impact of the dual regularization on performance, demonstrating that by achieving the right balance there

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyyue2024federated

Related papers

• Federated Offline Reinforcement Learning (2022)0.00
• Federated Ensemble-directed Offline Reinforcement Learning (2023)0.00
• PROTO: Iterative Policy Regularized Offline-to-online Reinforcement Learning (2023)0.00
• Federated Offline Reinforcement Learning: Collaborative Single-policy Coverage Suffices (2024)0.00
• Offline Policy Optimization In RL With Variance Regularizaton (2022)0.00
• Federated Offline Policy Learning (2023)0.00
• Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage (2023)0.00
• Provably Robust Federated Reinforcement Learning (2025)0.00