Federated Reinforcement Learning With Constraint Heterogeneity

Abstract

We study a Federated Reinforcement Learning (FedRL) problem with constraint heterogeneity. In our setting, we aim to solve a reinforcement learning problem with multiple constraints while \(N\) training agents are located in \(N\) different environments with limited access to the constraint signals and they are expected to collaboratively learn a policy satisfying all constraint signals. Such learning problems are prevalent in scenarios of Large Language Model (LLM) fine-tuning and healthcare applications. To solve the problem, we propose federated primal-dual policy optimization methods based on traditional policy gradient methods. Specifically, we introduce \(N\) local Lagrange functions for agents to perform local policy updates, and these agents are then scheduled to periodically communicate on their local policies. Taking natural policy gradient (NPG) and proximal policy optimization (PPO) as policy optimization methods, we mainly focus on two instances of our algorithms, ie, \{Fe

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyjin2024federated

Related papers

• Momentum For The Win: Collaborative Federated Reinforcement Learning Across Heterogeneous Environments (2024)0.00
• Fedhpd: Heterogeneous Federated Reinforcement Learning Via Policy Distillation (2025)2.26
• Finite-time Analysis Of On-policy Heterogeneous Federated Reinforcement Learning (2024)0.00
• Asynchronous Federated Reinforcement Learning With Policy Gradient Updates: Algorithm Design And Convergence Analysis (2024)0.00
• Federated Natural Policy Gradient And Actor Critic Methods For Multi-task Reinforcement Learning (2023)0.00
• Heterogeneous Multi-robot Reinforcement Learning (2023)6.77
• On The Linear Speedup Of Personalized Federated Reinforcement Learning With Shared Representations (2024)0.00
• Heterogeneity-aware Personalized Federated Learning Via Adaptive Dual-agent Reinforcement Learning (2025)0.00