Federated Distributional Reinforcement Learning With Distributional Critic Regularization

Abstract

Federated reinforcement learning typically aggregates value functions or policies by parameter averaging, which emphasizes expected return and can obscure statistical multimodality and tail behavior that matter in safety-critical settings. We formalize federated distributional reinforcement learning (FedDistRL), where clients parametrize quantile value function critics and federate these networks only. We also propose TR-FedDistRL, which builds a per client, risk-aware Wasserstein barycenter over a temporal buffer. This local barycenter provides a reference region to constrain the parameter averaged critic, ensuring necessary distributional information is not averaged out during the federation process. The distributional trust region is implemented as a shrink-squash step around this reference. Under fixed-policy evaluation, the feasibility map is nonexpansive and the update is contractive in a probe-set Wasserstein metric under evaluation. Experiments on a bandit, multi-agent gridworl

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymillard2026federated

Related papers

• Fedhpd: Heterogeneous Federated Reinforcement Learning Via Policy Distillation (2025)2.26
• On The Linear Speedup Of Personalized Federated Reinforcement Learning With Shared Representations (2024)0.00
• Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysis (2025)0.00
• Federated Natural Policy Gradient And Actor Critic Methods For Multi-task Reinforcement Learning (2023)0.00
• Federated Offline Policy Optimization With Dual Regularization (2024)3.58
• Momentum For The Win: Collaborative Federated Reinforcement Learning Across Heterogeneous Environments (2024)0.00
• Provably Robust Federated Reinforcement Learning (2025)0.00
• Federated Reinforcement Learning With Constraint Heterogeneity (2024)0.00