Emergent Cooperation Under Uncertain Incentive Alignment

Abstract

Understanding the emergence of cooperation in systems of computational agents is crucial for the development of effective cooperative AI. Interaction among individuals in real-world settings are often sparse and occur within a broad spectrum of incentives, which often are only partially known. In this work, we explore how cooperation can arise among reinforcement learning agents in scenarios characterised by infrequent encounters, and where agents face uncertainty about the alignment of their incentives with those of others. To do so, we train the agents under a wide spectrum of environments ranging from fully competitive, to fully cooperative, to mixed-motives. Under this type of uncertainty we study the effects of mechanisms, such as reputation and intrinsic rewards, that have been proposed in the literature to foster cooperation in mixed-motives environments. Our findings show that uncertainty substantially lowers the agents' ability to engage in cooperative behaviour, when that wou

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyorzan2024emergent

Related papers

• Cooperation And Reputation Dynamics With Reinforcement Learning (2021)3.58
• Emergent Cooperation Through Mutual Information Maximization (2020)0.00
• Intrinsic Fluctuations Of Reinforcement Learning Promote Cooperation (2022)9.23
• Emergent Coordination Through Competition (2019)0.00
• Cooperation Dynamics In Multi-agent Systems: Exploring Game-theoretic Scenarios With Mean-field Equilibria (2023)0.00
• Evolving Intrinsic Motivations For Altruistic Behavior (2018)2.26
• The Price Of Paranoia: Robust Risk-sensitive Cooperation In Non-stationary Multi-agent Reinforcement Learning (2026)0.00
• Improved Cooperation By Balancing Exploration And Exploitation In Intertemporal Social Dilemma Tasks (2021)0.00