Emergent Coordination And Phase Structure In Independent Multi-agent Reinforcement Learning

Abstract

A clearer understanding of when coordination emerges, fluctuates, or collapses in decentralized multi-agent reinforcement learning (MARL) is increasingly sought in order to characterize the dynamics of multi-agent learning systems. We revisit fully independent Q-learning (IQL) as a minimal decentralized testbed and run large-scale experiments across environment size L and agent density rho. We construct a phase map using two axes - the cooperative success rate (CSR) and a stability index derived from TD-error variance - revealing three distinct regimes: a coordinated and stable phase, a fragile transition region, and a jammed or disordered phase. A sharp double Instability Ridge separates these regimes and corresponds to persistent kernel drift, the time-varying shift of each agent's effective transition kernel induced by others' policy updates. Synchronization analysis further shows that temporal alignment is required for sustained cooperation, and that competition between drift and s

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyamaguchi2025emergent

Related papers

• Coordination-driven Learning In Multi-agent Problem Spaces (2018)0.00
• Dealing With Non-stationarity In Decentralized Cooperative Multi-agent Deep Reinforcement Learning Via Multi-timescale Learning (2023)0.00
• MA2QL: A Minimalist Approach To Fully Decentralized Multi-agent Reinforcement Learning (2022)0.00
• Learning To Coordinate Via Quantum Entanglement In Multi-agent Reinforcement Learning (2026)0.00
• Contextual Knowledge Sharing In Multi-agent Reinforcement Learning With Decentralized Communication And Coordination (2025)0.00
• Strategic Coordination For Evolving Multi-agent Systems: A Hierarchical Reinforcement And Collective Learning Approach (2025)0.00
• Influence-based Reinforcement Learning For Intrinsically-motivated Agents (2021)0.00
• Provably Efficient Multi-agent Reinforcement Learning With Fully Decentralized Communication (2021)0.00