Deterministic Model Of Incremental Multi-agent Boltzmann Q-learning: Transient Cooperation, Metastability, And Oscillations

Abstract

Multi-Agent Reinforcement Learning involves agents that learn together in a shared environment, leading to emergent dynamics sensitive to initial conditions and parameter variations. A Dynamical Systems approach, which studies the evolution of multi-component systems over time, has uncovered some of the underlying dynamics by constructing deterministic approximation models of stochastic algorithms. In this work, we demonstrate that even in the simplest case of independent Q-learning with a Boltzmann exploration policy, significant discrepancies arise between the actual algorithm and previous approximations. We elaborate why these models actually approximate interesting variants rather than the original incremental algorithm. To explain the discrepancies, we introduce a new discrete-time approximation model that explicitly accounts for agents' update frequencies within the learning process and show that its dynamics fundamentally differ from the simplified dynamics of prior models. We i

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygoll2024deterministic

Related papers

• Non-cooperative Multi-agent Systems With Exploring Agents (2020)0.00
• Beyond Strict Competition: Approximate Convergence Of Multi Agent Q-learning Dynamics (2023)0.00
• Convergence And Connectivity: Dynamics Of Multi-agent Q-learning In Random Networks (2025)0.00
• Unsynchronized Decentralized Q-learning: Two Timescale Analysis By Persistence (2023)2.26
• Modeling The Effects Of Environmental And Perceptual Uncertainty Using Deterministic Reinforcement Learning Dynamics With Partial Observability (2021)9.59
• The Dynamics Of Q-learning In Population Games: A Physics-inspired Continuity Equation Model (2022)0.00
• Emergent Coordination And Phase Structure In Independent Multi-agent Reinforcement Learning (2025)0.00
• On The Stability Of Learning In Network Games With Many Players (2024)0.00