Learning To Play Soccer From Scratch: Sample-efficient Emergent Coordination Through Curriculum-learning And Competition

Abstract

This work proposes a scheme that allows learning complex multi-agent behaviors in a sample efficient manner, applied to 2v2 soccer. The problem is formulated as a Markov game, and solved using deep reinforcement learning. We propose a basic multi-agent extension of TD3 for learning the policy of each player, in a decentralized manner. To ease learning, the task of 2v2 soccer is divided in three stages: 1v0, 1v1 and 2v2. The process of learning in multi-agent stages (1v1 and 2v2) uses agents trained on a previous stage as fixed opponents. In addition, we propose using experience sharing, a method that shares experience from a fixed opponent, trained in a previous stage, for training the agent currently learning, and a form of frame-skipping, to raise performance significantly. Our results show that high quality soccer play can be obtained with our approach in just under 40M interactions. A summarized video of the resulting game play can be found in https://youtu.be/f25l1j1U9RM.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysamtani2021learning

Related papers

• Emergent Coordination Through Competition (2019)0.00
• Tizero: Mastering Multi-agent Football With Curriculum Learning And Self-play (2023)2.26
• Stackelberg Games For Learning Emergent Behaviors During Competitive Autocurricula (2023)5.84
• Two-stage Training Algorithm For AI Robot Soccer (2021)0.00
• Rsoccer: A Framework For Studying Reinforcement Learning In Small And Very Small Size Robot Soccer (2021)8.35
• Collaboration Of AI Agents Via Cooperative Multi-agent Deep Reinforcement Learning (2019)0.00
• Coach-player Multi-agent Reinforcement Learning For Dynamic Team Composition (2021)0.00
• Mimicking To Dominate: Imitation Learning Strategies For Success In Multiagent Competitive Games (2023)0.00