Abstract
This study investigates the dynamics of agentic AI within multi-agent systems (MAS), focusing on coordination, negotiation, and cooperation mechanisms in competitive and collaborative digital ecosystems. Employing a simulation-based methodology utilizing multi-agent reinforcement learning (MARL) frameworks, the research analyzes hypothetical yet realistic datasets derived from environments like the StarCraft Multi-Agent Challenge (SMAC) and Multi-Agent Particle Environment (MPE). Key findings reveal that collaborative scenarios yield higher success rates (up to 64.47%) and shorter negotiation times compared to competitive ones, while mixed environments exhibit balanced but volatile cooperation indices. Algorithms such as Proximal Policy Optimization (PPO) demonstrate superior stability in convergence, though Deep Q-Networks (DQN) excel in reward maximization. The analysis underscores the need for adaptive negotiation protocols to mitigate autonomy-induced conflicts. Conclusions highlight implications for scalable AI deployment in real-world applications, such as supply chain optimization and autonomous robotics, advocating for hybrid governance models to foster emergent cooperation. This work bridges theoretical gaps in agentic interactions, offering reproducible insights for advancing MAS resilience.