Accmer: Accelerating Multi-agent Experience Replay With Cache Locality-aware Prioritization

Abstract

Multi-Agent Experience Replay (MER) is a key component of off-policy reinforcement learning~(RL) algorithms. By remembering and reusing experiences from the past, experience replay significantly improves the stability of RL algorithms and their learning efficiency. In many scenarios, multiple agents interact in a shared environment during online training under centralized training and decentralized execution~(CTDE) paradigm. Current multi-agent reinforcement learning~(MARL) algorithms consider experience replay with uniform sampling or based on priority weights to improve transition data sample efficiency in the sampling phase. However, moving transition data histories for each agent through the processor memory hierarchy is a performance limiter. Also, as the agents' transitions continuously renew every iteration, the finite cache capacity results in increased cache misses. To this end, we propose \name, that repeatedly reuses the transitions~(experiences) for a window of \(n\) step

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keygogineni2023accmer

Related papers

• MAC-PO: Multi-agent Experience Replay Via Collective Priority Optimization (2023)0.00
• Stabilising Experience Replay For Deep Multi-agent Reinforcement Learning (2017)0.00
• Higher Replay Ratio Empowers Sample-efficient Multi-agent Reinforcement Learning (2024)0.00
• Prioritized Guidance For Efficient Multi-agent Reinforcement Learning Exploration (2019)0.00
• Neighborhood Mixup Experience Replay: Local Convex Interpolation For Improved Sample Efficiency In Continuous Control Tasks (2022)0.00
• Efficient Episodic Memory Utilization Of Cooperative Multi-agent Reinforcement Learning (2024)0.00
• Associative Memory Based Experience Replay For Deep Reinforcement Learning (2022)6.34
• Prioritized Generative Replay (2024)0.00