Reinforcement Learning From Hierarchical Critics

Abstract

In this study, we investigate the use of global information to speed up the learning process and increase the cumulative rewards of reinforcement learning (RL) in competition tasks. Within the actor-critic RL, we introduce multiple cooperative critics from two levels of the hierarchy and propose a reinforcement learning from hierarchical critics (RLHC) algorithm. In our approach, each agent receives value information from local and global critics regarding a competition task and accesses multiple cooperative critics in a top-down hierarchy. Thus, each agent not only receives low-level details but also considers coordination from higher levels, thereby obtaining global information to improve the training performance. Then, we test the proposed RLHC algorithm against the benchmark algorithm, proximal policy optimisation (PPO), for two experimental scenarios performed in a Unity environment consisting of tennis and soccer agents' competitions. The results showed that RLHC outperforms the

Stats

• citations11
• S2 citations—
• github stars0
• HF likes0
• heat score8.09
• arxiv keycao2019reinforcement

Related papers

• Skill-critic: Refining Learned Skills For Hierarchical Reinforcement Learning (2023)7.50
• Developing Cooperative Policies For Multi-stage Reinforcement Learning Tasks (2022)0.00
• Reducing Overestimation Bias In Multi-agent Domains Using Double Centralized Critics (2019)0.00
• Natural Actor-critic Converges Globally For Hierarchical Linear Quadratic Regulator (2019)0.00
• Actor-attention-critic For Multi-agent Reinforcement Learning (2018)0.00
• Multi-agent Actor-critic For Mixed Cooperative-competitive Environments (2017)0.00
• Ensemble Reinforcement Learning In Continuous Spaces -- A Hierarchical Multi-step Approach For Policy Training (2022)2.26
• Stackelberg Actor-critic: Game-theoretic Reinforcement Learning Algorithms (2021)0.00