Modularity Benefits Reinforcement Learning Agents With Competing Homeostatic Drives

Abstract

The problem of balancing conflicting needs is fundamental to intelligence. Standard reinforcement learning algorithms maximize a scalar reward, which requires combining different objective-specific rewards into a single number. Alternatively, different objectives could also be combined at the level of action value, such that specialist modules responsible for different objectives submit different action suggestions to a decision process, each based on rewards that are independent of one another. In this work, we explore the potential benefits of this alternative strategy. We investigate a biologically relevant multi-objective problem, the continual homeostasis of a set of variables, and compare a monolithic deep Q-network to a modular network with a dedicated Q-learner for each variable. We find that the modular agent: a) requires minimal exogenously determined exploration; b) has improved sample efficiency; and c) is more robust to out-of-domain perturbation.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydulberg2022modularity

Related papers

• Modular Continual Learning In A Unified Visual Environment (2017)0.00
• Modular Multi-objective Deep Reinforcement Learning With Decision Values (2017)10.74
• Modular Networks Prevent Catastrophic Interference In Model-based Multi-task Reinforcement Learning (2021)0.00
• Continuous Homeostatic Reinforcement Learning For Self-regulated Autonomous Agents (2021)0.00
• Modularity In Reinforcement Learning Via Algorithmic Independence In Credit Assignment (2021)0.00
• Reinforcement Learning With Brain-inspired Modulation Can Improve Adaptation To Environmental Changes (2022)0.00
• Heterogeneous Knowledge For Augmented Modular Reinforcement Learning (2023)0.00
• Integrating Distributed Architectures In Highly Modular RL Libraries (2020)0.00