Modularity In Reinforcement Learning Via Algorithmic Independence In Credit Assignment

Abstract

Many transfer problems require re-using previously optimal decisions for solving new tasks, which suggests the need for learning algorithms that can modify the mechanisms for choosing certain actions independently of those for choosing others. However, there is currently no formalism nor theory for how to achieve this kind of modular credit assignment. To answer this question, we define modular credit assignment as a constraint on minimizing the algorithmic mutual information among feedback signals for different decisions. We introduce what we call the modularity criterion for testing whether a learning algorithm satisfies this constraint by performing causal analysis on the algorithm itself. We generalize the recently proposed societal decision-making framework as a more granular formalism than the Markov decision process to prove that for decision sequences that do not contain cycles, certain single-step temporal difference action-value methods meet this criterion while all policy-gr

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychang2021modularity

Related papers

• Counterfactual Credit Assignment In Model-free Reinforcement Learning (2020)0.00
• An Information-theoretic Perspective On Credit Assignment In Reinforcement Learning (2021)0.00
• Selective Credit Assignment (2022)0.00
• Asynchronous Credit Assignment For Multi-agent Reinforcement Learning (2024)0.00
• Modularity Benefits Reinforcement Learning Agents With Competing Homeostatic Drives (2022)0.00
• Decentralized Reinforcement Learning: Global Decision-making Via Local Economic Transactions (2020)0.00
• Cooperative Game-theoretic Credit Assignment For Multi-agent Policy Gradients Via The Core (2025)0.00
• Unified Algorithms For RL With Decision-estimation Coefficients: PAC, Reward-free, Preference-based Learning, And Beyond (2022)5.24