Policy Composition In Reinforcement Learning Via Multi-objective Policy Optimization

Abstract

We enable reinforcement learning agents to learn successful behavior policies by utilizing relevant pre-existing teacher policies. The teacher policies are introduced as objectives, in addition to the task objective, in a multi-objective policy optimization setting. Using the Multi-Objective Maximum a Posteriori Policy Optimization algorithm (Abdolmaleki et al. 2020), we show that teacher policies can help speed up learning, particularly in the absence of shaping rewards. In two domains with continuous observation and action spaces, our agents successfully compose teacher policies in sequence and in parallel, and are also able to further extend the policies of the teachers in order to solve the task. Depending on the specified combination of task and teacher(s), teacher(s) may naturally act to limit the final performance of an agent. The extent to which agents are required to adhere to teacher policies are determined by hyperparameters which determine both the effect of teachers on l

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymishra2023policy

Related papers

• Actor-critic Policy Optimization In Partially Observable Multiagent Environments (2018)0.00
• Reward Constrained Policy Optimization (2018)0.00
• Robust And Diverse Multi-agent Learning Via Rational Policy Gradient (2025)0.00
• Interpretable Multi-objective Reinforcement Learning Through Policy Orchestration (2018)0.00
• Sample-efficient Multi-objective Learning Via Generalized Policy Improvement Prioritization (2023)5.24
• Multi-path Policy Optimization (2019)0.00
• B2MAPO: A Batch-by-batch Multi-agent Policy Optimization To Balance Performance And Efficiency (2024)0.00
• Unified Policy Optimization For Continuous-action Reinforcement Learning In Non-stationary Tasks And Games (2022)2.26