Dynamic Value Estimation For Single-task Multi-scene Reinforcement Learning

Abstract

Training deep reinforcement learning agents on environments with multiple levels / scenes / conditions from the same task, has become essential for many applications aiming to achieve generalization and domain transfer from simulation to the real world. While such a strategy is helpful with generalization, the use of multiple scenes significantly increases the variance of samples collected for policy gradient computations. Current methods continue to view this collection of scenes as a single Markov Decision Process (MDP) with a common value function; however, we argue that it is better to treat the collection as a single environment with multiple underlying MDPs. To this end, we propose a dynamic value estimation (DVE) technique for these multiple-MDP environments, motivated by the clustering effect observed in the value function distribution across different scenes. The resulting agent is able to learn a more accurate and scene-specific value function estimate (and hence the advantag

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysingh2020dynamic

Related papers

• Rethinking Value Function Learning For Generalization In Reinforcement Learning (2022)0.00
• Distributional Reward Estimation For Effective Multi-agent Deep Reinforcement Learning (2022)0.00
• Diversity Through Exclusion (DTE): Niche Identification For Reinforcement Learning Through Value-decomposition (2023)0.00
• VDFD: Multi-agent Value Decomposition Framework With Disentangled World Model (2023)0.00
• SVDE: Scalable Value-decomposition Exploration For Cooperative Multi-agent Reinforcement Learning (2023)0.00
• MAVEN: Multi-agent Variational Exploration (2019)0.00
• A Model-based Approach For Sample-efficient Multi-task Reinforcement Learning (2019)0.00
• MMD-MIX: Value Function Factorisation With Maximum Mean Discrepancy For Cooperative Multi-agent Reinforcement Learning (2021)0.00