Multi-horizon Representations With Hierarchical Forward Models For Reinforcement Learning

Abstract

Learning control from pixels is difficult for reinforcement learning (RL) agents because representation learning and policy learning are intertwined. Previous approaches remedy this issue with auxiliary representation learning tasks, but they either do not consider the temporal aspect of the problem or only consider single-step transitions, which may cause learning inefficiencies if important environmental changes take many steps to manifest. We propose Hierarchical \(k\)-Step Latent (HKSL), an auxiliary task that learns multiple representations via a hierarchy of forward models that learn to communicate and an ensemble of \(n\)-step critics that all operate at varying magnitudes of step skipping. We evaluate HKSL in a suite of 30 robotic control tasks with and without distractors and a task of our creation. We find that HKSL either converges to higher or optimal episodic returns more quickly than several alternative representation learning approaches. Furthermore, we find that HKSL's

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymcinroe2022multi

Related papers

• Learning Representations In Model-free Hierarchical Reinforcement Learning (2018)11.49
• Learning Temporally-consistent Representations For Data-efficient Reinforcement Learning (2021)0.00
• Near-optimal Representation Learning For Hierarchical Reinforcement Learning (2018)0.00
• Exploiting Hierarchy For Learning And Transfer In Kl-regularized RL (2019)0.00
• Exploring The Limits Of Hierarchical World Models In Reinforcement Learning (2024)6.34
• On The Benefits Of Pixel-based Hierarchical Policies For Task Generalization (2024)0.00
• Reconciling Spatial And Temporal Abstractions For Goal Representation (2024)1.69
• Generating Adjacency-constrained Subgoals In Hierarchical Reinforcement Learning (2020)0.00