How To Sense The World: Leveraging Hierarchy In Multimodal Perception For Robust Reinforcement Learning Agents

Abstract

This work addresses the problem of sensing the world: how to learn a multimodal representation of a reinforcement learning agent's environment that allows the execution of tasks under incomplete perceptual conditions. To address such problem, we argue for hierarchy in the design of representation models and contribute with a novel multimodal representation model, MUSE. The proposed model learns hierarchical representations: low-level modality-specific representations, encoded from raw observation data, and a high-level multimodal representation, encoding joint-modality information to allow robust state estimation. We employ MUSE as the sensory representation model of deep reinforcement learning agents provided with multimodal observations in Atari games. We perform a comparative study over different designs of reinforcement learning agents, showing that MUSE allows agents to perform tasks under incomplete perceptual experience with minimal performance loss. Finally, we evaluate the per

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvasco2021how

Related papers

• Multi-horizon Representations With Hierarchical Forward Models For Reinforcement Learning (2022)0.00
• Zero-shot Cross-modal Transfer Of Reinforcement Learning Policies Through A Global Workspace (2024)0.00
• Hierarchical Reinforcement Learning In Complex 3D Environments (2023)0.00
• Playing Games In The Dark: An Approach For Cross-modality Transfer In Reinforcement Learning (2019)0.00
• Exploring The Limits Of Hierarchical World Models In Reinforcement Learning (2024)6.34
• Agents That Listen: High-throughput Reinforcement Learning With Multiple Sensory Systems (2021)8.09
• Learning Representations In Model-free Hierarchical Reinforcement Learning (2018)11.49
• Efficient Model-based Multi-agent Reinforcement Learning Via Optimistic Equilibrium Computation (2022)0.00