Playing Games In The Dark: An Approach For Cross-modality Transfer In Reinforcement Learning

Abstract

In this work we explore the use of latent representations obtained from multiple input sensory modalities (such as images or sounds) in allowing an agent to learn and exploit policies over different subsets of input modalities. We propose a three-stage architecture that allows a reinforcement learning agent trained over a given sensory modality, to execute its task on a different sensory modality-for example, learning a visual policy over image inputs, and then execute such policy when only sound inputs are available. We show that the generalized policies achieve better out-of-the-box performance when compared to different baselines. Moreover, we show this holds in different OpenAI gym and video game environments, even when using different multimodal generative models and reinforcement learning algorithms.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysilva2019playing

Related papers

• Zero-shot Cross-modal Transfer Of Reinforcement Learning Policies Through A Global Workspace (2024)0.00
• Visual Transfer Between Atari Games Using Competitive Reinforcement Learning (2018)7.50
• Pixel To Policy: DQN Encoders For Within & Cross-game Reinforcement Learning (2023)0.00
• Agents That Listen: High-throughput Reinforcement Learning With Multiple Sensory Systems (2021)8.09
• How To Sense The World: Leveraging Hierarchy In Multimodal Perception For Robust Reinforcement Learning Agents (2021)0.00
• Similarity-based Knowledge Transfer For Cross-domain Reinforcement Learning (2023)0.00
• Foundations For Transfer In Reinforcement Learning: A Taxonomy Of Knowledge Modalities (2023)0.00
• Few-shot Image-to-semantics Translation For Policy Transfer In Reinforcement Learning (2023)0.00