Few-shot Image-to-semantics Translation For Policy Transfer In Reinforcement Learning

Abstract

We investigate policy transfer using image-to-semantics translation to mitigate learning difficulties in vision-based robotics control agents. This problem assumes two environments: a simulator environment with semantics, that is, low-dimensional and essential information, as the state space, and a real-world environment with images as the state space. By learning mapping from images to semantics, we can transfer a policy, pre-trained in the simulator, to the real world, thereby eliminating real-world on-policy agent interactions to learn, which are costly and risky. In addition, using image-to-semantics mapping is advantageous in terms of the computational efficiency to train the policy and the interpretability of the obtained policy over other types of sim-to-real transfer strategies. To tackle the main difficulty in learning image-to-semantics mapping, namely the human annotation cost for producing a training dataset, we propose two techniques: pair augmentation with the transition

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysato2023few

Related papers

• Post-convergence Sim-to-real Policy Transfer: A Principled Alternative To Cherry-picking (2025)0.00
• Transfer Learning Across Simulated Robots With Different Sensors (2019)0.00
• Zero-shot Cross-modal Transfer Of Reinforcement Learning Policies Through A Global Workspace (2024)0.00
• Improving Sample Efficiency In Model-free Reinforcement Learning From Images (2019)16.99
• Playing Games In The Dark: An Approach For Cross-modality Transfer In Reinforcement Learning (2019)0.00
• Learning Robust And Adaptive Real-world Continuous Control Using Simulation And Transfer Learning (2018)0.00
• Contrastive Behavioral Similarity Embeddings For Generalization In Reinforcement Learning (2021)0.00
• Learning Invariances For Policy Generalization (2018)0.00