A Self-supervised Auxiliary Loss For Deep RL In Partially Observable Settings

Abstract

In this work we explore an auxiliary loss useful for reinforcement learning in environments where strong performing agents are required to be able to navigate a spatial environment. The auxiliary loss proposed is to minimize the classification error of a neural network classifier that predicts whether or not a pair of states sampled from the agents current episode trajectory are in order. The classifier takes as input a pair of states as well as the agent's memory. The motivation for this auxiliary loss is that there is a strong correlation with which of a pair of states is more recent in the agents episode trajectory and which of the two states is spatially closer to the agent. Our hypothesis is that learning features to answer this question encourages the agent to learn and internalize in memory representations of states that facilitate spatial reasoning. We tested this auxiliary loss on a navigation task in a gridworld and achieved 9.6% increase in accumulative episode reward compar

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyahmed2021a

Related papers

• Intrinsically Motivated Self-supervised Learning In Reinforcement Learning (2021)3.58
• Return-based Contrastive Representation Learning For Reinforcement Learning (2021)12.17
• When Does Self-prediction Help? Understanding Auxiliary Tasks In Reinforcement Learning (2024)0.00
• Improving Reinforcement Learning Efficiency With Auxiliary Tasks In Non-visual Environments: A Comparison (2023)2.26
• An Empirical Study On The Power Of Future Prediction In Partially Observable Environments (2024)0.00
• Continual Auxiliary Task Learning (2022)0.00
• Multi-agent Off-policy Actor-critic Reinforcement Learning For Partially Observable Environments (2024)2.26
• Ego-foresight: Self-supervised Learning Of Agent-aware Representations For Improved RL (2024)0.00