Distributionally Adaptive Meta Reinforcement Learning

Abstract

Meta-reinforcement learning algorithms provide a data-driven way to acquire policies that quickly adapt to many tasks with varying rewards or dynamics functions. However, learned meta-policies are often effective only on the exact task distribution on which they were trained and struggle in the presence of distribution shift of test-time rewards or transition dynamics. In this work, we develop a framework for meta-RL algorithms that are able to behave appropriately under test-time distribution shifts in the space of tasks. Our framework centers on an adaptive approach to distributional robustness that trains a population of meta-policies to be robust to varying levels of distribution shift. When evaluated on a potentially shifted test-time distribution of tasks, this allows us to choose the meta-policy with the most appropriate level of robustness, and use it to perform fast adaptation. We formally show how our framework allows for improved regret under distribution shift, and empirica

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyajay2022distributionally

Related papers

• Meta-reinforcement Learning Robust To Distributional Shift Via Model Identification And Experience Relabeling (2020)0.00
• Model-based Adversarial Meta-reinforcement Learning (2020)0.00
• A Tutorial On Meta-reinforcement Learning (2023)10.85
• Offline Meta-reinforcement Learning With Online Self-supervision (2021)0.00
• Context Meta-reinforcement Learning Via Neuromodulation (2021)6.34
• Meta-reinforcement Learning With Universal Policy Adaptation: Provable Near-optimality Under All-task Optimum Comparator (2024)0.00
• Efficient Meta Reinforcement Learning For Preference-based Fast Adaptation (2022)0.00
• Mitigating Distribution Shift In Model-based Offline RL Via Shifts-aware Reward Learning (2024)0.00