Rethinking Out-of-distribution Detection For Reinforcement Learning: Advancing Methods For Evaluation And Detection
2024 · Linas Nasvytis, Kai Sandbrink, Jakob Foerster, et al.
Abstract
While reinforcement learning (RL) algorithms have been successfully applied across numerous sequential decision-making problems, their generalization to unforeseen testing environments remains a significant concern. In this paper, we study the problem of out-of-distribution (OOD) detection in RL, which focuses on identifying situations at test time that RL agents have not encountered in their training environments. We first propose a clarification of terminology for OOD detection in RL, which aligns it with the literature from other machine learning domains. We then present new benchmark scenarios for OOD detection, which introduce anomalies with temporal autocorrelation into different components of the agent-environment loop. We argue that such scenarios have been understudied in the current literature, despite their relevance to real-world situations. Confirming our theoretical predictions, our experimental results suggest that state-of-the-art OOD detectors are not able to identify
Authors
(none)
Tags
Stats
Related papers
- Guaranteeing Out-of-distribution Detection In Deep RL Via Transition Estimation (2025)0.00
- Out-of-distribution Dynamics Detection: Rl-relevant Benchmarks And Results (2021)0.00
- Galilai: Out-of-task Distribution Detection Using Causal Active Experimentation For Safe Transfer RL (2021)0.00
- Uncertainty-based Out-of-distribution Detection In Deep Reinforcement Learning (2019)7.50
- Assessing The Impact Of Distribution Shift On Reinforcement Learning Performance (2024)0.00
- Sero: Self-supervised Reinforcement Learning For Recovery From Out-of-distribution Situations (2023)3.58
- The Role Of Pretrained Representations For The OOD Generalization Of Reinforcement Learning Agents (2021)0.00
- An Information-theoretic Analysis Of OOD Generalization In Meta-reinforcement Learning (2025)0.00