Assessing The Impact Of Distribution Shift On Reinforcement Learning Performance
2024 · Ted Fujimoto, Joshua Suetterlein, Samrat Chatterjee, et al.
Abstract
Research in machine learning is making progress in fixing its own reproducibility crisis. Reinforcement learning (RL), in particular, faces its own set of unique challenges. Comparison of point estimates, and plots that show successful convergence to the optimal policy during training, may obfuscate overfitting or dependence on the experimental setup. Although researchers in RL have proposed reliability metrics that account for uncertainty to better understand each algorithm's strengths and weaknesses, the recommendations of past work do not assume the presence of out-of-distribution observations. We propose a set of evaluation methods that measure the robustness of RL algorithms under distribution shifts. The tools presented here argue for the need to account for performance over time while the agent is acting in its environment. In particular, we recommend time series analysis as a method of observational RL evaluation. We also show that the unique properties of RL and simulated dyna
Authors
(none)
Tags
Stats
Related papers
- Mitigating Distribution Shift In Model-based Offline RL Via Shifts-aware Reward Learning (2024)0.00
- A Comparative Analysis Of Expected And Distributional Reinforcement Learning (2019)9.76
- Rethinking Out-of-distribution Detection For Reinforcement Learning: Advancing Methods For Evaluation And Detection (2024)2.26
- Beyond Expected Return: Accounting For Policy Reproducibility When Evaluating Reinforcement Learning Algorithms (2023)3.58
- Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage (2023)0.00
- Measuring The Reliability Of Reinforcement Learning Algorithms (2019)4.43
- Moments Matter:stabilizing Policy Optimization Using Return Distributions (2026)0.00
- A Differential Perspective On Distributional Reinforcement Learning (2025)0.00