Towards Data-driven Offline Simulations For Online Reinforcement Learning

Abstract

Modern decision-making systems, from robots to web recommendation engines, are expected to adapt: to user preferences, changing circumstances or even new tasks. Yet, it is still uncommon to deploy a dynamically learning agent (rather than a fixed policy) to a production system, as it's perceived as unsafe. Using historical data to reason about learning algorithms, similar to offline policy evaluation (OPE) applied to fixed policies, could help practitioners evaluate and ultimately deploy such adaptive agents to production. In this work, we formalize offline learner simulation (OLS) for reinforcement learning (RL) and propose a novel evaluation protocol that measures both fidelity and efficiency of the simulation. For environments with complex high-dimensional observations, we propose a semi-parametric approach that leverages recent advances in latent state discovery in order to achieve accurate and efficient offline simulations. In preliminary experiments, we show the advantage of our

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytang2022towards

Related papers

• Accelerating Offline Reinforcement Learning Application In Real-time Bidding And Recommendation: Potential Use Of Simulation (2021)0.00
• When To Trust Your Simulator: Dynamics-aware Hybrid Offline-and-online Reinforcement Learning (2022)2.26
• Offline Policy Evaluation For Reinforcement Learning With Adaptively Collected Data (2023)0.00
• Persim: Data-efficient Offline Reinforcement Learning With Heterogeneous Agents Via Personalized Simulators (2021)0.00
• Offsim: Offline Simulator For Model-based Offline Inverse Reinforcement Learning (2025)0.00
• AWAC: Accelerating Online Reinforcement Learning With Offline Datasets (2020)0.00
• Bridging The Gap Between Offline And Online Reinforcement Learning Evaluation Methodologies (2022)0.00
• Representation Matters: Offline Pretraining For Sequential Decision Making (2021)0.00