Offsim: Offline Simulator For Model-based Offline Inverse Reinforcement Learning
2025 · Woo-Jin Ahn, Sang-Ryul Baek, Yong-Jun Lee, et al.
Abstract
Reinforcement learning algorithms typically utilize an interactive simulator (i.e., environment) with a predefined reward function for policy training. Developing such simulators and manually defining reward functions, however, is often time-consuming and labor-intensive. To address this, we propose an Offline Simulator (OffSim), a novel model-based offline inverse reinforcement learning (IRL) framework, to emulate environmental dynamics and reward structure directly from expert-generated state-action trajectories. OffSim jointly optimizes a high-entropy transition model and an IRL-based reward function to enhance exploration and improve the generalizability of the learned reward. Leveraging these learned components, OffSim can subsequently train a policy offline without further interaction with the real environment. Additionally, we introduce OffSim\(^+\), an extension that incorporates a marginal reward for multi-dataset settings to enhance exploration. Extensive MuJoCo experiments d
Authors
(none)
Tags
Stats
Related papers
- Towards Data-driven Offline Simulations For Online Reinforcement Learning (2022)0.00
- Morel : Model-based Offline Reinforcement Learning (2020)0.00
- Offline Inverse RL: New Solution Concepts And Provably Efficient Algorithms (2024)0.00
- Persim: Data-efficient Offline Reinforcement Learning With Heterogeneous Agents Via Personalized Simulators (2021)0.00
- When To Trust Your Simulator: Dynamics-aware Hybrid Offline-and-online Reinforcement Learning (2022)2.26
- When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework For Offline Inverse Reinforcement Learning (2023)0.00
- Simudice: Offline Policy Optimization Through World Model Updates And DICE Estimation (2024)0.00
- Accelerating Offline Reinforcement Learning Application In Real-time Bidding And Recommendation: Potential Use Of Simulation (2021)0.00