Simudice: Offline Policy Optimization Through World Model Updates And DICE Estimation

Abstract

In offline reinforcement learning, deriving an effective policy from a pre-collected set of experiences is challenging due to the distribution mismatch between the target policy and the behavioral policy used to collect the data, as well as the limited sample size. Model-based reinforcement learning improves sample efficiency by generating simulated experiences using a learned dynamic model of the environment. However, these synthetic experiences often suffer from the same distribution mismatch. To address these challenges, we introduce SimuDICE, a framework that iteratively refines the initial policy derived from offline data using synthetically generated experiences from the world model. SimuDICE enhances the quality of these simulated experiences by adjusting the sampling probabilities of state-action pairs based on stationary DIstribution Correction Estimation (DICE) and the estimated confidence in the model's predictions. This approach guides policy improvement by balancing experi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybrita2024simudice

Related papers

• Optidice: Offline Policy Optimization Via Stationary Distribution Correction Estimation (2021)0.00
• Offsim: Offline Simulator For Model-based Offline Inverse Reinforcement Learning (2025)0.00
• Constrained Latent Action Policies For Model-based Offline Reinforcement Learning (2024)0.00
• Diffusion-dice: In-sample Diffusion Guidance For Offline Reinforcement Learning (2024)0.00
• Efficient Offline Policy Optimization With A Learned Model (2022)0.00
• Revisiting Design Choices In Offline Model-based Reinforcement Learning (2021)6.34
• Towards Data-driven Offline Simulations For Online Reinforcement Learning (2022)0.00
• Deployment-efficient Reinforcement Learning Via Model-based Offline Optimization (2020)0.00