Behavior Estimation From Multi-source Data For Offline Reinforcement Learning

Abstract

Offline reinforcement learning (RL) have received rising interest due to its appealing data efficiency. The present study addresses behavior estimation, a task that lays the foundation of many offline RL algorithms. Behavior estimation aims at estimating the policy with which training data are generated. In particular, this work considers a scenario where the data are collected from multiple sources. In this case, neglecting data heterogeneity, existing approaches for behavior estimation suffers from behavior misspecification. To overcome this drawback, the present study proposes a latent variable model to infer a set of policies from data, which allows an agent to use as behavior policy the policy that best describes a particular trajectory. This model provides with a agent fine-grained characterization for multi-source data and helps it overcome behavior misspecification. This work also proposes a learning algorithm for this model and illustrates its practical usage via extending an

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyzhang2022behavior

Related papers

• Reinforcement Learning For Individual Optimal Policy From Heterogeneous Data (2025)0.00
• Deployment-efficient Reinforcement Learning Via Model-based Offline Optimization (2020)0.00
• Matrix Estimation For Offline Reinforcement Learning With Low-rank Structure (2023)0.00
• Expert-supervised Reinforcement Learning For Offline Policy Learning And Evaluation (2020)0.00
• Expert Or Not? Assessing Data Quality In Offline Reinforcement Learning (2025)0.00
• Conservative Bayesian Model-based Value Expansion For Offline Policy Optimization (2022)0.00
• Behavior Prior Representation Learning For Offline Reinforcement Learning (2022)0.00
• Provably Efficient Offline Reinforcement Learning With Perturbed Data Sources (2023)0.00