Policy Finetuning In Reinforcement Learning Via Design Of Experiments Using Offline Data

Abstract

In some applications of reinforcement learning, a dataset of pre-collected experience is already available but it is also possible to acquire some additional online data to help improve the quality of the policy. However, it may be preferable to gather additional data with a single, non-reactive exploration policy and avoid the engineering costs associated with switching policies. In this paper we propose an algorithm with provable guarantees that can leverage an offline dataset to design a single non-reactive policy for exploration. We theoretically analyze the algorithm and measure the quality of the final policy as a function of the local coverage of the original dataset and the amount of additional data collected.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2023policy

Related papers

• Finetuning From Offline Reinforcement Learning: Challenges, Trade-offs And Practical Solutions (2023)0.00
• Offline Retraining For Online RL: Decoupled Policy Learning To Mitigate Exploration Bias (2023)2.56
• Leveraging Offline Data In Online Reinforcement Learning (2022)0.00
• Policy Finetuning: Bridging Sample-efficient Offline And Online Reinforcement Learning (2021)0.00
• Don't Change The Algorithm, Change The Data: Exploratory Data For Offline Reinforcement Learning (2022)0.00
• Revisiting Design Choices In Offline Model-based Reinforcement Learning (2021)6.34
• AWAC: Accelerating Online Reinforcement Learning With Offline Datasets (2020)0.00
• Adaptive Policy Selection And Fine-tuning Under Interaction Budgets For Offline-to-online Reinforcement Learning (2026)0.00