Policy Finetuning: Bridging Sample-efficient Offline And Online Reinforcement Learning

Abstract

Recent theoretical work studies sample-efficient reinforcement learning (RL) extensively in two settings: learning interactively in the environment (online RL), or learning from an offline dataset (offline RL). However, existing algorithms and theories for learning near-optimal policies in these two settings are rather different and disconnected. Towards bridging this gap, this paper initiates the theoretical study of policy finetuning, that is, online RL where the learner has additional access to a "reference policy" \(\mu\) close to the optimal policy \(\pi_\star\) in a certain sense. We consider the policy finetuning problem in episodic Markov Decision Processes (MDPs) with \(S\) states, \(A\) actions, and horizon length \(H\). We first design a sharp offline reduction algorithm -- which simply executes \(\mu\) and runs offline policy optimization on the collected dataset -- that finds an \(\epsilon\) near-optimal policy within \(\widetilde\{O\}(H^3SC^\star/\epsilon^2)\) episodes, w

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyxie2021policy

Related papers

• Finetuning From Offline Reinforcement Learning: Challenges, Trade-offs And Practical Solutions (2023)0.00
• Leveraging Offline Data In Online Reinforcement Learning (2022)0.00
• Adaptive Policy Selection And Fine-tuning Under Interaction Budgets For Offline-to-online Reinforcement Learning (2026)0.00
• Nearly Horizon-free Offline Reinforcement Learning (2021)0.00
• Near-optimal Provable Uniform Convergence In Offline Policy Evaluation For Reinforcement Learning (2020)0.00
• Policy Finetuning In Reinforcement Learning Via Design Of Experiments Using Offline Data (2023)0.00
• Policy Agnostic RL: Offline RL And Online RL Fine-tuning Of Any Class And Backbone (2024)0.00
• PROTO: Iterative Policy Regularized Offline-to-online Reinforcement Learning (2023)0.00