Guided Flow Policy: Learning From High-value Actions In Offline Reinforcement Learning

Abstract

Offline reinforcement learning often relies on behavior regularization that enforces policies to remain close to the dataset distribution. However, such approaches fail to distinguish between high-value and low-value actions in their regularization components. We introduce Guided Flow Policy (GFP), which couples a multi-step flow-matching policy with a distilled one-step actor. The actor directs the flow policy through weighted behavior cloning to focus on cloning high-value actions from the dataset rather than indiscriminately imitating all state-action pairs. In turn, the flow policy constrains the actor to remain aligned with the dataset's best transitions while maximizing the critic. This mutual guidance enables GFP to achieve state-of-the-art performance across 144 state and pixel-based tasks from the OGBench, Minari, and D4RL benchmarks, with substantial gains on suboptimal datasets and challenging tasks. Webpage: https://simple-robotics.github.io/publications/guided-flow-policy/

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytiofack2025guided

Related papers

• Evolving Diffusion And Flow Matching Policies For Online Reinforcement Learning (2025)0.00
• FM-IRL: Flow-matching For Reward Modeling And Policy Regularization In Reinforcement Learning (2025)0.00
• Value-guidance Meanflow For Offline Multi-agent Reinforcement Learning (2026)0.00
• Flowpg: Action-constrained Policy Gradient With Normalizing Flows (2024)0.00
• Reverse Flow Matching: A Unified Framework For Online Reinforcement Learning With Diffusion And Flow Policies (2026)0.00
• Constrained Policy Optimization With Explicit Behavior Density For Offline Reinforcement Learning (2023)0.00
• One-step Flow Q-learning: Addressing The Diffusion Policy Bottleneck In Offline Reinforcement Learning (2025)0.00
• Flow To Control: Offline Reinforcement Learning With Lossless Primitive Discovery (2022)3.58