Preferred-action-optimized Diffusion Policies For Offline Reinforcement Learning

Abstract

Offline reinforcement learning (RL) aims to learn optimal policies from previously collected datasets. Recently, due to their powerful representational capabilities, diffusion models have shown significant potential as policy models for offline RL issues. However, previous offline RL algorithms based on diffusion policies generally adopt weighted regression to improve the policy. This approach optimizes the policy only using the collected actions and is sensitive to Q-values, which limits the potential for further performance enhancement. To this end, we propose a novel preferred-action-optimized diffusion policy for offline RL. In particular, an expressive conditional diffusion model is utilized to represent the diverse distribution of a behavior policy. Meanwhile, based on the diffusion model, preferred actions within the same behavior distribution are automatically generated through the critic function. Moreover, an anti-noise preference optimization is designed to achieve policy im

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2024preferred

Related papers

• Diffusion Policies As An Expressive Policy Class For Offline Reinforcement Learning (2022)0.00
• Diffusion Policies Creating A Trust Region For Offline Reinforcement Learning (2024)8.04
• Diffusion Policies With Value-conditional Optimization For Offline Reinforcement Learning (2025)0.00
• Diffpogan: Diffusion Policies With Generative Adversarial Networks For Offline Reinforcement Learning (2024)0.00
• Entropy-regularized Diffusion Policy With Q-ensembles For Offline Reinforcement Learning (2024)3.58
• Diffusion Policy Through Conditional Proximal Policy Optimization (2026)0.00
• Policy Representation Via Diffusion Probability Model For Reinforcement Learning (2023)0.00
• Diffusion Actor-critic: Formulating Constrained Policy Iteration As Diffusion Noise Regression For Offline Reinforcement Learning (2024)2.92