Optimizing Latent Goal By Learning From Trajectory Preference

Abstract

A glowing body of work has emerged focusing on instruction-following policies for open-world agents, aiming to better align the agent's behavior with human intentions. However, the performance of these policies is highly susceptible to the initial prompt, which leads to extra efforts in selecting the best instructions. We propose a framework named Preference Goal Tuning (PGT). PGT allows an instruction following policy to interact with the environment to collect several trajectories, which will be categorized into positive and negative samples based on preference. Then we use preference learning to fine-tune the initial goal latent representation with the categorized trajectories while keeping the policy backbone frozen. The experiment result shows that with minimal data and training, PGT achieves an average relative improvement of 72.0% and 81.6% over 17 tasks in 2 different foundation policies respectively, and outperforms the best human-selected instructions. Moreover, PGT surpasses

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhao2024optimizing

Related papers

• Improved Exploration Through Latent Trajectory Optimization In Deep Deterministic Policy Gradient (2019)0.00
• Pref-guide: Continual Policy Learning From Real-time Human Feedback Via Preference-based Learning (2025)0.00
• PTR-PPO: Proximal Policy Optimization With Prioritized Trajectory Replay (2021)0.00
• TIC-GRPO: Provable And Efficient Optimization For Reinforcement Learning From Human Feedback (2025)0.00
• Phgpo: Pheromone-guided Policy Optimization For Long-horizon Tool Planning (2026)0.00
• Learning Policies From Self-play With Policy Gradients And MCTS Value Estimates (2019)0.00
• Variance Reduction Based Partial Trajectory Reuse To Accelerate Policy Gradient Optimization (2022)0.00
• Reusing Trajectories In Policy Gradients Enables Fast Convergence (2025)0.00