Dynamic Policy Fusion For User Alignment Without Re-interaction

Abstract

Deep reinforcement learning (RL) policies, although optimal in terms of task rewards, may not align with the personal preferences of human users. To ensure this alignment, a naive solution would be to retrain the agent using a reward function that encodes the user's specific preferences. However, such a reward function is typically not readily available, and as such, retraining the agent from scratch can be prohibitively expensive. We propose a more practical approach - to adapt the already trained policy to user-specific needs with the help of human feedback. To this end, we infer the user's intent through trajectory-level feedback and combine it with the trained task policy via a theoretically grounded dynamic policy fusion approach. As our approach collects human feedback on the very same trajectories used to learn the task policy, it does not require any additional interactions with the environment, making it a zero-shot approach. We empirically demonstrate in a number of environme

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypalattuparambil2024dynamic

Related papers

• Policy Fusion For Adaptive And Customizable Reinforcement Learning Agents (2021)0.00
• Flexible Attention-based Multi-policy Fusion For Efficient Deep Reinforcement Learning (2022)2.26
• Characterizing Policy Divergence For Personalized Meta-reinforcement Learning (2020)0.00
• Adaptive Learning Of Design Strategies Over Non-hierarchical Multi-fidelity Models Via Policy Alignment (2024)0.00
• Pref-guide: Continual Policy Learning From Real-time Human Feedback Via Preference-based Learning (2025)0.00
• Shared Autonomy Via Deep Reinforcement Learning (2018)15.40
• Directed Policy Gradient For Safe Reinforcement Learning With Human Advice (2018)0.00
• Efficient Meta Reinforcement Learning For Preference-based Fast Adaptation (2022)0.00