Reinforcement Learning From Reflective Feedback (RLRF): Aligning And Improving Llms Via Fine-grained Self-reflection

Abstract

Despite the promise of RLHF in aligning LLMs with human preferences, it often leads to superficial alignment, prioritizing stylistic changes over improving downstream performance of LLMs. Underspecified preferences could obscure directions to align the models. Lacking exploration restricts identification of desirable outputs to improve the models. To overcome these challenges, we propose a novel f