Arena-Hard

The 'Arena-Hard' dataset is a benchmark used to evaluate the performance of LLMs in alignment tasks by providing challenging scenarios that require reasoning and decision-making without verifiable ground-truth verifiers.

Papers using Arena-Hard (13)

QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards2026References Improve LLM Alignment in Non-Verifiable Domains2026Reward Model Routing in Alignment2025Online Rubrics Elicitation from Pairwise Comparisons2025TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization2025Pretrain Value, Not Reward: Decoupled Value Policy Optimization2025Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play2024DPO Meets PPO: Reinforced Token Optimization for RLHF2024SimPO: Simple Preference Optimization with a Reference-Free Reward2024 · 17 citesRLHF Workflow: From Reward Modeling to Online RLHF2024 · 3 citesThe Perfect Blend: Redefining RLHF with Mixture of Judges2024 · 2 citesAlphaDPO: Adaptive Reward Margin for Direct Preference Optimization2024T-REG: Preference Optimization with Token-Level Reward Regularization2024