JudgeBench

JudgeBench: A Benchmark for Evaluating LLM-Based Judges 📃 [Paper] • 💻 [Github] • 🤗 [Dataset] • 🏆 [Leaderboard] JudgeBench is a benchmark aimed at evaluating LLM-based judges for objective correctness on challenging response pairs. For more information on how the response pairs are constructed, please see our paper.

Papers using JudgeBench (5)

Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models2026Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge2026HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments2025