JudgeBench

JudgeBench: A Benchmark for Evaluating LLM-Based Judges 📃 [Paper] • 💻 [Github] • 🤗 [Dataset] • 🏆 [Leaderboard] JudgeBench is a benchmark aimed at evaluating LLM-based judges for objective correctness on challenging response pairs. For more information on how the response pairs are constructed, please see our paper.

Papers using JudgeBench (4)

Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge2026HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments2025