JudgeBench

JudgeBench: A Benchmark for Evaluating LLM-Based Judges 📃 [Paper] • 💻 [Github] • 🤗 [Dataset] • 🏆 [Leaderboard] JudgeBench is a benchmark aimed at evaluating LLM-based judges for objective correctness on challenging response pairs. For more information on how the response pairs are constructed, please see our paper.

Papers using JudgeBench (5)

IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models2026RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments2025Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity2025Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks2026