JudgeBench

JudgeBench: A Benchmark for Evaluating LLM-Based Judges 📃 [Paper] • 💻 [Github] • 🤗 [Dataset] • 🏆 [Leaderboard] JudgeBench is a benchmark aimed at evaluating LLM-based judges for objective correctness on challenging response pairs. For more information on how the response pairs are constructed, please see our paper.

Papers using JudgeBench (5)

Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks2026IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models2026RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards2025Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments2025