Chatbot Arena

The 'Chatbot Arena' is a dataset/benchmark used to evaluate the performance of large language models (LLMs) in pairwise judgments, focusing on their calibration and bias in assessing chatbot responses.

Papers using Chatbot Arena (10)

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings2025TOWER: Tree Organized Weighting For Evaluating Complex Instructions2024 · 1 citesSCOPE: Selective Conformal Optimized Pairwise LLM Judging2026Bridging Human and LLM Judgments: Understanding and Narrowing the Gap2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs2025Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models2025Investigating Non-Transitivity in LLM-as-a-Judge2025SkillAggregation: Reference-free LLM-Dependent Aggregation2024 · 1 citesA Statistical Framework for Ranking LLM-Based Chatbots2024