MMLU-Pro

A harder, cleaner MMLU variant with ten-way choices and more reasoning-heavy questions to reduce saturation.

Papers using MMLU-Pro (31)

Decision Protocols in Multi-Agent Large Language Model Conversations2026K-Quantization and its Impact on Output Performance2026Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient Reasoning2026D-COT: Disciplined Chain-of-Thought Learning for Efficient Reasoning in Small Language Models2026Benchmark Illusion: Disagreement Among Llms And Its Scientific Consequences2026AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewards2025Group-Aware Reinforcement Learning for Output Diversity in Large Language Models2025Let the Model Distribute Its Doubt: Confidence Estimation through Verbalized Probability Distribution2025AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment2025Can Multiple Responses from an LLM Reveal the Sources of Its Uncertainty?2025Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD2025Token Constraint Decoding Improves Robustness on Question Answering for Large Language Models2025AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection2025General-Reasoner: Advancing LLM Reasoning Across All Domains2025DyePack: Provably Flagging Test Set Contamination in LLMs Using Backdoors2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning2025When an LLM is apprehensive about its answers -- and when its uncertainty is justified2025T1: Tool-integrated Self-verification for Test-time Compute Scaling in Small Language Models2025Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings2025Reinforcing General Reasoning without Verifiers2025Answer Matching Outperforms Multiple Choice for Language Model Evaluation2025From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation2025Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD2025The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning2025AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection2025ReviewInstruct: A Review-Driven Multi-Turn Conversations Generation Method for Large Language Models2025INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profiling2025Calibrating LLM Confidence by Probing Perturbed Representation Stability2025Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?2025MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs2024