MMLU

Massive Multitask Language Understanding — multiple-choice questions across 57 subjects (STEM, humanities, law, medicine) testing broad knowledge and reasoning.

Papers using MMLU (94)

HRM-Text: Efficient Pretraining Beyond Scaling2026Humanity's Last Exam2025 · 21 citesPARTREP: Learning What to Repeat for Decoder-only LLMs2026Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM2026ELiTeFormer: An Efficient Transformer for FPGAs2026evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations2026Decision Protocols in Multi-Agent Large Language Model Conversations2026The Price of Format: Diversity Collapse in LLMs2025 · 2 citesPrompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation2026A general tensor-structured compression scheme for efficient large language models2026Indic-TunedLens: Interpreting Multilingual Models in Indian Languages2026Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation2026Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection2025 · 1 citesTurning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection2025 · 1 citesConfidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning2026Sequential Consensus for Multi-Agent LLM Debates: A Wald-SPRT compute governor with calibration-based failure detection2026YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptation2026Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories2026DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models2026AC-ODM: Actor--Critic Online Data Mixing for Sample-Efficient LLM Pretraining2026Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?2026Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs2026Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs2026AtManRL: Towards Faithful Reasoning via Differentiable Attention Saliency2026Learning When to Sample: Confidence-Aware Self-Consistency for Efficient LLM Chain-of-Thought Reasoning2026Residual Stream Analysis of Overfitting And Structural Disruptions2026SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing2026How do LLMs Compute Verbal Confidence2026LLM Compression by Block Removal with Constrained Binary Optimization2026FIT to Forget: Robust Continual Unlearning for Large Language Models2026Leveraging KV Similarity for Online Structured Pruning in LLMs2025Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information2025Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems2025Fine-Tuning on Noisy Instructions: Effects on Generalization and Performance2025Dr.LLM: Dynamic Layer Routing in LLMs2025EAPO: Enhancing Policy Optimization with On-Demand Expert Assistance2025On Robustness and Reliability of Benchmark-Based Evaluation of LLMs2025CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features2025Thinking Inside the Mask: In-Place Prompting in Diffusion LLMs2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs2025Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders2025GEM: Empowering LLM for both Embedding Generation and Language Understanding2025SELT: Self-Evaluation Tree Search for LLMs with Task Decomposition2025Token Constraint Decoding Improves Robustness on Question Answering for Large Language Models2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignment2025TPTT: Transforming Pretrained Transformers into Titans2025LM2: Large Memory Models2025LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers2025AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection2025Void in Language Models2025Interleaved Reasoning for Large Language Models via Reinforcement Learning2025ProtoReasoning: Prototypes as the Foundation for Generalizable Reasoning in LLMs2025Fine-Tuning on Noisy Instructions: Effects on Generalization and Performance2025Parrot: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs2025Control LLM: Controlled Evolution for Intelligence Retention in LLM2025Selective Self-to-Supervised Fine-Tuning for Generalization in Large Language Models2025YourBench: Easy Custom Evaluation Sets for Everyone2025B-score: Detecting biases in large language models using response history2025TPTT: Transforming Pretrained Transformer into Titans2025Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate2025Lizard: An Efficient Linearization Framework for Large Language Models2025CorrSteer: Steering Improves Task Performance and Safety in LLMs through Correlation-based Sparse Autoencoder Feature Selection2025A Single Character can Make or Break Your LLM Evals2025The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives2025Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework2025A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets2025AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection2025Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models2025Self-Reasoning Language Models: Unfold Hidden Reasoning Chains with Few Reasoning Catalyst2025Dual Decomposition of Weights and Singular Value Low Rank Adaptation2025Calibrating LLM Confidence by Probing Perturbed Representation Stability2025Efficient Data Selection at Scale via Influence Distillation2025More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment2025Large Language Models Could Be Rote Learners2025None of the Above, Less of the Right: Parallel Patterns between Humans and LLMs on Multi-Choice Questions Answering2025Order Independence With Finetuning2025Shifting Perspectives: Steering Vectors for Robust Bias Mitigation in LLMs2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?2025FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategy2025Forget What You Know about LLMs Evaluations -- LLMs are Like a Chameleon2025ORI: O Routing Intelligence2025Leveraging Uncertainty Estimation for Efficient LLM Routing2025Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perception2025Evaluating Expert Contributions in a MoE LLM for Quiz-Based Tasks2025Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data2025The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?2024 · 2 citesBridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments2024 · 2 citesLoLCATs: On Low-Rank Linearizing of Large Language Models2024 · 1 citesSelective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models2024BenTo: Benchmark Task Reduction with In-Context Transferability2024Adaptive Segment-level Reward: Bridging the Gap Between Action and Reward Space in Alignment2024TODO: Enhancing LLM Alignment with Ternary Preferences2024Reasoning Robustness of LLMs to Adversarial Typographical Errors2024Llama 3 Meets MoE: Efficient Upcycling2024