HumanEval

164 hand-written Python programming problems with unit tests, evaluating functional code generation from docstrings (pass@k).

Papers using HumanEval (45)

Improved Large Language Diffusion Models2026Can Language Models Replace Programmers For Coding? REPOCOD Says 'not Yet'2024 · 3 citesOn the Limits of Layer Pruning for Generative Reasoning in Large Language Models2026Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM2026Improving LLMs via Validator-to-Generator Alignment2026Strategies for Guiding LLMs to Use Software Design Patterns: A Case of Singleton2026Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding2025 · 2 citesEntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction2026OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs2025 · 1 citesFast-dLLM++: Fr\'{e}chet Profile Decoding for Faster Diffusion LLM Inference2026Signature filtering: a lightweight enhancement for statistical watermark detection in large language models2026Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation2026Training Language Models via Neural Cellular Automata2026Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Models2026S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models2026KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks2026AC-ODM: Actor--Critic Online Data Mixing for Sample-Efficient LLM Pretraining2026Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation2026RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding2026TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning2026IntentCoding: Amplifying User Intent in Code Generation2026Group-Aware Reinforcement Learning for Output Diversity in Large Language Models2025TGPR: Tree-Guided Policy Refinement for Robust Self-Debugging of LLMs2025From Implicit Exploration to Structured Reasoning: Leveraging Guideline and Refinement for LLMs2025EAPO: Enhancing Policy Optimization with On-Demand Expert Assistance2025More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation2025Efficient Code LLM Training via Distribution-Consistent and Diversity-Aware Data Selection2025Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs2025AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection2025Prior Prompt Engineering for Reinforcement Fine-Tuning2025CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models2025Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models2025Rethinking Verification for LLM Code Generation: From Generation to Testing2025dInfer: An Efficient Inference Framework for Diffusion Language Models2025Reasoning with Sampling: Your Base Model is Smarter Than You Think2025AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection2025Guided Code Generation with LLMs: A Multi-Agent Framework for Complex Code Tasks2025Leveraging Metamemory Mechanisms for Enhanced Data-Free Code Generation in LLMs2025SelfCodeAlign: Self-Alignment for Code Generation2024 · 2 citesAligning CodeLLMs with Direct Preference Optimization2024 · 1 citesSelf-Explained Keywords Empower Large Language Models for Code Generation2024Adaptive Segment-level Reward: Bridging the Gap Between Action and Reward Space in Alignment2024Learning to Reason via Self-Iterative Process Feedback for Small Language Models2024