HumanEval
Canonical242papers using it
2021first seen
164 hand-written Python programming problems with unit tests, evaluating functional code generation from docstrings (pass@k).
Papers using HumanEval (200)
- A Survey on Large Language Models for Code GenerationCODESIM: Multi-Agent Code Generation and Problem Solving through
Simulation-Driven Planning and DebuggingKodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for
CodingFully Autonomous Programming using Iterative Multi-Agent Debugging with
Large Language ModelsKodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for CodingPiloting Copilot, Codex, and StarCoder2: Hot Temperature, Cold Prompts, or Black Magic?Enhancing Code Generation via Bidirectional Comment-Level Mutual GroundingRegression Accumulation in Multi-Turn LLM Programming ConversationsType-Constrained Code Generation with Language ModelsGrammar-Based Code Representation: Is It a Worthy Pursuit for LLMs?Poison with Style: A Practical Poisoning Attack on Code Large Language ModelsFocused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone PointsEnhancing LLM-Based Code Generation with Complexity Metrics: A Feedback-Driven ApproachBenchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark QualityA Taxonomy of Inefficiencies in LLM-Generated Python CodePrompt Alchemy: Automatic Prompt Refinement for Enhancing Code
GenerationLarge Language Model Guided Self-Debugging Code GenerationOn the Effectiveness of Large Language Models in Domain-Specific Code
GenerationQualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM
Quality ChecksImporting Phantoms: Measuring LLM Package Hallucination VulnerabilitiesFrom Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical DebuggingmHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code GenerationFrom Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code IntelligenceAutomated Repair of Ambiguous Problem Descriptions for LLM-Based Code GenerationOpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMsARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative RefinementEnhancing LLM Code Generation with Ensembles: A Similarity-Based Selection ApproachCODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and DebuggingDafny as Verification-Aware Intermediate Language for Code GenerationFixing Function-Level Code Generation Errors for Foundation Large
Language ModelsCodeMirage: Hallucinations in Code Generated by Large Language ModelsWhere Is Self-admitted Code Generated by Large Language Models on GitHub?Improved Large Language Diffusion ModelsUsing Semantic Distance to Estimate Uncertainty in LLM-Based Code GenerationAn Execution-Verified Multi-Language Benchmark for Code Semantic ReasoningPrompt Optimization for LLM Code Generation via Reinforcement LearningEvaluating the Environmental Impact of using SLMs and Prompt Engineering for Code GenerationSolidCoder: Bridging the Mental-Reality Gap in LLM Code Generation through Concrete ExecutionRealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development PracticesMIST-RL: Mutation-based Incremental Suite Testing via Reinforcement LearningReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement LearningCangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming LanguageLLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback LoopsThink Anywhere in Code GenerationTextBFGS: A Case-Based Reasoning Approach to Code Optimization via Error-Operator RetrievalIntentCoding: Amplifying User Intent in Code GenerationOmniCode: A Benchmark for Evaluating Software Engineering AgentsBatCoder: Self-Supervised Bidirectional Code-Documentation Learning via Back-TranslationTowards Green AI: Decoding the Energy of LLM Inference in Software DevelopmentAutomated Test Suite Enhancement Using Large Language Models with Few-shot PromptingBootstrapping Code Translation with Weighted Multilanguage ExplorationAssessing and Improving the Representativeness of Code Generation Benchmarks Using Knowledge Units (KUs) of Programming Languages -- An Empirical StudyShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code GenerationBenchmarking Large Language Models for ABAP Code Generation: An Empirical Study on Iterative Improvement by Compiler FeedbackNOIR: Privacy-Preserving Generation of Code with Open-Source LLMsMulti-task Code LLMs: Data Mix or Model Merge?Adaptive Confidence Gating in Multi-Agent Collaboration for Efficient and Optimized Code GenerationPay for Hints, Not Answers: LLM Shepherding for Cost-Efficient InferenceGenerating Verifiable Chain of Thoughts from Exection-TracesCODE ACROSTIC: Robust Watermarking for Code GenerationAn Exploratory Study of Bayesian Prompt Optimization for Test-Driven Code Generation with Large Language ModelsPyBangla at BLP-2025 Task 2: Enhancing Bangla-to-Python Code Generation with Iterative Self-Correction and Multilingual AgentsHow Natural Language Proficiency Shapes GenAI Code for Software Engineering TasksEffective Code Membership Inference for Code Completion Models via Adversarial PromptsMulti-LLM Orchestration for High-Quality Code Generation: Exploiting Complementary Model StrengthsContractEval: A Benchmark for Evaluating Contract-Satisfying Assertions in Code GenerationAssessing Coherency and Consistency of Code Execution Reasoning by Large Language ModelsAdapTrack: Constrained Decoding without Distorting LLM's Output IntentUnderstanding the Characteristics of LLM-Generated Property-Based Tests in Exploring Edge CasesTALM: Dynamic Tree-Structured Multi-Agent Framework with Long-Term Memory for Scalable Code GenerationSK2Decompile: LLM-based Two-Phase Binary Decompilation from Skeleton to SkinEvaluating SAP Joule for Code GenerationA Multi-Language Object-Oriented Programming Benchmark for Large Language ModelsCodeGrad: Integrating Multi-Step Verification with Gradient-Based LLM RefinementStatic Analysis as a Feedback Loop: Enhancing LLM-Generated Code Beyond CorrectnessAlignment with Fill-In-the-Middle for Enhancing Code GenerationSTEPWISE-CODEX-Bench: Evaluating Complex Multi-Function Comprehension and Fine-Grained Execution ReasoningAssessing Small Language Models for Code Generation: An Empirical Study with BenchmarksRethinking Verification for LLM Code Generation: From Generation to TestingA Mixture of Linear Corrections Generates Secure CodeTurning the Tide: Repository-based Code ReflectionCREME: Robustness Enhancement of Code LLMs via Layer-Aware Model EditingMemoCoder: Automated Function Synthesis using LLM-Supported AgentsRethinking Verification for LLM Code Generation: From Generation to
TestingEfficient Code LLM Training via Distribution-Consistent and Diversity-Aware Data SelectionWhen Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task DescriptionsAdaDec: A Uncertainty-Guided Lookahead Decoding Framework for LLM-Based Code GenerationPrompt engineering and framework: implementation to increase code reliability based guideline for LLMsGuaranteed Guess: A Language Modeling Approach for CISC-to-RISC Transpilation with Testing GuaranteesBenchmarking AI Models in Software Engineering: A Review, Search Tool,
and Enhancement ProtocolGuaranteed Guess: A Language Modeling Approach for CISC-to-RISC
Transpilation with Testing GuaranteesFrom Code Foundation Models to Agents and Applications: A Practical Guide to Code IntelligenceRewriting Pre-Training Data Boosts LLM Performance in Math and CodeCodeMixBench: Evaluating Large Language Models on Code Generation with
Code-Mixed PromptsWeb-Bench: A LLM Code Benchmark Based on Web Standards and FrameworksRethinking Repetition Problems of LLMs in Code GenerationEvaluating Large Language Models for Code ReviewSelf-Correcting Code Generation Using Small Language ModelsFeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair TasksThrowBench: Benchmarking LLMs by Predicting Runtime ExceptionsModularization is Better: Effective Code Generation with Modular
PromptingCan LLMs Enable Verification in Mainstream Programming?ACECODER: Acing Coder RL via Automated Test-Case SynthesisReasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language
Models Through Logic Unit AlignmentUnitCoder: Scalable Iterative Code Synthesis with Unit Test GuidanceCode-Vision: Evaluating Multimodal LLMs Logic Understanding and Code
Generation CapabilitiesCodeCriticBench: A Holistic Code Critique Benchmark for Large Language
ModelsThinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal RefinementIsolating Language-Coding from Problem-Solving: Benchmarking LLMs with
PseudoEvalPragmatic Reasoning improves LLM Code GenerationGuided Code Generation with LLMs: A Multi-Agent Framework for Complex
Code TasksCoCoNUT: Structural Code Understanding does not fall out of a treeCodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code
GenerationLeveraging Metamemory Mechanisms for Enhanced Data-Free Code Generation
in LLMsPlanning-Driven Programming: A Large Language Model Programming WorkflowContext-Augmented Code Generation Using Programming Knowledge GraphsCan Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and ExecutionEffiLearner: Enhancing Efficiency of Generated Code via Self-OptimizationReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code GenerationLLM4Decompile: Decompiling Binary Code with Large Language ModelsEvaluating Large Language Models Trained on CodeCode Llama: Open Foundation Models for CodeCodeGen: An Open Large Language Model for Code with Multi-Turn Program
SynthesisCodeGeeX: A Pre-Trained Model for Code Generation with Multilingual
Benchmarking on HumanEval-XStarCoder: may the source be with you!Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of
Large Language Models for Code GenerationWizardCoder: Empowering Code Large Language Models with Evol-InstructEvaluating the Code Quality of AI-Assisted Code Generation Tools: An
Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPTCodeT: Code Generation with Generated TestsCodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems
for Real-World Repo-level Coding ChallengesSelf-Edit: Fault-Aware Code Editor for Code GenerationThe Stack: 3 TB of permissively licensed source codeDebug like a Human: A Large Language Model Debugger via Verifying
Runtime Execution Step-by-stepCYCLE: Learning to Self-Refine the Code GenerationMagicoder: Empowering Code Generation with OSS-InstructLiveCodeBench: Holistic and Contamination Free Evaluation of Large
Language Models for CodeClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on
Class-level Code GenerationInteractive Code Generation via Test-Driven User-Intent FormalizationMapCoder: Multi-Agent Code Generation for Competitive Problem SolvingOn Evaluating the Efficiency of Source Code Generated by LLMsAgentCoder: Multi-Agent-based Code Generation with Iterative Testing and
OptimisationSelfEvolve: A Code Evolution Framework via Large Language ModelsStructured Chain-of-Thought Prompting for Code GenerationOctoPack: Instruction Tuning Code Large Language ModelsParsel: Algorithmic Reasoning with Language Models by Composing
DecompositionsFault-Aware Neural Code RankersMultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural
Code GenerationCodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent TechnologyCrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code
CompletionUsing Large Language Models to Generate JUnit Tests: An Empirical StudySOEN-101: Code Generation by Emulating Software Process Models Using
Large Language Model AgentsPythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMsJavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating
Large Language ModelsOOP: Object-Oriented Programming Evaluation Benchmark for Large Language
ModelsInvestigating the Performance of Language Models for Completing Code in
Functional Programming Languages: a Haskell Case StudyReCode: Robustness Evaluation of Code Generation ModelsCodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code
GenerationNExT: Teaching Large Language Models to Reason about Code ExecutionSemCoder: Training Code Language Models with Comprehensive Semantics
ReasoningIs Self-Repair a Silver Bullet for Code Generation?CRUXEval: A Benchmark for Code Reasoning, Understanding and ExecutionPredicting Code Coverage without ExecutionOpenCodeInterpreter: Integrating Code Generation with Execution and
RefinementSoftware Vulnerability and Functionality Assessment using LLMsTop Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval:
Evolving Coding Benchmarks via LLMSelf-Organized Agents: A LLM Multi-Agent Framework toward Ultra
Large-Scale Code Generation and OptimizationFrom Code to Correctness: Closing the Last Mile of Code Generation with
Hierarchical DebuggingReasoning Runtime Behavior of a Program with LLM: How Far Are We?Large Language Model Evaluation Via Multi AI Agents: Preliminary resultsLeTI: Learning to Generate from Textual InteractionsEnhancing Large Language Models in Coding Through Multi-Perspective
Self-ConsistencyCodeChain: Towards Modular Code Generation Through Chain of
Self-revisions with Representative Sub-modulesNoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional
CorrectnessHumanEval on Latest GPT Models -- 2024CodeShell Technical ReportLow-Cost Language Models: Survey and Performance Evaluation on Python
Code GenerationXFT: Unlocking the Power of Code Instruction Tuning by Simply Merging
Upcycled Mixture-of-ExpertsNaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and
Natural User PromptsMHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code GenerationHow Efficient is LLM-Generated Code? A Rigorous & High-Standard
BenchmarkTowards Understanding the Characteristics of Code Generation Errors Made
by Large Language ModelsSelfCodeAlign: Self-Alignment for Code GenerationA Preliminary Study of Multilingual Code Language Models for Code
Generation Task Using Translated BenchmarksThe Program Testing Ability of Large Language Models for CodeThe RealHumanEval: Evaluating Large Language Models' Abilities to
Support ProgrammersAutoCoder: Enhancing Code Large Language Model with
\textsc{AIEV-Instruct}Qiskit Code Assistant: Training LLMs for generating Quantum Computing
CodeDivide-and-Conquer Meets Consensus: Unleashing the Power of Functions in
Code GenerationTowards Large Language Model Aided Program RefinementDOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code
GenerationPlanning In Natural Language Improves LLM Search For Code GenerationHow Do Your Code LLMs Perform? Empowering Code Instruction Tuning with
High-Quality DataMulti-Programming Language Ensemble for Code Generation in Large
Language ModelSelection of Prompt Engineering Techniques for Code Generation through
Predicting Code ComplexityTraining Language Models on Synthetic Edit Sequences Improves Code
SynthesisSelf-Evolving Multi-Agent Collaboration Networks for Software
DevelopmentCodeTree: Agent-guided Tree Search for Code Generation with Large
Language ModelsDoes Few-Shot Learning Help LLM Performance in Code Synthesis?PerfCodeGen: Improving Performance of LLM Generated Code with Execution
Feedback