HumanEval

HumanEval-X is a benchmark for the evaluation of the multilingual ability of code generative models. It consists of 820 high-quality human-crafted data samples (each with test cases) in Python, C++, Java, JavaScript, and Go, and can be used for various tasks.

Papers using HumanEval (27)

AutoResearch: An Execution-Grounded Multi-Agent Framework for Reliable Research Workflow Automation2026How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval2026FASE: Fast Adaptive Semantic Entropy for Code Quality2026Smarter Saboteurs, Better Fixers: Scaling & Security in Linear Multi-Agent Workflows2026Strategies for Guiding LLMs to Use Software Design Patterns: A Case of Singleton2026Poison with Style: A Practical Poisoning Attack on Code Large Language Models2026Honest Lying: Understanding Memory Confabulation in Reflexive Agents2026G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks2024 · 1 citesSafesieve: From Heuristics To Experience In Progressive Pruning For Llm-based Multi-agent Communication2025SLIM-RL: Risk-Budgeted Random-Masking RL for Diffusion LLMs Without Trajectory Slicing2026OmniCode: A Benchmark for Evaluating Software Engineering Agents2026TDPGen: Optimizing Agentic Code Generation via Test-Driven Planning and Hierarchical ReAct2026R2V Agent: Teaching SLMs When to Ask for Help2026Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration2026CARD: Towards Conditional Design of Multi-agent Topological Structures2026Think Anywhere in Code Generation2026Adaptive Confidence Gating In Multi-agent Collaboration For Efficient And Optimized Code Generation2026MAR:Multi-Agent Reflexion Improves Reasoning Abilities in LLMs2025A Multi-Agent Framework for Stateful Inference-Time Search2025From Implicit Exploration to Structured Reasoning: Leveraging Guideline and Refinement for LLMs2025AgentGroupChat-V2: Divide-and-Conquer Is What LLM-Based Multi-Agent System Need2025Nexus: A Lightweight and Scalable Multi-Agent Framework for Complex Tasks Automation2025Guided Code Generation with LLMs: A Multi-Agent Framework for Complex Code Tasks2025Leveraging Metamemory Mechanisms for Enhanced Data-Free Code Generation in LLMs2025Divide-and-Conquer Meets Consensus: Unleashing the Power of Functions in Code Generation2024 · 1 citesCPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks2024FALCON: Feedback-driven Adaptive Long/short-term memory reinforced Coding Optimization system2024