HumanEval

HumanEval-X is a benchmark for the evaluation of the multilingual ability of code generative models. It consists of 820 high-quality human-crafted data samples (each with test cases) in Python, C++, Java, JavaScript, and Go, and can be used for various tasks.

Papers using HumanEval (15)

Smarter Saboteurs, Better Fixers: Scaling & Security in Linear Multi-Agent Workflows2026Honest Lying: Understanding Memory Confabulation in Reflexive Agents2026G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks2024 · 1 citesSafesieve: From Heuristics To Experience In Progressive Pruning For Llm-based Multi-agent Communication2025OmniCode: A Benchmark for Evaluating Software Engineering Agents2026TDPGen: Optimizing Agentic Code Generation via Test-Driven Planning and Hierarchical ReAct2026R2V Agent: Teaching SLMs When to Ask for Help2026Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration2026CARD: Towards Conditional Design of Multi-agent Topological Structures2026Adaptive Confidence Gating In Multi-agent Collaboration For Efficient And Optimized Code Generation2026MAR:Multi-Agent Reflexion Improves Reasoning Abilities in LLMs2025A Multi-Agent Framework for Stateful Inference-Time Search2025AgentGroupChat-V2: Divide-and-Conquer Is What LLM-Based Multi-Agent System Need2025Nexus: A Lightweight and Scalable Multi-Agent Framework for Complex Tasks Automation2025Guided Code Generation with LLMs: A Multi-Agent Framework for Complex Code Tasks2025