HumanEval

HumanEval-X is a benchmark for the evaluation of the multilingual ability of code generative models. It consists of 820 high-quality human-crafted data samples (each with test cases) in Python, C++, Java, JavaScript, and Go, and can be used for various tasks.

Papers using HumanEval (20)

ACECODER: Acing Coder RL via Automated Test-Case Synthesis2025 · 3 citesGroup-Aware Reinforcement Learning for Output Diversity in Large Language Models2025 · 1 citesWarm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings2025 · 1 citesSLIM-RL: Risk-Budgeted Random-Masking RL for Diffusion LLMs Without Trajectory Slicing2026LEMON: Learning Executable Multi-Agent Orchestration via Counterfactual Reinforcement Learning2026PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play2026Beyond Execution: Static-Analysis Rewards and Hint-Conditioned Diffusion RL for Code Generation2026Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo2026MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning2026ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning2026Execution-Grounded Credit Assignment for GRPO in Code Generation2026Reasoning with Sampling: Your Base Model is Smarter Than You Think2025CosmoCore Affective Dream-Replay Reinforcement Learning for Code Generation2025Learning to Reason as Action Abstractions with Scalable Mid-Training RL2025Prior Prompt Engineering for Reinforcement Fine-Tuning2025Optimal Policy Minimum Bayesian Risk2025FALCON: Feedback-driven Adaptive Long/short-term memory reinforced Coding Optimization system2024RLHF Workflow: From Reward Modeling to Online RLHF2024 · 3 citesPolicy Filtration for RLHF to Mitigate Noise in Reward Models2024CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks2024