MBPP

Mostly Basic Python Problems — ~1,000 entry-level programming tasks with tests, for evaluating code generation.

Papers using MBPP (140)

A Survey on Large Language Models for Code Generation2024 · 59 citesCODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging2025 · 7 citesKodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding2025 · 6 citesKodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding2025 · 5 citesRegression Accumulation in Multi-Turn LLM Programming Conversations2026Type-Constrained Code Generation with Language Models2025 · 3 citesGrammar-Based Code Representation: Is It a Worthy Pursuit for LLMs?2025 · 3 citesPoison with Style: A Practical Poisoning Attack on Code Large Language Models2026Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points2025 · 3 citesEnhancing LLM-Based Code Generation with Complexity Metrics: A Feedback-Driven Approach2025 · 2 citesDynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation2025 · 2 citesLearning to Generate Unit Tests for Automated Debugging2025 · 2 citesQualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks2025 · 2 citesAutomated Repair of Ambiguous Problem Descriptions for LLM-Based Code Generation2025 · 1 citesOpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs2025 · 1 citesCODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging2025 · 1 citesCodeMirage: Hallucinations in Code Generated by Large Language Models2024 · 1 citesSePO: Self-Evolving Prompt Agent for System Prompt Optimization2026Using Semantic Distance to Estimate Uncertainty in LLM-Based Code Generation2026An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoning2026ACE: Self-Evolving LLM Coding Framework via Adversarial Unit Test Generation and Preference Optimization2026Prompt Optimization for LLM Code Generation via Reinforcement Learning2026Domain-Adaptable Reinforcement Learning for Code Generation with Dense Rewards2026Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedback2026Evaluating the Environmental Impact of using SLMs and Prompt Engineering for Code Generation2026FLeX: Fourier-based Low-rank EXpansion for multilingual transfer2026MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning2026ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning2026Think Anywhere in Code Generation2026TextBFGS: A Case-Based Reasoning Approach to Code Optimization via Error-Operator Retrieval2026BatCoder: Self-Supervised Bidirectional Code-Documentation Learning via Back-Translation2026Assessing and Improving the Representativeness of Code Generation Benchmarks Using Knowledge Units (KUs) of Programming Languages -- An Empirical Study2026NOIR: Privacy-Preserving Generation of Code with Open-Source LLMs2026Multi-task Code LLMs: Data Mix or Model Merge?2026Adaptive Confidence Gating in Multi-Agent Collaboration for Efficient and Optimized Code Generation2026Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference2026Shape of Thought: When Distribution Matters More than Correctness in Reasoning Tasks2025Summary-Mediated Repair: Can LLMs use code summarisation as a tool for program repair?2025ContractEval: A Benchmark for Evaluating Contract-Satisfying Assertions in Code Generation2025AdapTrack: Constrained Decoding without Distorting LLM's Output Intent2025SALT4Decompile: Inferring Source-level Abstract Logic Tree for LLM-Based Binary Decompilation2025Program Synthesis via Test-Time Transduction2025Reinforcement Learning-Guided Chain-of-Draft for Token-Efficient Code Generation2025Protocode: Prototype-Driven Interpretability for Code Generation in LLMs2025Static Analysis as a Feedback Loop: Enhancing LLM-Generated Code Beyond Correctness2025Alignment with Fill-In-the-Middle for Enhancing Code Generation2025STEPWISE-CODEX-Bench: Evaluating Complex Multi-Function Comprehension and Fine-Grained Execution Reasoning2025Assessing Small Language Models for Code Generation: An Empirical Study with Benchmarks2025Dr. Boot: Bootstrapping Program Synthesis Language Models to Perform Repairing2025CREME: Robustness Enhancement of Code LLMs via Layer-Aware Model Editing2025MemoCoder: Automated Function Synthesis using LLM-Supported Agents2025Efficient Code LLM Training via Distribution-Consistent and Diversity-Aware Data Selection2025When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions2025AdaDec: A Uncertainty-Guided Lookahead Decoding Framework for LLM-Based Code Generation2025CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks2025Rethinking Repetition Problems of LLMs in Code Generation2025Self-Correcting Code Generation Using Small Language Models2025What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces2025Modularization is Better: Effective Code Generation with Modular Prompting2025Memorize or Generalize? Evaluating LLM Code Generation with Code Rewriting2025ACECODER: Acing Coder RL via Automated Test-Case Synthesis2025Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment2025UnitCoder: Scalable Iterative Code Synthesis with Unit Test Guidance2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation2025CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models2025Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement2025Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEval2025Pragmatic Reasoning improves LLM Code Generation2025CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation2025Planning-Driven Programming: A Large Language Model Programming Workflow2024Context-Augmented Code Generation Using Programming Knowledge Graphs2024Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'2024EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization2024 · 1 citesReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generation2024Code Llama: Open Foundation Models for Code2023 · 414 citesWizardCoder: Empowering Code Large Language Models with Evol-Instruct2023 · 84 citesTeaching Large Language Models to Self-Debug2023 · 76 citesCodeT: Code Generation with Generated Tests2022 · 65 citesThe Stack: 3 TB of permissively licensed source code2022 · 40 citesDebug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step2024 · 40 citesCYCLE: Learning to Self-Refine the Code Generation2024 · 29 citesProgram Synthesis with Large Language Models2021 · 28 citesLiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code2024 · 28 citesTowards AI-Assisted Synthesis of Verified Dafny Methods2024 · 26 citesInteractive Code Generation via Test-Driven User-Intent Formalization2022 · 24 citesPrompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code2023 · 23 citesMapCoder: Multi-Agent Code Generation for Competitive Problem Solving2024 · 23 citesOn Evaluating the Efficiency of Source Code Generated by LLMs2024 · 21 citesAgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation2023 · 20 citesStructured Chain-of-Thought Prompting for Code Generation2023 · 17 citesAceCoder: Utilizing Existing Code to Enhance Code Generation2023 · 16 citesFault-Aware Neural Code Rankers2022 · 13 citesMultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation2022 · 13 citesCrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion2023 · 11 citesSOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents2024 · 10 citesPythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs2024 · 9 citesInterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback2023 · 7 citesLarge Language Model-Aware In-Context Learning for Code Generation2023 · 7 citesOOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models2024 · 7 citesReCode: Robustness Evaluation of Code Generation Models2022 · 6 citesNExT: Teaching Large Language Models to Reason about Code Execution2024 · 6 citesOpenCodeInterpreter: Integrating Code Generation with Execution and Refinement2024 · 4 citesSoftware Vulnerability and Functionality Assessment using LLMs2024 · 4 citesLeTI: Learning to Generate from Textual Interactions2023 · 2 citesEnhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency2023 · 2 citesCodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules2023 · 2 citesXFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts2024 · 2 citesNaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts2024 · 2 citesMHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation2024 · 2 citesRLTF: Reinforcement Learning from Unit Test Feedback2023 · 1 citesThe Program Testing Ability of Large Language Models for Code2023 · 1 citesDivide-and-Conquer Meets Consensus: Unleashing the Power of Functions in Code Generation2024 · 1 citesPlanning In Natural Language Improves LLM Search For Code Generation2024 · 1 citesSelection of Prompt Engineering Techniques for Code Generation through Predicting Code Complexity2024 · 1 citesTraining Language Models on Synthetic Edit Sequences Improves Code Synthesis2024 · 1 citesCodeTree: Agent-guided Tree Search for Code Generation with Large Language Models2024 · 1 citesPerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback2024 · 1 citesAlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement2024 · 1 citesDecoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Data2023Instruction Fusion: Advancing Prompt Evolution through Hybridization2023Unsupervised Evaluation of Code LLMs with Round-Trip Correctness2024DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuning2024Test-Driven Development for Code Generation2024Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective2024InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models2024Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting2024$\textbf{PLUM}$: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Cases2024Code-Optimise: Self-Generated Preference Data for Correctness and Efficiency2024Brevity is the soul of wit: Pruning long files for code generation2024InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct2024Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer2024$\mathbb{USCD}$: Improving Code Generation of LLMs by Uncertainty-Aware Selective Contrastive Decoding2024Self-Explained Keywords Empower Large Language Models for Code Generation2024FALCON: Feedback-driven Adaptive Long/short-term memory reinforced Coding Optimization system2024Demo-Craft: Using In-Context Learning to Improve Code Generation in Large Language Models2024DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs2024HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation2024Policy Filtration in RLHF to Fine-Tune LLM for Code Generation2024Can Language Models Replace Programmers? REPOCOD Says 'Not Yet'2024