SWE-bench

Dataset Summary SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 2,294 Issue-Pull Request pairs from 12 popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution. The dataset was released a

Papers using SWE-bench (9)

Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning2025 · 21 citesTerminus-4B: Can a Smaller Model Replace Frontier LLMs at Agentic Execution Tasks?2026BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models2026Orchard: An Open-Source Agentic Modeling Framework2026A Rubric-Supervised Critic from Sparse Real-World Outcomes2026Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization2026SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent2025SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints2025Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering2025