SWE-bench Verified

Dataset Summary SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality. SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. See this post for more details on the human-validation process. The dataset collects 500 test I

Papers using SWE-bench Verified (6)

HarnessBridge: Learnable Bidirectional Controller for LLM Agent Harness2026Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents2026The Limits of Long-Context Reasoning in Automated Bug Fixing2026Self-Abstraction from Grounded Experience for Plan-Guided Policy Refinement2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution2025Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs2025