SWE-bench Verified

Dataset Summary SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality. SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. See this post for more details on the human-validation process. The dataset collects 500 test I

Papers using SWE-bench Verified (16)

LLM-as-a-Verifier: A General-Purpose Verification Framework2026Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality2026SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution2025 · 2 citesLean4Agent: Formal Modeling and Verification for Agent Workflow and Trajectory2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents2026HarnessBridge: Learnable Bidirectional Controller for LLM Agent Harness2026Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment2026SWE-World: Building Software Engineering Agents in Docker-Free Environments2026Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents2026Scaling Test-Time Compute for Agentic Coding2026Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills2026Decentralized Multi-Agent Systems with Shared Context2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning2026The Limits of Long-Context Reasoning in Automated Bug Fixing2026Self-Abstraction from Grounded Experience for Plan-Guided Policy Refinement2025Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs2025