SWE-bench
Emerging11papers using it
2024first seen
SWE-bench is a benchmark dataset used to evaluate the performance of Software Engineering agents, specifically comparing the effectiveness of observation masking versus LLM summarization in managing context histories during complex task-solving.
Papers using SWE-bench (11)
- PACE: A Proxy for Agentic Capability EvaluationLearn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in
Realistic EnvironmentsFeatureBench: Benchmarking Agentic Coding for Complex Feature DevelopmentAgents Explore but Agents Ignore: LLMs Lack Environmental CuriosityConsistency Amplifies: How Behavioral Variance Shapes Agent AccuracyTo Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program RepairThe Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context ManagementSWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering TasksImpossibleBench: Measuring LLMs' Propensity of Exploiting Test CasesSWE-bench Goes Live!SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications