SWE-Bench: The Real-World Software Engineering Benchmark

What SWE-Bench Tests

SWE-Bench (Software Engineering Benchmark) presents AI systems with real GitHub issues from popular open-source Python repositories and asks them to write code patches that fix those issues, verified by running the repository's existing test suite. Unlike HumanEval, which tests isolated function generation from a clean specification, SWE-Bench requires understanding a real codebase, interpreting an ambiguous bug report written by a human, and producing a patch that integrates cleanly with existing code, the actual workflow of software engineering.

The benchmark was introduced by Carlos Jimenez, John Yang, and colleagues from Princeton and other institutions, published in 2024. It sources issues from well-maintained Python repositories including Django, Flask, NumPy, SciPy, Matplotlib, Scikit-learn, and others, projects with large, complex codebases that have been actively maintained for years. Each task provides the issue description, the full repository at the time of the issue, and a set of tests that the correct fix must pass. Validation is objective: the patch either makes the tests pass or it doesn't.

How SWE-Bench Was Constructed

SWE-Bench's 2,294 tasks were collected by scraping resolved GitHub issues from twelve popular Python repositories, filtering to issues that were resolved with a single commit and that had associated test changes confirming the fix. The test-confirmed resolution criterion is critical: it means there's an objective, executable ground truth for what counts as a correct solution, the tests that were written to verify the fix must pass.

The collection pipeline had to handle significant noise: many GitHub issues are vague, some resolutions are non-obvious from the issue text, and some fixes required understanding weeks of repository context. This real-world messiness is intentional, SWE-Bench deliberately tests the ability to navigate ambiguous, under-specified engineering tasks, because that's what real software engineering looks like.

A subset of the original benchmark was found to have issues with test reliability and problem specification quality. This led to SWE-Bench Verified, a human-validated subset of 500 problems confirmed by professional software engineers to be: correctly specified, solvable from the provided information, and reliably tested. SWE-Bench Verified is now the standard evaluation subset used by AI labs and researchers.

SWE-Bench Verified: The Trusted Evaluation Set

SWE-Bench Verified (released in 2024) is a 500-problem subset where professional software engineers manually reviewed every task to confirm: the issue is clearly described, the expected fix is unambiguous, the verification tests reliably distinguish correct from incorrect solutions, and the task is solvable without additional context not provided in the repository. Tasks that failed any of these criteria were excluded.

The validation process found that roughly 20-25% of original SWE-Bench tasks had quality issues, tests that were flaky, issue descriptions too vague to determine correct behavior, or resolutions that depended on information outside the repository. SWE-Bench Verified's higher quality makes scores more reliable and more comparable across evaluation runs. When a lab reports 'SWE-Bench' scores without specifying Verified, the result may include some unreliable tasks and should be treated with additional skepticism.

What SWE-Bench Scores Mean

SWE-Bench scores are reported as the percentage of tasks where the model's patch makes all relevant tests pass. Because the tasks are drawn from real repositories with complex codebases, this is a genuinely hard problem, early attempts by frontier models (GPT-4, Claude 2) achieved only 1-4% on the full benchmark. The rapid improvement since then is one of the most striking trajectories in AI capability research.

Score interpretation: below 20% means the system struggles with most realistic engineering tasks. 20-40% represents meaningful capability on straightforward issues in familiar codebases. 40-60% represents strong practical engineering assistance, resolving a majority of typical issues without human intervention. Above 60% represents state-of-the-art agentic software engineering capability. Note that 'tasks' vary in difficulty; aggregate scores blend easy and hard tasks, and detailed analysis should examine which problem types each system handles well.

Why Agent Systems Dominate SWE-Bench

Single-shot model calls cannot solve SWE-Bench at high rates. The task requires multiple steps: understanding the issue, exploring the codebase to find relevant files, reading those files, forming a hypothesis, writing a patch, running tests, and iterating based on results. Top SWE-Bench systems are agent systems, they give the underlying model access to tools (file reading, code execution, test running, web search) and let it work through the problem iteratively, often over many minutes.

Top-performing systems as of mid-2025 include SWE-agent (Princeton), Devin (Cognition), OpenHands (All-Hands AI), and proprietary systems from the major AI labs. These systems typically use a combination of: exploratory code navigation to understand the codebase structure, iterative patch-and-test loops, and in some cases external retrieval to identify similar past fixes. The model at the center matters, but the scaffolding, tool design, agent loop structure, context management, often matters as much or more.

This means SWE-Bench measures system capability, not just model capability. A GPT-4o instance embedded in an excellent agent scaffold may outperform a more capable base model in a naive scaffold. When comparing SWE-Bench scores across different systems, check whether they're using comparable scaffolding or whether one system has a significant architectural advantage.

Current Leaderboard

As of mid-2025 on SWE-Bench Verified: leading agent systems achieve 60-72% resolution rates. Claude Sonnet 4.6 in an agent framework reaches approximately 72%, OpenHands with Claude Opus 4 reaches around 65%, and SWE-agent with o3 reaches around 62%. These numbers have increased dramatically from early 2024, when the best systems scored 20-30%. The improvement reflects both better underlying models and significantly more sophisticated agent scaffolding.

A useful reference point: a skilled human engineer resolves most of the SWE-Bench Verified tasks when given sufficient time, the tasks are solvable. The question is how reliably AI systems can match that performance with limited compute and time budgets. At 60-72% resolution, top systems are genuinely useful for automated issue resolution on real codebases, though they still fail on the hardest, most architecturally complex problems.

Limitations and What SWE-Bench Reveals

SWE-Bench's Python focus is its main scope limitation, software engineering spans many languages, and Python's dynamic nature, rich tooling, and clear error messages may make it more accessible to AI systems than lower-level languages like C++ or Rust. Analogous benchmarks for other languages and for multi-language systems are being developed, but Python remains the standard for now.

What SWE-Bench reveals practically: for teams building AI-assisted software engineering tools, SWE-Bench Verified scores are the most predictive available signal for whether a system will provide real value on real code. HumanEval and MBPP scores correlate poorly with SWE-Bench, a model that tops HumanEval may still struggle with real issues. If your use case involves automated debugging, pull request generation, or autonomous issue resolution, SWE-Bench Verified is the benchmark to prioritize in model selection.