Benchmark Almanac

SWE-PolyBench

v20250402 22.6% tops its leaderboard Claude 3.5 Sonnet (October 2024) 14.1% leads the models we track Not in model cards

Coding agents resolve 2,110 tasks from 21 repositories in Java, JavaScript, TypeScript and Python. Tasks cover bug fixes, features and refactors, checked by execution.

A stratified 500-task subset (SWE-PolyBench500) is common.

In the authors’ words

Coding agents powered by large language models have shown impressive capabilities in software engineering tasks, but evaluating their performance across diverse programming languages and real-world scenarios remains challenging. We introduce SWE-PolyBench, a new multi-language benchmark for repository-level, execution-based evaluation of coding agents. SWE-PolyBench contains 2110 instances from 21 repositories and includes tasks in Java (165), JavaScript (1017), TypeScript (729) and Python (199), covering bug fixes, feature additions, and code refactoring. We provide a task and repository-stratified subsample (SWE-PolyBench500) and release an evaluation harness allowing for fully automated evaluation. To enable a more comprehensive comparison of coding agents, this work also presents a novel set of metrics rooted in syntax tree analysis. We evaluate leading open source coding agents on SWE-PolyBench, revealing their strengths and limitations across languages, task types, and complexity classes. Our experiments show that current agents exhibit uneven performances across languages and struggle with complex problems while showing higher performance on simpler tasks. SWE-PolyBench aims to drive progress in developing more versatile and robust AI coding assistants for real-world software engineering. Our datasets and code are available at: https://github.com/amazon-science/SWE-PolyBench

Abstract of SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents (arXiv:2504.08703, CC0)
Made by Amazon Released Apr 2025 Fixing real repositories All tasks public Last model added Jan 2025 Scores refreshed 2 Nov 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
amazon-science/SWE-PolyBench 89 stars Paper Leaderboard amazon-science.github.io
7
Entries on its own leaderboard, 6 models we track

v20250402 tops the leaderboard with 22.6%

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Claude 3.5 Sonnet (October 2024), at 14.1%.

The newest model with a score here came out in Jan 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 22.6 v20250402
  2. 14.1 Claude 3.5 Sonnet (October 2024)
  3. 11.5 DeepSeek-R1
  4. 9.86 Claude 3.5 Haiku
  5. 6.02 Llama 3.3 70B
  6. 5.88 Mistral Large 2
  7. 5.31 DeepSeek-R1-Distill-Llama-70B
  1. 1 v20250402 22.6%
  2. 2 Claude 3.5 Sonnet (October 2024) 14.1%
  3. 3 DeepSeek-R1 11.5%
  4. 4 Claude 3.5 Haiku 9.86%
  5. 5 Llama 3.3 70B 6.02%
  6. 6 Mistral Large 2 Nov 2024 5.88%
  7. 7 DeepSeek-R1-Distill-Llama-70B 5.31%
#EntryScoreSettingDateSource
1 v20250402 22.6% Amazon Q Developer Agent Apr 2025 Official leaderboard
2 Claude 3.5 Sonnet (October 2024) 14.1% Aider-PB Oct 2024 Official leaderboard
3 DeepSeek-R1 11.5% Aider-PB Jan 2025 Official leaderboard
4 Claude 3.5 Haiku 9.86% Aider-PB Oct 2024 Official leaderboard
5 Llama 3.3 70B 6.02% Aider-PB Dec 2024 Official leaderboard
6 Mistral Large 2 Nov 2024 5.88% Aider-PB Nov 2024 Official leaderboard
7 DeepSeek-R1-Distill-Llama-70B 5.31% Aider-PB Jan 2025 Official leaderboard

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard Not tested since Apr 2025
Claude 3.5 Sonnet (October 2024) 14.1%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Claude 3.5 Sonnet (October 2024) 0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Claude 3.5 Sonnet (October 2024)

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Coding

All 243 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Opus 5.5 93.9%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471