Benchmark Almanac

SWE-bench Multilingual

Claude Opus 5.5 93.9% leads since Sep 2026 In use

300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests.

Released with SWE-smith. Paper link is the SWE-smith paper.

Made by SWE-bench team (Princeton/Stanford) Released Apr 2025 Fixing real repositories All tasks public Last in a launch post Oct 2026 Last model added Oct 2026 Scores refreshed 2 Nov 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Code A program or patch run against tests.
SWE-bench/SWE-bench 6,129 stars Paper Dataset Leaderboard
70
Models scored in lab cards
63
Launch posts list it, from 24 labs
36
System cards report it, from 14 labs

Claude Opus 5.5 leads with 93.9%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

70 models. Scroll sideways for the rest.

  1. 93.9 Claude Opus 5.5
  2. 90.3 Claude Sonnet 5.5
  3. 89.5 Claude Opus 5
  4. 89.1 Claude Fable 5.1
  5. 88.5 Ling-3.1-flash
  6. 87.3 Claude Mythos Preview
  7. 84.4 Claude Opus 4.8
  8. 84.0 Intern-S2-397B
  9. 83.7 Claude Haiku 5.5
  10. 82.9 Tencent Hy4 preview
  11. 81.0 Qwen3.8 Flash
  12. 80.5 Claude Opus 4.7
  13. 80.5 Qwen3.8-Omni-Flash
  14. 79.8 Composer 2.5
  15. 78.5 Laguna S 2.1
  16. 78.3 Qwen3.7-Max
  17. 78.3 Claude Sonnet 5
  18. 78.0 Grok 4.5
  19. 78.0 Beam 1.0
  20. 77.8 Claude Opus 4.6
  21. 77.8 SWE-1.7
  22. 77.3 LongCat-2.0
  23. 76.7 Kimi K2.6
  24. 76.5 MiniMax-M2.7
  25. 76.2 Claude Opus 4.5
  26. 76.2 DeepSeek-V4-Pro
  27. 75.9 Claude Sonnet 4.6
  28. 75.8 Qwen3.7-Plus
  29. 75.8 Tencent Hy3
  30. 74.1 MiniMax-M2.5
  31. 73.8 Qwen 3.6 Plus
  32. 73.7 Composer 2
  33. 73.3 GLM-5
  34. 73.3 DeepSeek-V4-Flash
  35. 73.0 Kimi K2.5
  36. 72.5 MiniMax-M2.1
  37. 72.4 Step 3.7 Flash
  38. 71.7 MiMo-V2-Flash
  39. 71.7 Seed2.0 Pro
  40. 71.7 MiMo-V2-Pro
  41. 71.3 Qwen3.6 27B
  42. 70.2 DeepSeek-V3.2
  43. 69.3 Qwen3.5 397B-A17B
  44. 67.7 Nemotron 3.5 Super VL
  45. 67.7 Nemotron 3 Ultra
  46. 67.4 Step 3.5 Flash
  47. 67.3 Laguna M.1
  48. 67.2 Qwen 3.6 35B-A3B
  49. 66.7 GLM-4.7
  50. 66.6 Seed2.0 Lite (0428)
  51. 65.5 MAI-Code-1-Flash
  52. 64.7 Spark-X2.6-4B
  53. 63.1 Laguna XS 2.1
  54. 63.0 KAT-Coder-V2.5-Dev
  55. 62.8 qwen3-coder-next
  56. 62.4 Laguna XS.2
  57. 61.3 Devstral 2
  58. 61.1 Kimi K2 Thinking
  59. 57.9 DeepSeek-V3.2-Exp
  60. 57.8 DeepSeek-V3.1-Terminus
  61. 56.5 MiniMax-M2
  62. 55.9 Kimi K2
  63. 55.7 Devstral Small 2
  64. 54.7 Qwen3-Coder-480B-A35B
  65. 54.5 DeepSeek-V3.1
  66. 47.3 Kimi K2
  67. 45.8 Nemotron 3 Super
  68. 41.9 Granite 4.2 30B
  69. 39.3 Nemotron 3.5 Lightning 30B-A3B
  70. 23.0 LFM3-1.7B
  1. 1 Claude Opus 5.5 93.9%
  2. 2 Claude Sonnet 5.5 90.3%
  3. 3 Claude Opus 5 89.5%
  4. 4 Claude Fable 5.1 89.1%
  5. 5 Ling-3.1-flash 88.5%
  6. 6 Claude Mythos Preview 87.3%
  7. 7 Claude Opus 4.8 84.4%
  8. 8 Intern-S2-397B 84%
  9. 9 Claude Haiku 5.5 83.7%
  10. 10 Tencent Hy4 preview 82.9%
  11. 11 Qwen3.8 Flash 81%
  12. 12 Claude Opus 4.7 80.5%
  13. 13 Qwen3.8-Omni-Flash 80.5%
  14. 14 Composer 2.5 79.8%
  15. 15 Laguna S 2.1 78.5%
  16. 16 Qwen3.7-Max 78.3%
  17. 17 Claude Sonnet 5 78.3%
  18. 18 Grok 4.5 78%
  19. 19 Beam 1.0 78%
  20. 20 Claude Opus 4.6 77.8%
  21. 21 SWE-1.7 77.8%
  22. 22 LongCat-2.0 77.3%
  23. 23 Kimi K2.6 76.7%
  24. 24 MiniMax-M2.7 76.5%
  25. 25 Claude Opus 4.5 76.2%
  26. 26 DeepSeek-V4-Pro 76.2%
  27. 27 Claude Sonnet 4.6 75.9%
  28. 28 Qwen3.7-Plus 75.8%
  29. 29 Tencent Hy3 75.8%
  30. 30 MiniMax-M2.5 74.1%
  31. 31 Qwen 3.6 Plus 73.8%
  32. 32 Composer 2 73.7%
  33. 33 GLM-5 73.3%
  34. 34 DeepSeek-V4-Flash 73.3%
  35. 35 Kimi K2.5 73%
  36. 36 MiniMax-M2.1 72.5%
  37. 37 Step 3.7 Flash 72.4%
  38. 38 MiMo-V2-Flash 71.7%
  39. 39 Seed2.0 Pro 71.7%
  40. 40 MiMo-V2-Pro 71.7%
  41. 41 Qwen3.6 27B 71.3%
  42. 42 DeepSeek-V3.2 70.2%
  43. 43 Qwen3.5 397B-A17B 69.3%
  44. 44 Nemotron 3.5 Super VL 67.7%
  45. 45 Nemotron 3 Ultra 67.7%
  46. 46 Step 3.5 Flash 67.4%
  47. 47 Laguna M.1 67.3%
  48. 48 Qwen 3.6 35B-A3B 67.2%
  49. 49 GLM-4.7 66.7%
  50. 50 Seed2.0 Lite (0428) 66.6%
  51. 51 MAI-Code-1-Flash 65.5%
  52. 52 Spark-X2.6-4B 64.7%
  53. 53 Laguna XS 2.1 63.1%
  54. 54 KAT-Coder-V2.5-Dev 63%
  55. 55 qwen3-coder-next 62.8%
  56. 56 Laguna XS.2 62.4%
  57. 57 Devstral 2 61.3%
  58. 58 Kimi K2 Thinking 61.1%
  59. 59 DeepSeek-V3.2-Exp 57.9%
  60. 60 DeepSeek-V3.1-Terminus 57.8%
  61. 61 MiniMax-M2 56.5%
  62. 62 Kimi K2 Sep 2025 55.9%
  63. 63 Devstral Small 2 55.7%
  64. 64 Qwen3-Coder-480B-A35B 54.7%
  65. 65 DeepSeek-V3.1 54.5%
  66. 66 Kimi K2 Jul 2025 47.3%
  67. 67 Nemotron 3 Super 45.8%
  68. 68 Granite 4.2 30B 41.9%
  69. 69 Nemotron 3.5 Lightning 30B-A3B 39.3%
  70. 70 LFM3-1.7B 23%
#ModelScoreSettingDateSource
1 Claude Opus 5.5 93.9% Sep 2026 Model card
2 Claude Sonnet 5.5 90.3% Sep 2026 Model card
3 Claude Opus 5 89.5% Jul 2026 Model card
4 Claude Fable 5.1 89.1% Sep 2026 Model card
5 Ling-3.1-flash 88.5% Oct 2026 Model card
6 Claude Mythos Preview 87.3% Apr 2026 Model card
7 Claude Opus 4.8 84.4% May 2026 Model card
8 Intern-S2-397B 84% Sep 2026 Model card
9 Claude Haiku 5.5 83.7% Oct 2026 Model card
10 Tencent Hy4 preview 82.9% Aug 2026 Model card
11 Qwen3.8 Flash 81% Aug 2026 Model card
12 Claude Opus 4.7 80.5% Apr 2026 Model card
13 Qwen3.8-Omni-Flash 80.5% Sep 2026 Model card
14 Composer 2.5 79.8% May 2026 Model card
15 Laguna S 2.1 78.5% Jul 2026 Model card
16 Qwen3.7-Max 78.3% May 2026 Model card
17 Claude Sonnet 5 78.3% Jun 2026 Model card
18 Grok 4.5 78% Jul 2026 Model card
19 Beam 1.0 78% Oct 2026 Model card
20 Claude Opus 4.6 77.8% Feb 2026 Model card
21 SWE-1.7 77.8% Jul 2026 Model card
22 LongCat-2.0 77.3% Jul 2026 Model card
23 Kimi K2.6 76.7% Apr 2026 Model card
24 MiniMax-M2.7 76.5% Mar 2026 Model card
25 Claude Opus 4.5 76.2% Nov 2025 Model card
26 DeepSeek-V4-Pro 76.2% Apr 2026 Model card
27 Claude Sonnet 4.6 75.9% Feb 2026 Model card
28 Qwen3.7-Plus 75.8% Jun 2026 Model card
29 Tencent Hy3 75.8% Jul 2026 Model card
30 MiniMax-M2.5 74.1% Feb 2026 Model card
31 Qwen 3.6 Plus 73.8% Apr 2026 Model card
32 Composer 2 73.7% Mar 2026 Model card
33 GLM-5 73.3% Feb 2026 Model card
34 DeepSeek-V4-Flash 73.3% Apr 2026 Model card
35 Kimi K2.5 73% Jan 2026 Model card
36 MiniMax-M2.1 72.5% Dec 2025 Model card
37 Step 3.7 Flash 72.4% May 2026 Model card
38 MiMo-V2-Flash 71.7% Dec 2025 Model card
39 Seed2.0 Pro 71.7% Feb 2026 Model card
40 MiMo-V2-Pro 71.7% Mar 2026 Model card
41 Qwen3.6 27B 71.3% Apr 2026 Model card
42 DeepSeek-V3.2 70.2% Dec 2025 Model card
43 Qwen3.5 397B-A17B 69.3% Feb 2026 Model card
44 Nemotron 3.5 Super VL 67.7% Oct 2026 Model card
45 Nemotron 3 Ultra 67.7% Jun 2026 Model card
46 Step 3.5 Flash 67.4% Feb 2026 Model card
47 Laguna M.1 67.3% Apr 2026 Model card
48 Qwen 3.6 35B-A3B 67.2% Apr 2026 Model card
49 GLM-4.7 66.7% Dec 2025 Model card
50 Seed2.0 Lite (0428) 66.6% Apr 2026 Model card
51 MAI-Code-1-Flash 65.5% Jun 2026 Model card
52 Spark-X2.6-4B 64.7% Oct 2026 Model card
53 Laguna XS 2.1 63.1% Jul 2026 Model card
54 KAT-Coder-V2.5-Dev 63% Jul 2026 Model card
55 qwen3-coder-next 62.8% Feb 2026 Model card
56 Laguna XS.2 62.4% Apr 2026 Model card
57 Devstral 2 61.3% Dec 2025 Model card
58 Kimi K2 Thinking 61.1% Nov 2025 Model card
59 DeepSeek-V3.2-Exp 57.9% Sep 2025 Model card
60 DeepSeek-V3.1-Terminus 57.8% Sep 2025 Model card
61 MiniMax-M2 56.5% Oct 2025 Model card
62 Kimi K2 Sep 2025 55.9% Sep 2025 Model card
63 Devstral Small 2 55.7% Dec 2025 Model card
64 Qwen3-Coder-480B-A35B 54.7% Jul 2025 Model card
65 DeepSeek-V3.1 54.5% Aug 2025 Model card
66 Kimi K2 Jul 2025 47.3% Jul 2025 Model card
67 Nemotron 3 Super 45.8% Mar 2026 Model card
68 Granite 4.2 30B 41.9% Aug 2026 Model card
69 Nemotron 3.5 Lightning 30B-A3B 39.3% Aug 2026 Model card
70 LFM3-1.7B 23% Oct 2026 Model card

The benchmark's own leaderboard has 13 models. We plot every source in the charts below.

The best score, 72.7%, costs $0.35 a task

Runs on the cost frontier score higher than any run that's cheaper. MiniMax-M2.5 comes within 4.4 points of the best for $0.10 a task, 3 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI Google DeepMind MiniMax Cost frontier One model at each effort level
0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 Cost per task, log scale Gemini 3 Flash MiniMax-M2.5 GPT-5 mini 0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 Cost per task, log scale Gemini 3 Flash GPT-5 mini
ModelEffortHarnessScoreCost per taskOutput tokens
Gemini 3 Flash mini-SWE-agent 72.7% $0.35
Claude Opus 4.6 mini-SWE-agent 72% $0.66
Claude Opus 4.5 mini-SWE-agent 70.7% $0.83
GLM-5 mini-SWE-agent 69.7% $0.64
Gemini 3 Pro mini-SWE-agent 68.7% $1.02
MiniMax-M2.5 mini-SWE-agent 68.3% $0.10
Kimi K2.5 mini-SWE-agent 67.3% $0.69
Claude Sonnet 4.5 mini-SWE-agent 67% $0.67
GPT-5.2 high mini-SWE-agent 66.7% $0.54
GPT-5.2 Codex mini-SWE-agent 66.3% $0.66
Claude Haiku 4.5 mini-SWE-agent 64.7% $0.38
DeepSeek-V3.2 mini-SWE-agent 59% $0.38
GPT-5 mini mini-SWE-agent 39.7% $0.05

5 models have held number one since its release in Apr 2025

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Anthropic Google DeepMind Moonshot AI Qwen DeepSeek
Own leaderboard Not tested since Feb 2026
Gemini 3 Flash 72.7%
Lab claims Passed 85% in Apr 2026, by Claude Mythos Preview
Claude Opus 5.5 93.9%

The best score passed 85% in Apr 2026

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Claude Opus 5.5 Claude Opus 5 Claude Mythos Preview Claude Opus 4.6 Gemini 3 Pro DeepSeek-V3.1-Terminus Kimi K2 0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Claude Opus 5.5 Claude Mythos Preview

15 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Moonshot AI
Qwen
DeepSeek
MiniMax
Mistral AI
Xiaomi Corp
Z.ai
ByteDance Seed
NVIDIA
Cursor
Anthropic
Poolside
StepFun
Microsoft
Tencent
Meituan
Cognition
Kuaishou
IBM
Shanghai AI Lab
Reflection
Ant Group
iFlytek
Liquid AI
DateLaunch postScoreAs the post writes it
27 Oct 2026 LFM3-1.7B 23 SWE-bench Multilingual (OpenCode)
22 Oct 2026 Spark-X2.6-4B 64.7 SWE Multilingual
15 Oct 2026 Nemotron 3.5 Super VL 67.7 SWE-bench Multilingual (OpenCode harness)
13 Oct 2026 Ling-3.1-flash 88.5 SWE-bench Multilingual
7 Oct 2026 Claude Haiku 5.5 83.7 SWE-bench Multilingual
5 Oct 2026 Beam 1.0 78 SWEBench Multilingual
28 Sep 2026 Claude Sonnet 5.5 90.3 SWE-Bench Multilingual
22 Sep 2026 Claude Opus 5.5 93.9 SWE-bench Multilingual
18 Sep 2026 Qwen3.8-Omni-Flash 80.5 SWE-bench Multilingual (mini-SWE-agent)
13 Sep 2026 Intern-S2-397B 84 SWE-bench-Multilingual (Mini-SWE-Agent)
1 Sep 2026 Claude Fable 5.1 89.1 SWE-bench Multilingual
28 Aug 2026 Tencent Hy4 preview 82.9 SWE-bench Multilingual
26 Aug 2026 Qwen3.8 Flash 81 SWE-bench Multilingual (mini-SWE-agent)
25 Aug 2026 Granite 4.2 30B 41.9 SWE Bench Multilingual
11 Aug 2026 Nemotron 3.5 Lightning 30B-A3B 39.3 SWE-bench Multilingual
24 Jul 2026 Claude Opus 5 89.5 SWE-bench Multilingual
23 Jul 2026 KAT-Coder-V2.5-Dev 63 SWE-bench Multilingual
21 Jul 2026 Laguna S 2.1 78.5 SWE-Bench Multilingual
8 Jul 2026 SWE-1.7 77.8 SWE-Bench Multilingual (Devin CLI)
5 Jul 2026 LongCat-2.0 77.3 SWE-bench Multilingual
2 Jul 2026 Laguna XS 2.1 63.1 SWE-bench Multilingual
2 Jul 2026 Tencent Hy3 75.8 SWE-bench Multilingual
4 Jun 2026 Nemotron 3 Ultra 67.7 SWE-Bench Multilingual
2 Jun 2026 MAI-Code-1-Flash 65.5 SWE-Bench Multilingual
1 Jun 2026 Qwen3.7-Plus 75.8 SWE-Multilingual
29 May 2026 Step 3.7 Flash 72.4 SWE-MTLG
28 May 2026 Claude Opus 4.8 84.4 SWE-bench Multilingual
20 May 2026 Qwen3.7-Max 78.3 SWE-Multilingual
18 May 2026 Composer 2.5 79.8 SWE-Bench Multilingual
28 Apr 2026 Laguna XS.2 62.4 SWE-bench Multilingual (mean pass@1 over 7 runs)
28 Apr 2026 Laguna M.1 67.3 SWE-bench Multilingual (mean pass@1 over 7 runs)
28 Apr 2026 Seed2.0 Lite (0428) 66.6 SWE Multilingual
24 Apr 2026 DeepSeek-V4-Pro 76.2 SWE Multilingual (Resolved)
22 Apr 2026 Qwen3.6 27B 71.3 SWE-bench Multilingual
20 Apr 2026 Kimi K2.6 76.7 SWE-Bench Multilingual
16 Apr 2026 Claude Opus 4.7 80.5 SWE-bench Multilingual
15 Apr 2026 Qwen 3.6 35B-A3B 67.2 SWE-bench Multilingual
7 Apr 2026 Claude Mythos Preview 87.3 SWE-bench Multilingual
2 Apr 2026 Qwen 3.6 Plus 73.8 SWE-bench Multilingual
19 Mar 2026 Composer 2 73.7 SWE-bench Multilingual
18 Mar 2026 MiMo-V2-Pro 71.7 SWE-bench Multilingual
18 Mar 2026 MiniMax-M2.7 76.5 SWE Multilingual (text)
11 Mar 2026 Nemotron 3 Super 45.8 SWE-Bench Multilingual (OpenHands)
16 Feb 2026 Qwen3.5 397B-A17B 69.3 SWE-bench Multilingual
14 Feb 2026 Seed2.0 Pro 71.7 SWE Multilingual
12 Feb 2026 MiniMax-M2.5 74.1 SWE-Bench Multilingual
11 Feb 2026 GLM-5 73.3 SWE-bench Multilingual
2 Feb 2026 qwen3-coder-next 62.8 SWE-Bench Multilingual (w/ SWE-Agent)
27 Jan 2026 Kimi K2.5 73 SWE-Bench Multilingual
23 Dec 2025 MiniMax-M2.1 72.5 SWE-bench Multilingual
22 Dec 2025 GLM-4.7 66.7 SWE-bench Multilingual
16 Dec 2025 MiMo-V2-Flash 71.7 SWE-Bench Multilingual
9 Dec 2025 Devstral Small 2 55.7 SWE Bench Multilingual (HF card)
9 Dec 2025 Devstral 2 61.3 SWE Bench Multilingual (HF card)
1 Dec 2025 DeepSeek-V3.2 70.2 SWE Multilingual (Resolved)
6 Nov 2025 Kimi K2 Thinking 61.1 SWE-bench Multilingual, w/ tools
27 Oct 2025 MiniMax-M2 56.5 SWE-bench Multilingual
29 Sep 2025 DeepSeek-V3.2-Exp 57.9 SWE-bench Multilingual
22 Sep 2025 DeepSeek-V3.1-Terminus 57.8 SWE-bench Multilingual
5 Sep 2025 Kimi K2 55.9 SWE-Bench Multilingual, ACC
21 Aug 2025 DeepSeek-V3.1 54.5 SWE-bench Multilingual (Agent mode)
22 Jul 2025 Qwen3-Coder-480B-A35B 54.7 SWE-bench Multilingual
11 Jul 2025 Kimi K2 47.3 SWE-bench Multilingual (Agentic Coding), Single Attempt (Acc)

36 system cards report it, from 14 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
15 Oct 2026 Nemotron 3.5 Super VL technical report 67.7 SWE-bench Multilingual, OpenCode harness (Table 10) Capabilities
15 Oct 2026 Nemotron 3.5 Lightning 30B-A3B technical report 37.1 SWE-Bench Multilingual, BF16 acc. (Table 22) Capabilities
7 Oct 2026 Claude Haiku 5.5 system card 83.7 SWE-bench Multilingual, average of five trials, max effort (Table 8.1.A) Capabilities
28 Sep 2026 Claude Sonnet 5.5 system card 90.3 SWE-Bench Multilingual, average of five trials (Table 8.1.A) Capabilities
22 Sep 2026 Qwen3.8-Omni-Flash technical report 80.5 SWE-bench Multilingual, mini-SWE-agent (Table 2) Capabilities
22 Sep 2026 Claude Opus 5.5 system card 93.9 SWE-bench Multilingual, average of five trials, max effort (Table 8.1.A, 8.2) Capabilities
1 Sep 2026 Claude Fable 5.1 system card 89.1 SWE-bench Multilingual (Table 8.1.A) Capabilities
24 Jul 2026 Claude Opus 5 system card 89.5 SWE-bench Multilingual, average of five trials (Table 8.1.A, 8.2) Capabilities
14 Jul 2026 Grok 4.5 model card 78 SWE-Bench Multilingual, resolve rate, high, runs reported by Cursor (2.5) Capabilities
30 Jun 2026 Claude Sonnet 5 system card 78.3 SWE-bench Multilingual, average of five trials (8.2) Capabilities
9 Jun 2026 Nemotron 3 Ultra technical report 67.7 SWE-Bench Multilingual (Table 10) Capabilities
2 Jun 2026 MAI-Code-1-Flash model card 65.5 SWE-Bench Multilingual, pass rate, Copilot production harness (Coding table) Capabilities
28 May 2026 Claude Opus 4.8 system card 84.4 SWE-bench Multilingual, average of five trials, max effort (Table 8.1.A, 8.2) Capabilities
26 May 2026 MiniMax-M2 technical report 56.5 SWE-bench Multilingual, MiniMax-M2 bar, printed label (Figure 9) Capabilities
26 May 2026 MiniMax-M2.7 technical report 76.5 SWE-bench Multilingual, Claude Code scaffold, avg of 4 trials (Table 4) Capabilities
26 May 2026 MiniMax-M2.5 technical report 74.1 SWE-bench Multilingual, Claude Code scaffold, avg of 4 trials (Table 4) Capabilities
25 May 2026 Laguna XS.2 technical report 57.7 SWE-bench Multilingual, pool harness, 500 steps, mean pass@1 over 4 runs, some task images and verifiers patched (Table 6) Capabilities
25 May 2026 Laguna M.1 technical report 63.1 SWE-bench Multilingual, pool harness, 500 steps, mean pass@1 over 4 runs, some task images and verifiers patched (Table 7) Capabilities
26 Apr 2026 DeepSeek-V4-Pro technical report 76.2 SWE Multilingual (Resolved), DS-V4-Pro Max (Table 6) Capabilities
26 Apr 2026 DeepSeek-V4-Flash technical report 73.3 SWE Multilingual (Resolved), DeepSeek-V4-Flash Max (Table 7) Capabilities
16 Apr 2026 Claude Opus 4.7 system card 80.5 SWE-bench Multilingual, average of 5 trials (Table 8.1.A, 8.2) Capabilities
7 Apr 2026 Claude Mythos Preview system card 87.3 SWE-bench Multilingual, adaptive thinking, max effort, average of 5 trials (Table 6.3.A, 6.4) Capabilities
25 Mar 2026 Composer 2 technical report 73.7 SWE-bench Multi., Cursor's own harness (Table 1 and 7.2) Capabilities
11 Mar 2026 Nemotron 3 Super technical report 45.8 SWE-Bench Multilingual (OpenHands) (Table 5) Capabilities
28 Feb 2026 qwen3-coder-next technical report 62.8 SWE-Bench Multilingual, SWE-Agent. MiniSWE-Agent 56.2, OpenHands 64.3 (Table 4) Capabilities
17 Feb 2026 GLM-5 technical report 73.3 SWE-bench Multilingual, OpenHands (Table 7) Capabilities
17 Feb 2026 Claude Sonnet 4.6 system card 75.9 SWE-bench Multilingual, adaptive thinking, max effort (2.2) Capabilities
14 Feb 2026 Seed2.0 Pro model card 71.7 SWE Multilingual (Table 11) Capabilities
11 Feb 2026 Step 3.5 Flash technical report 67.4 SWE Multilingual (Table 5) Capabilities
5 Feb 2026 Claude Opus 4.6 system card 77.8 SWE-bench Multilingual, adaptive thinking, max effort, average of 25 trials (2.4) Capabilities
30 Jan 2026 Kimi K2.5 technical report 73 SWE-Bench Multilingual (Table 4) Capabilities
6 Jan 2026 MiMo-V2-Flash technical report 71.7 SWE-Bench Multilingual (Table 9) Capabilities
1 Dec 2025 DeepSeek-V3.2 technical report 70.2 SWE Multilingual (Resolved), DeepSeek-V3.2 Thinking (Table 2) Capabilities
24 Nov 2025 Claude Opus 4.5 system card 76.2 SWE-bench Multilingual, no thinking and 64k thinking both 76.20% (Table 2.4.A) Capabilities
29 Sep 2025 DeepSeek-V3.2-Exp technical report 57.9 SWE-bench Multilingual (Agent mode), DeepSeek-V3.2-Exp (Table 1) Capabilities
21 Jul 2025 Kimi K2 technical report 47.3 SWE-bench Multilingual (Pass@1), single-attempt agentic (Table 3) Capabilities

More in Coding

All 243 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471
  • NL2Repo-Bench In use From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate. DeepSeek V4.1 Flash 1017 68.2%