Benchmark Almanac

SWE-Lancer

GPT-5.3 Codex 81.4% leads since Feb 2026 Fading

Models complete real Upwork software tasks, from bug fixes to features, and pick between engineering proposals. Scored by end-to-end tests and payout earned.

Only the Diamond split is public. Some cards report dollars earned rather than pass rate. Since 2025-07-17 the public repo holds 198 of the 237 IC SWE Diamond tasks. OpenAI dropped 39 tasks that failed offline, so later scores use a smaller set.

In the authors’ words

We introduce SWE-Lancer, a benchmark of over 1,400 freelance software engineering tasks from Upwork, valued at $1 million USD total in real-world payouts. SWE-Lancer encompasses both independent engineering tasks–ranging from $50 bug fixes to $32,000 feature implementations–and managerial tasks, where models choose between technical implementation proposals. Independent tasks are graded with end-to-end tests triple-verified by experienced software engineers, while managerial decisions are assessed against the choices of the original hired engineering managers. We evaluate model performance and find that frontier models are still unable to solve the majority of tasks. To facilitate future research, we open-source a unified Docker image and a public evaluation split, SWE-Lancer Diamond (https://github.com/openai/SWELancer-Benchmark). By mapping model performance to monetary value, we hope SWE-Lancer enables greater research into the economic impact of AI model development.

Abstract of SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? (arXiv:2502.12115, CC0)
Made by OpenAI Released Feb 2025 Fixing real repositories Public tasks plus a private test set Last in a launch post Feb 2026 , dropped before it saturated Last model added Feb 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Code A program or patch run against tests.
openai/frontier-evals 1,331 stars Paper
10
Models scored in lab cards
10
Launch posts list it, from 2 labs
9
System cards report it, from 3 labs

GPT-5.3 Codex leads with 81.4% among models released by Feb 2026

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

The newest model with a score here came out in Feb 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 81.4 GPT-5.3 Codex
  2. 80.0 GPT-5.1-Codex-Max
  3. 55.0 GPT-5
  4. 54.0 o3
  5. 53.0 OpenAI deep research
  6. 52.0 o4-mini
  7. 51.0 GPT-5 mini
  8. 49.4 Seed2.0 Pro
  9. 39.1 Kimi K2
  10. 32.6 GPT-4.5
  1. 1 GPT-5.3 Codex 81.4%
  2. 2 GPT-5.1-Codex-Max 80%
  3. 3 GPT-5 55%
  4. 4 o3 54%
  5. 5 OpenAI deep research 53%
  6. 6 o4-mini 52%
  7. 7 GPT-5 mini 51%
  8. 8 Seed2.0 Pro 49.4%
  9. 9 Kimi K2 Jul 2025 39.1%
  10. 10 GPT-4.5 32.6%
#ModelScoreSettingDateSource
1 GPT-5.3 Codex 81.4% Feb 2026 Model card
2 GPT-5.1-Codex-Max 80% Nov 2025 Model card
3 GPT-5 55% Aug 2025 Model card
4 o3 54% Apr 2025 Model card
5 OpenAI deep research 53% Feb 2025 Model card
6 o4-mini 52% Apr 2025 Model card
7 GPT-5 mini 51% Aug 2025 Model card
8 Seed2.0 Pro 49.4% Feb 2026 Model card
9 Kimi K2 Jul 2025 39.1% Jul 2025 Model card
10 GPT-4.5 32.6% Feb 2025 Model card

5 models have held number one since its release in Feb 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Lab claims Not tested since Feb 2026
GPT-5.3 Codex 81.4%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) GPT-5.3 Codex GPT-5.1-Codex-Max GPT-5 o3 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) GPT-5.3 Codex GPT-5

Labs headline it less often now, last in Feb 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

OpenAI
ByteDance Seed
DateLaunch postScoreAs the post writes it
14 Feb 2026 Seed2.0 Pro 49.4 SWE-Lancer
5 Feb 2026 GPT-5.3 Codex 81.4 SWE-Lancer IC Diamond
19 Nov 2025 GPT-5.1-Codex-Max 79.9 SWE-Lancer IC SWE, xhigh
7 Aug 2025 GPT-5 nano 49000 SWE-Lancer: IC SWE Diamond Freelance Coding Tasks ($49K)
7 Aug 2025 GPT-5 mini 75000 SWE-Lancer: IC SWE Diamond Freelance Coding Tasks ($75K)
16 Apr 2025 o4-mini 65792 SWE-Lancer: IC SWE Diamond (o4-mini-high), dollars earned
16 Apr 2025 o3 65250 SWE-Lancer: IC SWE Diamond (dollars earned), o3-high
14 Apr 2025 GPT-4.1 nano 9000 SWE-Lancer (IC-Diamond subset) $9K (3.7%)
14 Apr 2025 GPT-4.1 mini 31000 SWE-Lancer (IC-Diamond subset) $31K (13.1%)
27 Feb 2025 GPT-4.5 32.6 SWE-Lancer Diamond (coding)*

9 system cards report it, from 3 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
14 Feb 2026 Seed2.0 Pro model card 49.4 SWE-Lancer (Table 11) Capabilities
18 Nov 2025 GPT-5.1-Codex-Max system card 80 SWE-Lancer IC SWE Diamond, pass@1 avg of 3, printed chart label (Figure 8, 5.1.3.1) AI R&D risk
7 Aug 2025 GPT-5 system card 55 SWE-Lancer IC SWE Diamond, pass@1 avg of 3, no browsing launch candidate (Figure 24) AI R&D risk
7 Aug 2025 GPT-5 mini system card 51 SWE-Lancer IC SWE Diamond, pass@1 avg of 3, gpt-5-thinking-mini launch candidate (Figure 24) AI R&D risk
21 Jul 2025 Kimi K2 technical report 39.1 SWE-Lancer (Pass@1) (Table 3) Capabilities
16 Apr 2025 o4-mini system card 52 SWE-Lancer Diamond (IC SWE), pass@1, o4-mini no browsing launch candidate, printed chart label (Figure 21) AI R&D risk
16 Apr 2025 o3 system card 54 SWE-Lancer Diamond (IC SWE), pass@1, o3 no browsing launch candidate, printed chart label (Figure 21) AI R&D risk
27 Feb 2025 GPT-4.5 system card 20 SWE-Lancer Diamond IC SWE tasks, pass@1, post-mitigation, earned $41,625 (4.6.6) Autonomy
25 Feb 2025 OpenAI deep research system card 53 SWE-Lancer Diamond (IC SWE), pass@1 averaged over 3 runs, post-mitigation with browsing, $87,708 earned (without browsing 48%, $75,583), July 2025 dataset update (3.4.7) Autonomy

More in Coding

All 243 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Opus 5.5 93.9%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471