CRUXEval
qwen3-coder-next 95.9% leads since Feb 2026 Fading Models predict the output of 800 short Python functions for a given input, or an input that yields a given output. Scored by pass@1.
In the authors’ words
We present CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each function comes with an input-output pair, leading to two natural tasks: input prediction and output prediction. First, we propose a generic recipe for generating our execution benchmark which can be used to create future variation of the benchmark. Second, we evaluate twenty code models on our benchmark and discover that many recent high-scoring models on HumanEval do not show the same improvements on our benchmark. Third, we show that simple CoT and fine-tuning schemes can improve performance on our benchmark but remain far from solving it. The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75% and 81% on input and output prediction, respectively. In contrast, Code Llama 34B achieves a pass@1 of 50% and 46% on input and output prediction, highlighting the gap between open and closed source models. As no model is close to acing CRUXEval, we provide examples of consistent GPT-4 failures on simple programs as a lens into its code reasoning capabilities and areas for improvement.
- Level
- School Students, from grade school to high school
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Short answer A number, word or short string matched to a key.
qwen3-coder-next leads with 95.9% among models released by Apr 2026
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
The newest model with a score here came out in Apr 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6 Astra
- Claude Opus 5.5
- GPT-6.1 Terra
- GPT-6.1 Sol
- and 16 more
- 1 qwen3-coder-next 95.9%
- 2
Code World Model (CWM) 94.3% - 3
Falcon-H1-34B 73.1% - 4
DeepSeek-V3 69.8% - 5 DeepSeek-V2 (MoE-236B, May 2024) 61.5%
- 6 Granite 4.1 30B 55.8%
- 7
Codestral 55.5% - 8 Granite 4.0 H Small 50.3%
- 9 granite-4.0-1b 33.1%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | qwen3-coder-next | 95.9% | Feb 2026 | Model card | |
| 2 | Code World Model (CWM) | 94.3% | Sep 2025 | Model card | |
| 3 | Falcon-H1-34B | 73.1% | May 2025 | Model card | |
| 4 | DeepSeek-V3 | 69.8% | Dec 2024 | Model card | |
| 5 | DeepSeek-V2 (MoE-236B, May 2024) | 61.5% | May 2024 | Model card | |
| 6 | Granite 4.1 30B | 55.8% | Apr 2026 | Model card | |
| 7 | Codestral | 55.5% | Jan 2025 | Model card | |
| 8 | Granite 4.0 H Small | 50.3% | Oct 2025 | Model card | |
| 9 | granite-4.0-1b | 33.1% | Oct 2025 | Model card |
The benchmark's own leaderboard has 34 entries, 11 of them models we track. We plot every source in the charts below.
Scores at each effort level
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Claude 3 Opus | CoT | 82% | |||
| GPT-4 Turbo Apr 2024 | CoT | 82% | |||
| GPT-4 Jun 2023 | CoT | 77.1% | |||
| GPT-4 Jun 2023 | 68.7% | ||||
| GPT-4 Turbo Apr 2024 | 67.7% | ||||
| Claude 3 Opus | 65.8% | ||||
| GPT-3.5 Turbo Jun 2023 | CoT | 59% | |||
| GPT-3.5 Turbo Jun 2023 | 49.4% | ||||
 StarCoder 2 15B | 47.1% | ||||
| Mixtral 8x7B | 40.5% | ||||
| StarCoder 2 7B | 36% | ||||
| Mistral 7B v0.1 | 34.3% | ||||
| StarCoder 2 3B | 34.2% | ||||
 Phi-2 | 33.5% | ||||
| Phi-1.5 | 27.5% |
2 models have held number one since its release in Jan 2024
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
The best score passed 85% in Sep 2025
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Labs headline it less often now, last in Apr 2026
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 29 Apr 2026 | Granite 4.1 30B | 55.8 | CRUXEval-O (pass@1) |
| 28 Oct 2025 | granite-4.0-1b | 33.1 | CRUXEval-O (pass@1) |
| 2 Oct 2025 | Granite 4.0 H Small | 50.3 | CRUXEval-O (pass@1) |
| 20 May 2025 | Falcon-H1-34B | 73.1 | CRUXEval |
| 28 Jan 2025 | Qwen2.5-Max | Best of CRUX-I / CRUX-O | |
| 13 Jan 2025 | Codestral | 55.5 | CruxEval |
| 26 Dec 2024 | DeepSeek-V3 | 69.8 | Best of CRUXEval-I (Acc.) / CRUXEval-O (Acc.) |
5 system cards report it, from 5 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 28 Feb 2026 | qwen3-coder-next technical report | 95.9 | CRUXEval (Table 6) | Capabilities |
| 30 Sep 2025 | Code World Model (CWM) technical report | 94.3 | CruxEval-output pass@1, language reasoning with CoT, full trace prediction 87.7 (Table 8). Non-reasoning CRUX-O 83.4 (Table 13) | Capabilities |
| 30 Jul 2025 | Falcon-H1-34B technical report | 73.1 | CRUXEval pass@1, input and output average (Table 24) | Capabilities |
| 4 Jun 2025 |  Seed-Coder-8B-Instruct technical report | CRUXEval Input-CoT 63.3, Output-CoT 67.1, no overall score (Table 12) | Capabilities | |
| 7 May 2024 | DeepSeek-V2 (MoE-236B, May 2024) technical report | 61.5 | CRUXEval-I-COT 2-shot, DeepSeek-V2 Chat (RL) (Table 3) | Capabilities |
More in Coding
- SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
- SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Opus 5.5 93.9%
- LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
- DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
- SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
- Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471