- OSWorld (screenshot only, 50 steps) Overall 22
- SWE-bench Verified (% of problems which pass all tests) 49
- Mean of TAU-bench Retail (pass^1) / TAU-bench Airline (pass^1) 57.6
- MMMU (validation) 70.4
- MathVista (testmini) 70.7
- AI2D (test) 95.3
- ChartQA (test, relaxed accuracy) 90.8
- DocVQA (test, ANLS score) 94.2
- GPQA (Diamond) (0-shot CoT) 65
- MMLU (5-shot CoT) 90.5
- MMLU Pro (0-shot CoT) 78
- MATH (0-shot CoT) 78.3
- HumanEval (0-shot) 93.7
- MGSM (0-shot CoT) 92.5
- DROP (F1 Score, 3-shot) 88.3
- BIG-Bench Hard (3-shot CoT) 93.2
- AIME 2024 (Maj@64 0-shot CoT) 27.6
- IFEval 90.2
Claude 3.5 Sonnet (October 2024)
Claude 3.5 Sonnet (October 2024) ranks first on 5 of the 91 benchmarks it has scores on.
Anthropic Released 22 Oct 2024
95
Benchmarks with a score
5
Ranked first, out of five or more models
13
In the top three
Anthropic Oct 2024
Claude 3.5 Sonnet (October 2024)
Closed weights
133.4
#152 of 271
Epoch Capabilities Index
- Humanity's Last Exam 4.08% #45 /47
- ZeroBench 3.00% #45 /66
- GPQA Diamond 55.3% #138 /219
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BIG-Bench Hard | 93.2% | #2 of 71 | Model card | |
| MGSM | 92.5% | #3 of 52 | Model card | |
| ToRR | 49.9 | #1 of 14 | Official leaderboard | |
| MMLU | 90.5% | #10 of 127 | Model card | |
| MMBench-CN v1.1 | 82.5% | #29 of 178 | Official leaderboard | |
| IFEval | 90.2% | #23 of 126 | Model card | |
| DROP | 88.3% | #10 of 40 | Model card | |
| Lech Mazur Creative Writing | 8.03 | #18 of 44 | Epoch AI | |
| SuperGPQA | 48.2% | #24 of 58 | Official leaderboard | |
| LMArena (Elo) | 1,374 | #156 of 333 | Official leaderboard | |
| Creation-MMBench | -15.46 | #9 of 19 | Official leaderboard | |
| MMLU-Pro | 78% | #86 of 151 | Model card | |
| LongBench v2 | 46.7% | #19 of 33 | Official leaderboard | |
| AgentIF | 56.6% | #7 of 12 | Official leaderboard | |
| LLM Confabulation Leaderboard | 19.9% | #37 of 58 | Official leaderboard | |
| Arena-Hard v2.0 | 33% | #16 of 23 | Official leaderboard | |
| Honesty human feedback win rate (Anthropic) | 55% | #5 of 6 | Model card | |
| Humanity's Last Exam | 4.08% | #45 of 47 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ZebraLogic | 36.2% | #7 of 46 | Official leaderboard | |
| MATH | 78.3% | #25 of 93 | Model card | |
| MATH Level 5 | 57% | #44 of 94 | Epoch AI | |
| ForecastBench | 63.2 | #44 of 83 | Epoch AI | |
| Misguided Attention | 37.2% | #27 of 45 | Official leaderboard | |
| GPQA Diamond | 55.3% | #138 of 219 | Epoch AI | |
| SimpleBench | 41.4% | #65 of 100 | Official leaderboard | |
| DTBench | 46.4% | #117 of 178 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 8.47% | #154 of 202 | Epoch AI | |
| U-MATH | 38.7% | #23 of 30 | Official leaderboard | |
| FrontierMath | 2.07% | #54 of 70 | Epoch AI | |
| FrontierMath Tier 4 | 0% | #47 of 55 | Epoch AI | |
| AIME 2024 | 27.6% | #81 of 87 | Model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 93.7% | #6 of 95 | Model card | |
| HumanEval-V | 36.8% | #2 of 24 | Official leaderboard | |
| SWE-PolyBench | 14.1% | #1 of 6 | Official leaderboard | |
| MHPP | 48.1% | #4 of 23 | Official leaderboard | |
| SciCode | 4.60% | #4 of 18 | Official leaderboard | |
| Multi-SWE-bench | 7.97% | #3 of 11 | Official leaderboard | |
| Internal agentic coding evaluation (Anthropic) | 78% | #1 of 3 | Model card | |
| Aider Polyglot | 51.6% | #26 of 54 | Epoch AI | |
| METR Time Horizon | 30 min | #15 of 31 | Official leaderboard | |
| WeirdML | 40% | #80 of 130 | Epoch AI | |
| METR Time Horizon 1.1 | 21 min | #16 of 26 | Official leaderboard | |
| LiveCodeBench v5 | 36.5% | #16 of 21 | Official leaderboard | |
| LiveCodeBench v6 | 36.4% | #16 of 21 | Official leaderboard | |
| SWE-bench Multimodal | 25.3% | #7 of 9 | Official leaderboard | |
| GSO-Bench | 4.60% | #24 of 30 | Official leaderboard | |
| SWE-bench Verified | 49% | #128 of 150 | Model card | |
| ArtifactsBench | 39.9% | #19 of 20 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Factorio Learning Environment | 293206.0 | #1 of 6 | Official leaderboard | |
| Elimination Game | 5 | #14 of 56 | Official leaderboard | |
| BALROG | 32.6% | #15 of 37 | Epoch AI | |
| Berkeley Function Calling Leaderboard (BFCL) | 56.9% | #33 of 67 | Official leaderboard | |
| LisanBench | 459.7 | #69 of 133 | Official leaderboard | |
| Step Race | 2 | #34 of 59 | Official leaderboard | |
| LLM Chess | -99 | #72 of 124 | Official leaderboard | |
| tau-bench | 57.6% | #15 of 23 | Model card | |
| Galileo Agent Leaderboard | 0.80 | #25 of 31 | Official leaderboard | |
| OSWorld | 14.9% | #10 of 12 | Model card | |
| OSWorld-Verified | 22% | #37 of 37 | Model card |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TheAgentCompany | 24% | #6 of 14 | Epoch AI | |
| MedHELM | 54.2 | #6 of 13 | Official leaderboard | |
| CadEval | 48% | #7 of 15 | Epoch AI | |
| Vending-Bench | $2,218 | #6 of 9 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AI2D | 95.3% | #2 of 44 | Model card | |
| MEGA-Bench | 54.3% | #1 of 22 | Official leaderboard | |
| ChartQA | 90.8% | #3 of 37 | Model card | |
| Video-MMMU | 65.8% | #4 of 23 | Official leaderboard | |
| KRETA | 80.5% | #3 of 17 | Official leaderboard | |
| MMStar | 65.1% | #31 of 173 | Official leaderboard | |
| MMVU | 65.2% | #6 of 28 | Official leaderboard | |
| MMBench | 82.6% | #35 of 155 | Official leaderboard | |
| MMBench-EN v1.1 | 80.9% | #47 of 178 | Official leaderboard | |
| CCBench (MMBench Chinese Culture) | 56.1% | #46 of 152 | Official leaderboard | |
| Image2Struct | 60.5 | #8 of 18 | Official leaderboard | |
| DocVQA | 94.2% | #15 of 33 | Model card | |
| VHELM | 49.7 | #24 of 41 | Official leaderboard | |
| MMMU | 70.4% | #57 of 97 | Model card | |
| Video-MME | 60% | #30 of 50 | Epoch AI | |
| MathVista | 70.7% | #40 of 64 | Model card | |
| VISTA | 38.7% | #35 of 54 | Official leaderboard | |
| ZeroBench | 3.00% | #45 of 66 | Official leaderboard | |
| Vision Arena | 1,161 | #95 of 136 | Official leaderboard | |
| GeoBench | 62% | #19 of 27 | Epoch AI | |
| EnigmaEval | 0.91% | #34 of 41 | Epoch AI | |
| VPCT | 33% | #27 of 28 | Epoch AI |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AIR-Bench 2024 | 90.8% | #2 of 82 | Official leaderboard | |
| AgentDojo | 1.11% | #1 of 20 | Official leaderboard | |
| XSTest | 4.30% | #4 of 16 | Model card | |
| WildChat non-toxic incorrect refusals (Anthropic) | 5.30% | #2 of 3 | Model card | |
| WildChat toxic refusals (Anthropic) | 89.2% | #2 of 3 | Model card | |
| Harmlessness human feedback win rate (Anthropic) | 48% | #3 of 4 | Model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 84.6 | #9 of 71 | Official leaderboard | |
| OpenVLM Leaderboard | 70.6 | #33 of 173 | Official leaderboard | |
| LiveBench | 59% | #16 of 48 | Epoch AI | |
| HELM Capabilities | 65.3 | #32 of 60 | Official leaderboard |
Anthropic's models over time
Anthropic's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 8 of them aren't in the launch post.
Capabilities
- OSWorld 14.9
- SWE-bench Verified 49
- tau-bench
- Internal agentic coding evaluation (Anthropic) 78
- MMMU 70.4
- MathVista 70.7
- AI2D 95.3
- ChartQA 90.8
- DocVQA 94.2
- GPQA Diamond 65
- MMLU 90.5
- MMLU-Pro 78
- MATH 78.3
- HumanEval 93.7
- MGSM 92.5
- DROP 88.3
- BIG-Bench Hard 93.2
- AIME 2024 16
- IFEval 90.2
- Human feedback win rates (Anthropic)