- MMLU (5-shot CoT) 88.2
- MATH (Maj@32 4-shot) 73.7
- GSM8K (0-shot CoT) 95
- HumanEval (0-shot) 84.9
- GPQA (Diamond) (Maj@32 5-shot CoT) 59.5
- MGSM (0-shot) 90.7
- DROP (F1 Score, 3-shot) 83.1
- BIG-Bench-Hard (3-shot CoT) 86.8
- ARC-Challenge (25-shot) 96.4
- HellaSwag (10-shot) 95.4
- PubMedQA (5-shot) 75.8
- WinoGrande (5-shot) 88.5
- RACE-H (5-shot) 92.9
- APPS (0-shot) 70.2
- MBPP (Pass@1) 86.4
- MMMU (val) Overall 59.4
- DocVQA (test, ANLS score) 89.3
- MathVista (testmini) CoT 50.5
- AI2D (test) 88.1
- ChartQA (test, relaxed accuracy) CoT 80.8
Claude 3 Opus
Claude 3 Opus ranks first on 6 of the 101 benchmarks it has scores on.
112
Benchmarks with a score
6
Ranked first, out of five or more models
17
In the top three
Anthropic Feb 2024
Claude 3 Opus
Closed weights Flagship
126.9
#177 of 271
Epoch Capabilities Index
- ZeroBench 0% #61 /66
- GPQA Diamond 47.2% #160 /219
- Vectara Hallucination Leaderboard (HHEM) 10.1% #19 /148
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HellaSwag | 95.4% | #2 of 47 | Model card | |
| ARC-Challenge | 96.4% | #2 of 45 | Model card | |
| RepoQA | 90.6% | #1 of 22 | Official leaderboard | |
| WinoGrande | 88.5% | #3 of 42 | Model card | |
| ThaiExam | 68.6% | #2 of 24 | Official leaderboard | |
| MixEval | 88.1% | #3 of 34 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 10.1% | #19 of 148 | Official leaderboard | |
| QuALITY | 90.5% | #1 of 7 | Model card | |
| RACE-H | 92.9% | #1 of 7 | Model card | |
| AlpacaEval 2.0 | 40.5% | #6 of 40 | Official leaderboard | |
| EQ-Bench v2 | 82.2 | #14 of 85 | Official leaderboard | |
| WildBench | 52.1 | #8 of 47 | Official leaderboard | |
| MGSM | 90.7% | #9 of 52 | Model card | |
| MMLU | 88.2% | #26 of 127 | Model card | |
| BIG-Bench Hard | 86.8% | #15 of 71 | Model card | |
| Okapi MMLU | 79.1% | #2 of 8 | Model card | |
| 100Q-Hard (Anthropic) | 46.5% | #2 of 7 | Model card | |
| GRE Verbal | 166.0 | #2 of 6 | Model card | |
| MMLU-Pro | 68.5% | #90 of 212 | Official leaderboard | |
| Needle In A Haystack | 99.4% | #4 of 9 | Model card | |
| DROP | 83.1% | #20 of 40 | Model card | |
| LSAT | 161.0 | #2 of 4 | Model card | |
| RewardBench | 80.1% | #29 of 48 | Official leaderboard | |
| LMArena (Elo) | 1,322 | #206 of 333 | Official leaderboard | |
| GRE Analytical Writing | 5.00 | #2 of 3 | Model card | |
| LLM Confabulation Leaderboard | 22.7% | #45 of 58 | Official leaderboard | |
| RewardBench 2 | 57.4% | #34 of 42 | Official leaderboard | |
| MMBench-CN v1.1 | 58.1% | #148 of 178 | Official leaderboard | |
| SEAL Precise Instruction Following | 80.1 | #15 of 18 | Official leaderboard | |
| SimpleQA Verified | 12.6% | #79 of 84 | Epoch AI | |
| SEAL Multilingual | 919 | #18 of 19 | Official leaderboard | |
| Multi-factual (Anthropic) | 62.8% | #1 of 1 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ForecastBench | 54.5 | #10 of 86 | Official leaderboard | |
| MathBench | 63% | #3 of 21 | Official leaderboard | |
| GPQA Main | 49.1% | #1 of 7 | Model card | |
| GSM8K | 95% | #13 of 87 | Model card | |
| GSM1k | 95.2% | #4 of 15 | Official leaderboard | |
| ZebraLogic | 27% | #15 of 46 | Official leaderboard | |
| AMC 10 2023 | 72.0 | #1 of 3 | Model card | |
| AMC 12 2023 | 63.0 | #1 of 3 | Model card | |
| AMC 8 2023 | 84.0 | #1 of 3 | Model card | |
| GPQA Extended | 48.8% | #1 of 3 | Model card | |
| MATH | 73.7% | #35 of 93 | Model card | |
| GRE Quantitative | 159.0 | #3 of 6 | Model card | |
| OlympiadBench | 7.65% | #4 of 7 | Official leaderboard | |
| MATH Level 5 | 37.5% | #65 of 94 | Epoch AI | |
| GPQA Diamond | 47.2% | #160 of 219 | Epoch AI | |
| DTBench | 36% | #134 of 178 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 4.72% | #172 of 202 | Epoch AI | |
| SimpleBench | 23.5% | #87 of 100 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 86.4% | #6 of 66 | Model card | |
| CRUXEval | 82% | #1 of 11 | Official leaderboard | |
| BigCodeBench | 51.5% | #16 of 69 | Official leaderboard | |
| MBPP+ | 73.3% | #9 of 35 | Official leaderboard | |
| HumanEval+ | 77.4% | #14 of 46 | Official leaderboard | |
| APPS | 70.2% | #1 of 3 | Model card | |
| Aider-Edit | 68.4% | #14 of 37 | Official leaderboard | |
| BigCodeBench-Hard | 26% | #40 of 96 | Official leaderboard | |
| HumanEval | 84.9% | #40 of 95 | Model card | |
| SciCode | 1.50% | #8 of 18 | Official leaderboard | |
| SWE-bench | 10.5% | #6 of 9 | Official leaderboard | |
| SEAL Coding | 959 | #19 of 26 | Official leaderboard | |
| METR Time Horizon | 7.1 min | #24 of 31 | Official leaderboard | |
| SWE-bench Lite | 11.7% | #12 of 15 | Official leaderboard | |
| METR Time Horizon 1.1 | 4.0 min | #22 of 26 | Official leaderboard | |
| WeirdML | 19.2% | #102 of 112 | Official leaderboard | |
| SWE-bench Verified | 15.8% | #55 of 60 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GTA | 23.4% | #4 of 15 | Official leaderboard | |
| Berkeley Function Calling Leaderboard (BFCL) | 57% | #32 of 67 | Official leaderboard | |
| LisanBench | 249.0 | #84 of 133 | Official leaderboard | |
| ToolComp | 52.8% | #21 of 32 | Official leaderboard | |
| LLM Chess | -146 | #88 of 124 | Official leaderboard | |
| Chess Puzzles | 3.00% | #79 of 83 | Epoch AI |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBE (Multistate Bar Examination) | 85% | #1 of 4 | Model card | |
| PubMedQA | 75.8% | #3 of 3 | Model card |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MTVQA-Test | 25.7% | #4 of 18 | Official leaderboard | |
| TOMATO | 28.2% | #8 of 22 | Official leaderboard | |
| BLINK | 44.1% | #4 of 11 | Official leaderboard | |
| Image2Struct | 62.3 | #7 of 18 | Official leaderboard | |
| MMMU | 59.4% | #60 of 136 | Official leaderboard | |
| AI2D | 88.1% | #24 of 44 | Model card | |
| CharXiv | 30.2% | #44 of 77 | Official leaderboard | |
| MMStar | 45.7% | #123 of 173 | Official leaderboard | |
| MATH-Vision (MathVision) | 27.1% | #78 of 105 | Official leaderboard | |
| VHELM | 31.9 | #32 of 41 | Official leaderboard | |
| CCBench (MMBench Chinese Culture) | 26.3% | #123 of 152 | Official leaderboard | |
| DocVQA | 89.3% | #28 of 33 | Model card | |
| ChartQA | 80.8% | #32 of 37 | Model card | |
| VISTA | 27.8% | #47 of 54 | Official leaderboard | |
| Vision Arena | 1,062 | #119 of 136 | Official leaderboard | |
| EnigmaEval | 0.82% | #36 of 41 | Epoch AI | |
| MMBench | 63.3% | #137 of 155 | Official leaderboard | |
| MMBench-EN v1.1 | 60.1% | #160 of 178 | Official leaderboard | |
| ZeroBench | 0% | #61 of 66 | Official leaderboard | |
| MathVista | 50.5% | #60 of 64 | Model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SimpleSafetyTests | 100% | #1 of 75 | Official leaderboard | |
| Anthropic Red Team | 99.8% | #8 of 75 | Official leaderboard | |
| HELM Safety | 96.8 | #13 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 84.4% | #16 of 82 | Official leaderboard | |
| Situational Awareness Dataset (SAD) | 53.4% | #4 of 17 | Official leaderboard | |
| XSTest | 9.00% | #5 of 16 | Model card | |
| SEAL Adversarial Robustness | 13.0 | #3 of 8 | Official leaderboard | |
| AgentDojo | 11.3% | #9 of 20 | Official leaderboard | |
| Cybench | 10% | #18 of 24 | Official leaderboard | |
| OR-Bench | 91% | #24 of 28 | Official leaderboard | |
| Multimodal policy red-teaming (Anthropic) | 97.9% | #2 of 2 | Model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 68.3 | #24 of 71 | Official leaderboard | |
| LiveBench | 49.2% | #27 of 48 | Epoch AI | |
| OpenVLM Leaderboard | 54.4 | #110 of 173 | Official leaderboard | |
| LMCA | 17% | #113 of 139 | Epoch AI | |
| Conceptual Reasoning Index | 24.0 | #129 of 149 | Official leaderboard | |
| ACCoRD | 33.1 | #139 of 150 | Official leaderboard |
Anthropic's models over time
Anthropic's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 23 of them aren't in the launch post.
Capabilities
- MMLU 88.2
- MATH 61
- GSM8K 95
- HumanEval 84.9
- GPQA Diamond 50.4
- GPQA Main 49.1
- GPQA Extended 48.8
- MGSM 90.7
- DROP 83.1
- BIG-Bench Hard 86.8
- ARC-Challenge 96.4
- HellaSwag 95.4
- PubMedQA 75.8
- WinoGrande 88.5
- RACE-H 92.9
- APPS 70.2
- MBPP 86.4
- LSAT 161
- MBE (Multistate Bar Examination) 85
- AMC 12 2023 63
- AMC 10 2023 72
- AMC 8 2023 84
- GRE Quantitative 159
- GRE Verbal 166
- GRE Analytical Writing 5
- MMMU 59.4
- DocVQA 89.3
- MathVista 50.5
- AI2D 88.1
- ChartQA 80.8
- Okapi MMLU 79.1
- QuALITY 90.5
- Needle In A Haystack 99.4