Model Card and Evaluations for Claude Models 11 Jul 2023
- Codex P@1 (0-shot) 71.2
- GSM8k (0-shot CoT) 88
- MMLU (5-shot CoT) 78.5
- TriviaQA (5-shot) 87.5
- QuALITY (5-shot) 83.2
- ARC-Challenge (5-shot) 91
- RACE-H (5-shot) 88.3
Claude 2
Claude 2 places in the top three on RACE-H.
31
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
Anthropic Jul 2023
Claude 2
Closed weights Flagship
120.1
#196 of 271
Epoch Capabilities Index
- GPQA Diamond 34.7% #187 /219
- Vectara Hallucination Leaderboard (HHEM) 17.4% #5 /148
- SWE-bench Lite 3.00% #13 /15
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Vectara Hallucination Leaderboard (HHEM) | 17.4% | #5 of 148 | Official leaderboard | |
| TriviaQA | 87.5% | #6 of 39 | Model card | |
| TruthfulQA | 69% | #6 of 28 | Model card | |
| ARC-Challenge | 91% | #14 of 45 | Model card | |
| GRE Analytical Writing | 5.00 | #1 of 3 | Model card | |
| RACE-H | 88.3% | #3 of 7 | Model card | |
| AlpacaEval 2.0 | 28.2% | #18 of 40 | Official leaderboard | |
| AlpacaEval | 91.4% | #9 of 19 | Official leaderboard | |
| EQ-Bench v2 | 72.9 | #46 of 85 | Official leaderboard | |
| QuALITY | 83.2% | #4 of 7 | Model card | |
| MMLU | 78.5% | #79 of 127 | Model card | |
| GRE Verbal | 165.0 | #5 of 6 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 88% | #46 of 87 | Model card | |
| GRE Quantitative | 154.0 | #5 of 6 | Model card | |
| GPQA Diamond | 34.7% | #187 of 219 | Epoch AI | |
| DTBench | 19.7% | #157 of 178 | Official leaderboard | |
| MATH Level 5 | 11.7% | #84 of 94 | Epoch AI | |
| OTIS Mock AIME 2024-2025 | 2.50% | #181 of 202 | Epoch AI |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 61.6% | #26 of 46 | Official leaderboard | |
| HumanEval | 71.2% | #68 of 95 | Model card | |
| Bird-SQL | 49% | #10 of 13 | Official leaderboard | |
| SWE-bench | 1.96% | #7 of 9 | Official leaderboard | |
| SWE-bench Lite | 3.00% | #13 of 15 | Official leaderboard | |
| SWE-bench Verified | 4.40% | #58 of 60 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBE (Multistate Bar Examination) | 76.5% | #2 of 4 | Model card | |
| USMLE Step 1 (sample questions) | 68.9% | #1 of 1 | Model card | |
| USMLE Step 2 (sample questions) | 63.3% | #1 of 1 | Model card | |
| USMLE Step 3 (sample questions) | 67.2% | #1 of 1 | Model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Combined HHH evals (Anthropic) | 86% | #1 of 1 | Model card | |
| Harmfulness scores on held-out prompts (Anthropic) | 1.20% | #1 of 1 | Model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 48.9 | #39 of 71 | Official leaderboard |
Anthropic's models over time
Anthropic's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 13 of them aren't in the launch post.
Model Card and Evaluations for Claude Models 11 Jul 2023
Capabilities
- FLORES-200
- HumanEval 71.2
- GSM8K 88
- MMLU 78.5
- TriviaQA 87.5
- QuALITY 83.2
- ARC-Challenge 91
- RACE-H 88.3
- GRE Verbal 165
- GRE Quantitative 154
- GRE Analytical Writing 5
- MBE (Multistate Bar Examination) 76.5
- USMLE Step 1 (sample questions) 68.9
- USMLE Step 2 (sample questions) 63.3
- USMLE Step 3 (sample questions) 67.2
Hallucination
- TruthfulQA 69