Claude 2.1
Claude 2.1.
Anthropic Released 21 Nov 2023
12
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Anthropic Nov 2023
Claude 2.1
Closed weights
119.2
#199 of 271
Epoch Capabilities Index
- GPQA Diamond 33% #192 /219
- Situational Awareness Dataset (SAD) 46.2% #10 /17
- OTIS Mock AIME 2024-2025 1.94% #186 /202
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MMLU | 73.5% | #46 of 127 | Epoch AI | |
| EQ-Bench v2 | 74.0 | #39 of 85 | Official leaderboard | |
| AlpacaEval 2.0 | 25.3% | #21 of 40 | Official leaderboard | |
| AlpacaEval | 87.1% | #12 of 19 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ForecastBench | 58.6 | #18 of 86 | Official leaderboard | |
| GPQA Diamond | 33% | #192 of 219 | Epoch AI | |
| DTBench | 18.3% | #159 of 178 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 1.94% | #186 of 202 | Epoch AI |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| WeirdML | 7.06% | #112 of 112 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Situational Awareness Dataset (SAD) | 46.2% | #10 of 17 | Official leaderboard | |
| OR-Bench | 99.8% | #28 of 28 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 43.7 | #46 of 71 | Official leaderboard |
Anthropic's models over time
Anthropic's models in color, against every other model on the Epoch Capabilities Index, by release date.