Cohere Command A
Cohere Command A ranks first on 2 of the 42 benchmarks it has scores on.
Cohere Released 13 Mar 2025 59
Benchmarks with a score
2
Ranked first, out of five or more models
3
In the top three
Cohere Mar 2025 Cohere Command A
Open weights
No Epoch Capabilities Index score yet
- LongBench v2 43.4% #25 /27
- CritPt 0% #127 /160
- Vals Finance Agent 4.23% #47 /49
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| DROP | 91.1% | #6 of 40 | Model card | |
| RULER | 95% | #6 of 38 | Model card | |
| IFEval | 90.9% | #20 of 126 | Model card | |
| MGSM | 90.1% | #12 of 52 | Model card | |
| RepoQA | 92.6% | #1 of 4 | Model card | |
| MMLU | 85.5% | #50 of 127 | Model card | |
| Vectara Hallucination Leaderboard (2025 dataset) | 9.30% | #47 of 104 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 4.50% | #72 of 148 | Official leaderboard | |
| LMArena (Elo) | 1,354 | #171 of 333 | Official leaderboard | |
| Wer wird Millionär? LLM Benchmark | 2800.0 | #41 of 76 | Official leaderboard | |
| INCLUDE | 74.3% | #14 of 25 | Model card | |
| HELM Arabic | 77.7 | #17 of 28 | Official leaderboard | |
| Global MMLU Lite | 83.8% | #29 of 43 | Official leaderboard | |
| MMLU-Pro | 69.6% | #112 of 151 | Model card | |
| Phare | 60.9% | #49 of 62 | Official leaderboard | |
| HELM Arabic Enterprise | 58.0 | #21 of 23 | Official leaderboard | |
| LongBench v2 | 43.4% | #25 of 27 | Model card | |
| StrategyQA | 76.7% | #2 of 2 | Model card | |
| Bamboogle | 76% | #1 of 1 | Model card | |
| Enterprise RAG evaluation (Cohere) | 4.73 | #1 of 1 | Model card | |
| FLORES-200 | 81.2 | #1 of 1 | Model card | |
| HotpotQA | 92.1% | #1 of 1 | Model card | |
| InFoBench | 94.9% | #1 of 1 | Model card | |
| Language Confusion Benchmark | 93% | #1 of 1 | Model card | |
| NTREX-128 | 68.8 | #1 of 1 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MATH | 80% | #23 of 93 | Model card | |
| DTBench | 35.5% | #137 of 178 | Official leaderboard | |
| Misguided Attention | 30.1% | #35 of 45 | Official leaderboard | |
| CritPt | 0% | #127 of 160 | Epoch AI | |
| GPQA Diamond | 50.8% | #229 of 284 | Model card | |
| AIME 2024 | 23.3% | #83 of 87 | Model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP+ | 86.2% | #1 of 15 | Model card | |
| Bird-SQL | 59.5% | #3 of 13 | Model card | |
| HumanEvalPack | 76.2% | #1 of 3 | Model card | |
| HumanEval | 85.4% | #38 of 95 | Model card | |
| Spider | 80.2% | #2 of 4 | Model card | |
| BigCodeBench | 45.4% | #8 of 11 | Model card | |
| Aider Polyglot | 12% | #49 of 54 | Epoch AI | |
| LiveCodeBench v5 | 26.9% | #42 of 44 | Model card | |
| SWE-bench Verified | 26.8% | #146 of 150 | Model card | |
| LBPP (Less Basic Python Problems) | 51.5% | #2 of 2 | Model card | |
| HumanEval-COBOL (Cohere) | 25.3% | #1 of 1 | Model card | |
| Multi-dialect SQL evaluation (Cohere) | 55.3% | #1 of 1 | Model card |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BFCL V4 | 46.5% | #24 of 65 | Official leaderboard | |
| Berkeley Function Calling Leaderboard (BFCL) | 63.8% | #38 of 47 | Model card | |
| tau-bench | 51.7% | #21 of 23 | Model card |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CaseLaw v2 | 64.5% | #13 of 53 | Official leaderboard | |
| LegalBench | 79.7% | #88 of 138 | Official leaderboard | |
| TaxEval v2 | 61.4% | #110 of 129 | Official leaderboard | |
| CorpFin v2 | 46% | #102 of 119 | Official leaderboard | |
| Vals Finance Agent v1 | 4.29% | #43 of 47 | Official leaderboard | |
| Vals Finance Agent | 4.23% | #47 of 49 | Official leaderboard | |
| Cohere enterprise human evaluation | 51.7% | #1 of 1 | Model card | |
| Enterprise generative evaluation (Cohere) | 94.2% | #1 of 1 | Model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| XSTest | 1.10% | #1 of 16 | Model card | |
| Default safety evaluation (Cohere) | 70.4% | #1 of 1 | Model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Conceptual Reasoning Index | 20.7 | #132 of 149 | Official leaderboard | |
| LMCA | 10.3% | #125 of 139 | Epoch AI | |
| ACCoRD | 37.3 | #135 of 150 | Official leaderboard |
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 35 of them aren't in the launch post.
Capabilities
- MMLU 85.5
- MMLU-Pro 69.6
- GPQA Diamond 50.8
- IFEval 90.9
- InFoBench 94.9
- ChatRAG Bench 72.9
- StrategyQA 76.7
- Bamboogle 76
- DROP 91.1
- HotpotQA 92.1
- Berkeley Function Calling Leaderboard (BFCL) 63.8
- tau-bench 51.7
- NTREX-128 68.8
- FLORES-200 81.2
- m-ArenaHard
- m-tau-bench (Cohere)
- Language Confusion Benchmark 93
- AL-QASIDA
- MBPP+ 86.2
- LiveCodeBench v5 26.9
- BigCodeBench 45.4
- LBPP (Less Basic Python Problems) 51.5
- HumanEvalPack 76.2
- HumanEval 85.4
- HumanEval-COBOL (Cohere) 25.3
- RepoQA 92.6
- SWE-bench Verified 26.8
- Aider Polyglot 14.7
- Spider 80.2
- Bird-SQL 59.5
- Multi-dialect SQL evaluation (Cohere) 55.3
- MATH 80
- AIME 2024 23.3
- Enterprise generative evaluation (Cohere) 94.2
- Enterprise RAG evaluation (Cohere) 4.73
- RULER 95
- LongBench v2 43.4
- Cohere enterprise human evaluation
- MGSM 90.1
- INCLUDE 74.3