Capabilities
- tau-bench
- Berkeley Function Calling Leaderboard (BFCL) 76.4
- BrowseComp 21.3
- MMLU-Pro 81.4
- AIME 2024 89.4
- MATH-500 98.1
- SciCode 37.3
- GPQA Diamond 75
- Humanity's Last Exam 10.6
- LiveCodeBench v5 70.7
- Artificial Analysis Intelligence Index v2 64.8
- SWE-bench Verified 57.6
- Terminal-Bench 30
- MMLU 87.4
- IFEval 86.3
- SysBench 77.4
- MultiChallenge 42.5
- Novel logical reasoning problems expert evaluation (Z.ai) 53.4
Hallucination
- SimpleQA 14.5
Safety
- SafetyBench 87.8