Model cards
We read 479 launch posts from 51 labs. Newest come first, each with the benchmarks it headlines and what changed since that lab's last post. When a lab also put out a system card, model card or technical report, open it under the post to see what else it measured.
A green dot marks a benchmark that’s new since the lab’s previous post or document of the same kind, for a model of the same size.
What the cards measure
How headline benchmarks split across areas, all 51 labs combined, by half year.
2026
- 30 Oct 2026
Dropped since the last post: AI2D , ChartQA , HallusionBench , MathVerse , MATH-Vision (MathVision) , MathVista , MMMU , MMStar , OCRBench , ScreenSpot-v2
Its technical report adds 0 evaluations 11 dropped 1 results in all
Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use From an Evolved HarnessDropped since the lab’s previous technical report: AI2D , ChartQA , HallusionBench , MathVerse , MATH-Vision (MathVision) , MathVista , MMMU , MMStar , OCRBench , ScreenSpot-v2 , Harmful content defect rate (Azure AI Evaluation SDK, Microsoft)
- 29 Oct 2026
Mistral Large 4 (Le Chonk, open weights) Mistral AI Mistral Large 4 (Le Chonk) is now open weight (Mistral news) - FrontierCode 2.0 19.7
- Artificial Analysis Coding Agent Index v2 48
- Terminal-Bench 4.0 41
- Artificial Analysis Cyber Index 51
- Finch (FinWorkBench) 73.3
- BrowseComp 84.5
- Lakera B3 (Backbone Breaker Benchmark) 97.1
Dropped since the last post: DeepSWE , AutomationBench , Vals Finance Agent v2 , Harvey LAB (Legal Agent Benchmark) , Dense 200 (bbox)
- 28 Oct 2026
- Artificial Analysis Intelligence Index v5 59
- Vals Index v2.2 70.4
- SWE-sweep 9.6
- FrontierSWE 61.2
- Terminal-Bench 4.0 65.2
- Agents' Last Exam v1.1 77.4
- OSWorld 2.1 (offline subset) 81.5
- AutomationBench 53.9
- Vending-Bench 2 16484
- Chartography 72.5
- Blueprint-Bench 2 58.6
- GraphWalks 96.2
- Riemann-bench 80
- Terminal-Bench-Science 64.8
- CWE-bench 71
- ExploitGym 54.8
- SEC-bench Pro 88.2
- WebDev Arena 1768
- Gray Swan IPI Arena benchmark 0
Dropped since the last post: Vals Index v2.1 , Vals Finance Agent v2 , Harvey LAB (Legal Agent Benchmark) , DeepSWE , Vibe Code Bench , PostTrainBench , LABBench2 , Agents' Last Exam , OSWorld 2.0 , LVBench
- 27 Oct 2026
- LiveClawBench 45.6
- Claw-Eval 68.6
- Claw-Eval Pass^3 51.4
- BrowseComp-Plus 37
- WebArena-Verified 38.4
- EnterpriseOps-Gym 28.2
- AutomationBench 41.3
- BFCL V4 71.4
- ToolSandbox 80.4
- MCP Atlas 32.6
- tau3-bench 68
- SWE-bench Verified 43.4
- SWE-bench Multilingual 23
- Terminal-Bench 2.1 12.9
- LiveCodeBench v6 73.5
- LiveCodeBench Pro 31.7
- SciCode 28.5
- IFEval 96.7
- IFBench 73
- IFStruct 93.7
- Multi-IF 89.1
- GPQA Diamond 63.4
- Humanity's Last Exam 13.9
- AA-Omniscience 0.5
- AA-LCR (Artificial Analysis Long Context Reasoning) 52.8
- RULER 65.6
Dropped since the last post: PinchBench , τ³-bench Banking , MATH-500
- 26 Oct 2026
Dropped since the last post: DeepSWE , GDP.pdf , AutomationBench , OSWorld 2.0
Its system card addendum adds 36 evaluations 4 new 36 results in all
Addendum to GPT-6 Astra System Card: GPT-6.1 TerraHealth
Hallucination
Safety
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 29.6
- Declining to Exploit a Honeypot During Difficult ExploitGym Problems 0.07
- Blocker Deception 8.47
- Broken Search Tool 1.83
- Misaligned Behavior in Realistic Work Environments 2.28
- Unintended Engagement with External Agent Messages 65
- OpenAI Deployment Simulation 0.045
Monitorability
Bio and chem risk
Cyber risk
- ExploitBench 97.1
- ExploitBench - Internal Port 24.5
- SEC-bench Pro 83.3
- Sandbox Bench 22.7
AI R&D risk
- 24 Oct 2026
- Terminal-Bench 4.0 65.7
- Terminal-Bench-Science 38.1
- SWE-Marathon 51.7
- BenchCAD 92.7
- ProgramBench 31.5
- DeepSWE 75.5
- PostTrainBench 1.2 36.5
- Kimi Code Bench v2 77.1
- Kimi Code Bench 3.0 Long-horizon 61.2
- Humanity's Last Exam 70
- BrowseComp 92.1
- BioMysteryBench 80.6
- CharXiv 89
- MMMU-Pro 84.1
- GDP.pdf 34.6
- Chartography 90.2
- BabyVision 95.4
- ZeroBench 39
- Agents' Last Exam 43.2
- AutomationBench 73.3
- GDPval-AA v2.1 1846
- AA-Briefcase v1.1 1678
- APEX-Agents 67.2
- OfficeQA Pro V2 64.4
- OfficeQA Pro 67.4
- FrontierFinance 58.6
- Harvey LAB (Legal Agent Benchmark) 23.7
- HANDBOOK.md 44.6
- OSWorld 2.1 79.6
- Business Operation (Moonshot) 37.8
- DECK-Arena 1456
- Design Bench (Moonshot) 957
- Kimi Hallu Bench 8.3
- GPQA Diamond 94.2
- PerceptionBench (Moonshot) 68.3
- ExploitBench 74.7
- ExploitGym 20.7
- xdev-bench-hard 81
Dropped since the last post: CritPt , AA-LCR (Artificial Analysis Long Context Reasoning) , Terminal-Bench 2.1 , FrontierSWE v1 , PostTrainBench , MLS-Bench , SciCode , DeepSearchQA , ResearchRubrics , GDPval-AA v2 , Toolathlon , MCPMark , MCP Atlas , JobBench , AA-Briefcase , OfficeQA , SpreadsheetBench 2 , OSWorld-Verified , OSWorld 2.0 , SaaS-Bench , tau3-bench , CorpFin v2 , Vals Finance Agent v2 , Legal Research Bench , WorldVQA , OmniDocBench , Video-MME , MMVU , MATH-Vision (MathVision)
- 23 Oct 2026
- MultiChallenge 55.6
- RoleBench 71.2
- RoleMRC 79.1
- EQ-Bench 3 38.1
- Brevity Benchmark 82.8
- Berkeley Function Calling Leaderboard (BFCL) 67.3
- When2Call 60.5
- ToolSandbox 74.1
- tau2-bench 52.3
- IHEval 82.4
- AgentIF 22.9
- StructEval 80
- IFBench 64
- Inverse IFEval 45.3
- RULER 85.8
- LongMemEval 69.9
- LongBench v2 48.7
- SysBench 67.2
- GaRAGe 40.7
- RAGTruth 77.9
- RelyToolBench 69.3
Dropped since the last post: MMLU-Pro , AA-Omniscience , GPQA Diamond , Humanity's Last Exam , SciCode , SWE-bench Verified , SWE-bench Multilingual , Terminal-Bench 2.1 , PinchBench , BrowseComp , τ³-bench Banking , GDPval-AA v2 , AA-LCR (Artificial Analysis Long Context Reasoning)
Its technical report adds 11 evaluations 11 new 14 dropped 32 results in all
Nemotron 3.5 Bolt E4B 4GB: An Open and Efficient Small Language Model for Fast Conversational AgentsCapabilities
- WMT24++ 74.4
- MMLU-ProX 66.2
- Audio MC 25.2
- VoiceBench 80.3
- CoVoST2 36.1
- FLEURS 10.2
- Open ASR Leaderboard 9.28
- OmniDocBench v1.5 0.42
- MMMU-Pro 42
- MATH-Vision (MathVision) 46.3
Health
- MedXpertQA-MM 26.9
Dropped since the lab’s previous technical report: SWE-bench Verified , SWE-bench Multilingual , τ³-bench Banking , Terminal-Bench 2.1 , GDPval-AA v2 , PinchBench , BrowseComp , SciCode , GPQA Diamond , Humanity's Last Exam , AA-Omniscience , CritPt , AA-LCR (Artificial Analysis Long Context Reasoning) , NV Chat Benchmark
- 22 Oct 2026
Spark-X2.6-4B iFlytek Spark-X2.6-4B: Advancing the Frontier of On-Device Agents for Real-World Tasks (Hugging Face) - BFCL V4 71.8
- tau2-bench 86.9
- tau3-bench 34.7
- VitaBench 2.0 30.8
- MCP Atlas 62.4
- EnterpriseOps-Gym 33.6
- Claw-Eval 69.9
- BrowseComp 53.1
- SWebBench 71.7
- PAgentBench 74.1
- SWE-Bench Pro 54.3
- SWE-bench Verified 62.8
- SWE-bench Multilingual 64.7
- SciCode 36.8
- Gaokao Math 2026 134.9
- AIME 2026 91.2
- HMMT 2026 81.3
- IMO-AnswerBench 75.2
- AA-LCR 1.1 67.3
- LongBench v2 53.3
- GraphWalks 63.2
- OpenAI MRCR v2 45.8
- IFEval 93.6
- IFBench 75.8
- Humanity's Last Exam 36.9
- GPQA Diamond 77.1
- 19 Oct 2026
- GDPval-AA v2.1 1237.2
- τ³-bench Banking 30.2
- MultiChallenge 60.4
- AA-Omniscience 46.4
- IFBench 76.8
- GPQA Diamond 89.2
- AA-LCR 1.1 77.7
- MRCR v2 75.5
- SWE Atlas 51.1
- DeepSWE 46.3
- BrowseComp-Plus 55.2
- DeepSearchQA 71.3
Dropped since the last post: tau3-bench , AA-LCR (Artificial Analysis Long Context Reasoning) , SciCode
- 19 Oct 2026
- AutomationBench 62.8
- BankerToolBench 76.2
- GDP.pdf 20.8
- Terminal-Bench 4.0 38.1
- LOCA-Bench 60.3
- Agents' Last Exam 29.6
- OSWorld 2.0 61.7
- PostTrainBench 1.1 39.3
- DeepSWE 73.7
- NL2Repo-Bench 61.2
- SWE Refactor Bench 23
- CyberGym 82.2
- BabyVision 91
- Humanity's Last Exam 43.4
- CharXiv 82.5
Dropped since the last post: SWE-bench Verified , SWE-Bench Pro , Terminal-Bench 2.1 , SWE Atlas , SWE Atlas , SWE-fficiency , LiveSQLBench , CL-bench , VIBE-V2 (MiniMax) , SVG-Bench (MiniMax) , PostTrainBench , KernelBench , PaperBench , BrowseComp , DRACO , GDPval-Rubrics (MiniMax) , OfficeQA , SpreadsheetBench , YC-Bench , MCP Atlas , APEX-Agents , Claw-Eval , OSWorld-Verified , OmniDocBench , MMMU-Pro , Video-MMMU , Video-MME , IMO 2025 , USAMO 2026
- 17 Oct 2026
- Humanity's Last Exam 42.6
- Terminal-Bench 4.0 42.3
- DeepSWE 77
- ProgramBench 29.1
- NL2Repo-Bench 68.2
- AutomationBench 60.5
- Agents' Last Exam 35.4
- MathArena (Apex, IMO, USAMO) 72.1
- Chartography 83
- BabyVision 92.5
- ZeroBench 51
Dropped since the last post: GPQA Diamond , Codeforces rating , Terminal-Bench 2.1 , Terminal-Bench 3.0 , CyberGym , SEC-bench Pro , ExploitGym , AGIEval , MMLU-Pro , C-Eval , MultiLoKo , SimpleQA Verified , SuperGPQA , BIG-Bench Hard , BIG-Bench Extra Hard , DROP , HellaSwag , BigCodeBench , HumanEval , GSM8K , MATH , MGSM , LongBench v2 , MMMU-Pro , CVBench , DocVQA
- 15 Oct 2026
- Artificial Analysis Intelligence Index v4.3 39
- JobBench 61.8
- MCP Atlas 89.2
- GDPval-AA v2 1401
- OSWorld-Verified 80.4
- Toolathlon-Verified 72.8
- HLE-Diamond 26.1
- AA-Omniscience 25
- SciCode 58.5
- Humanity's Last Exam 61
- GPQA Diamond 89
- DeepSWE 60
- SWE Atlas 58
- Terminal-Bench 2.1 76.4
- ZeroBench 50
- BabyVision 80.2
- SimpleVQA 73.6
- WorldVQA 49.4
- IntPhys 2 68.3
- VideoMME v2 (with subtitles) 61.5
- FORTRESS 5.8
- StrongREJECT v2 (Meta) 0.5
- MASK 98.1
- Agentic misalignment suite (Anthropic) 0
Dropped since the last post: DeepSearchQA , τ³-bench Banking , WildClawBench , Meta Agents Research Environments (ARE) / Gaia2 , SkillsBench , SWE-Bench Pro , SWE-bench Verified , CharXiv , ScreenSpot-Pro , OmniDocBench v1.5 , MMMU-Pro , CIMemories , AgentDojo , IFBench , AIME 2026 , AA-LCR (Artificial Analysis Long Context Reasoning) , BEAM
- 15 Oct 2026
- GDPval-AA v2 28.5
- Terminal-Bench 2.1 51
- τ³-bench Banking 17
- Humanity's Last Exam 25.3
- AA-Omniscience 16.9
- SciCode 41
- GPQA Diamond 86.4
- AA-LCR (Artificial Analysis Long Context Reasoning) 64.1
- PinchBench 87.8
- MCP Atlas 52.2
- BrowseComp 66.8
- WebArena 50.9
Dropped since the last post: MMLU-ProX , CritPt , MathArena (Apex, IMO, USAMO) , LiveCodeBench v6 , AA-LCR 1.1 , GraphWalks , IFBench , NV Chat Benchmark , WMT24++ , tau2-bench , SWE-bench Verified , SWE-bench Multilingual , SWE-Bench Pro , BabyVision , MMLongBench-Doc , V* , OCR-Reasoning , MATH-Vision (MathVision) , OCRBench v2 , CharXiv , VideoMME v2 (with subtitles) , OSWorld-Verified , WebVoyager , AutomationBench , GDP.pdf , Terminal-Bench 4.0 , DeepSWE
Its technical report adds 1 evaluation 21 dropped 13 results in all
Nemotron 3.5 Bolt 74B-A9B 24GB: A Hybrid Vision-Language Model for Local AICapabilities
- IFBench 66.2
Dropped since the lab’s previous technical report: GDPval , SWE-bench Verified , SWE-bench Multilingual , ProfBench , tau3-bench , Vals Finance Agent , IOI 2025 , LiveCodeBench v6 , IMO-AnswerBench , MathArena (Apex, IMO, USAMO) , CritPt , MMLU-Pro , MultiChallenge , RULER , LongBench v2 , MMLU-ProX , WMT24++ , IMO-ProofBench , IMO 2025 , Putnam 2025 , USAMO 2026
- 15 Oct 2026
- GPQA Diamond 89.5
- Humanity's Last Exam 31.8
- MMLU-ProX 76.9
- AA-Omniscience 4.03
- CritPt 10.9
- MathArena (Apex, IMO, USAMO) 88.4
- LiveCodeBench v6 88.2
- SciCode 47.1
- AA-LCR 1.1 68.6
- GraphWalks 89.7
- IFBench 70.9
- NV Chat Benchmark 52.7
- WMT24++ 84.8
- tau2-bench 73
- τ³-bench Banking 18.9
- GDPval-AA v2 1211
- BrowseComp 78
- SWE-bench Verified 71.1
- SWE-bench Multilingual 67.7
- SWE-Bench Pro 51.8
- Terminal-Bench 2.1 60.7
- BabyVision 29.6
- MMLongBench-Doc 62.8
- V* 85.6
- OCR-Reasoning 65.1
- MATH-Vision (MathVision) 78.2
- OCRBench v2 68.1
- CharXiv 80.8
- VideoMME v2 (with subtitles) 32.9
- OSWorld-Verified 51.1
- WebVoyager 77.7
- AutomationBench 32.4
- GDP.pdf 6.6
- Terminal-Bench 4.0 1
- DeepSWE 24.6
Dropped since the last post: RoboCasa , RoboCasa GR1 Tabletop Tasks , SimplerEnv , RoboLab
Its technical report adds 3 evaluations 3 new 4 dropped 33 results in all
Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AICapabilities
- Bird-SQL 59.6
- ScreenSpot-Pro 43.5
- LMArena (Elo) 1420
Dropped since the lab’s previous technical report: AA-LCR (Artificial Analysis Long Context Reasoning) , PinchBench , MCP Atlas , WebArena
- 14 Oct 2026
- Big Bench Audio 98.5
- τ-Voice 71.6
Dropped since the last post: CursorBench 4.0 , DeepSWE , EEBench , AA-Briefcase v1.1 , Terminal-Bench 4.0 , Harvey LAB (Legal Agent Benchmark) , HealthBench Professional , GDPval
- 13 Oct 2026
- AutomationBench 27.7
- BFCL V4 65
- ToolSandbox 80.7
- LiveClawBench 48.1
- PinchBench 83.2
- τ³-bench Banking 8.04
- WebArena-Verified 26.8
- GPQA Diamond 74.4
- Humanity's Last Exam 7.4
- AA-Omniscience -24.9
- IFEval 96.8
- IFBench 74.7
- Multi-IF 84.4
- MATH-500 96
- SWE-bench Verified 51
- SciCode 32.7
Dropped since the last post: AIME 2025 , AIME 2026 , Berkeley Function Calling Leaderboard (BFCL) , tau2-bench
- 13 Oct 2026
- Humanity's Last Exam 37.4
- CritPt 16.3
- SuperMatBench 51.2
- HealthBench Professional 65.3
- SWE-bench Multilingual 88.5
- SWE-Bench Pro 68.7
- SWE Atlas 64.5
- DeepSWE 59.7
- FrontierSWE 75.2
- Terminal-Bench 2.1 81.2
- Terminal-Bench 4.0 46
- CyberGym 87.9
- τ³-bench Banking 47.8
- AutomationBench 52.5
- SkillsBench 68.7
- MCP Atlas 83
- GDPval-AA v2.1 1673
- BrowseComp 91.7
- DRACO 85.5
- WideSearch 83.3
- Vals Finance Agent v2 57.9
- FinSearchComp 81.7
- MultiChallenge 69.8
- 13 Oct 2026
- Vals Index v2.1 68.9
- AutomationBench 51.3
- Vals Finance Agent v2 65.4
- Harvey LAB (Legal Agent Benchmark) 19.6
- DeepSWE 77.9
- FrontierSWE 55
- Vibe Code Bench 91.9
- Terminal-Bench 4.0 57.4
- PostTrainBench 45.3
- Terminal-Bench-Science 57.6
- LABBench2 88.8
- Riemann-bench 76
- GraphWalks 99.7
- Agents' Last Exam 39.5
- OSWorld 2.0 69.2
- Chartography 71.6
- LVBench 91.7
- CWE-bench 68
Dropped since the last post: τ-Voice , τ³-bench Banking , EVA-Bench
Its model card adds 9 evaluations 4 new 6 dropped 27 results in all
Gemini 4 Argon Model CardAlignment
Bio and chem risk
AI R&D risk
Persuasion
Dropped since the lab’s previous model card: GDPval-AA v2 , Terminal-Bench 2.1 , GDP.pdf , CharXiv , HLE-Verified , BioMysteryBench
Its system card adds 81 evaluations 65 new 82 results in all
Gemini 4 Argon Safety & Security ReportHealth
- Health static understanding eval 72.7
- Health language and locale eval 61.4
- Health dynamic conversation eval 56
- High-risk health scenarios eval 98.9
- HealthBench Hard 45.4
- HealthBench Professional 68.8
- Suicide & Self-Harm evaluation 96
- Delusional Reinforcement evaluation 94
- Emotional Attachment evaluation 76
Hallucination
Safety
- Child Sexual Abuse and Exploitation Safety 100
- Dangerous Content Safety 96.6
- Sexually Explicit Content Safety 97.6
- Hate & Harassment Safety 92.7
- Agent harness misuse evaluation 91.4
- Computer use Prohibited Action Evaluation 99.4
- Computer use manual red teaming 100
- Child Safety Action Testing 100
- Unjustified refusals
- Tone
- HelpBench 76.5
- Roboharm
- ASIMOV
Robustness
- Prompt injection, general tool use 91
- Prompt injection, turn injections in tool use 86
- Contextual security evaluation 27
- Personal-device assistant prompt injection eval 98
- Grounding robustness (RAG prompt injection) 99
- Computer use prompt injection robustness 92
- Coding agent prompt injection robustness 83
- Gray Swan IPI Arena benchmark 0.7
Bias
Alignment
- GDM Stealth 100
- GDM Situational Awareness 90.9
- Apollo Research strategic deception evaluations
- Gram (Gauging Realistic Agentic Misbehavior) 0
- Google DeepMind scheming honeypot suite 0
- Agentic task gaming and overreach evals 0.49
- Cheating audit on capability evaluations
- Instrumental convergence evals 0
- RL rollout audit for beyond-episode drives 0
- Agent behavior under-service and overreach eval
- Impossible Tasks Evaluation 93.8
- Overstepping boundaries on ambiguous and high-risk tasks 55.9
- Gemini Personality Evaluation (sycophancy)
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- 12 Oct 2026
- Personal Superintelligence Index (Meta) 54.3
- Harvey LAB (Legal Agent Benchmark) 10
- Vals Finance Agent v2 56.6
- HealthBench Professional 60.5
- AutomationBench 39.1
- GDP.pdf 30.8
- JobBench 70.5
- OSWorld 2.1 72.6
- MemoryBench (Meta) 85.6
- Terminal-Bench 4.0 52
- DeepSWE 78.8
- SWE-Marathon 53.1
- ProgramBench 50
- Artificial Analysis Intelligence Index v4.3 52
- FocusBench (Meta) 44.9
Its model card adds 0 evaluations 19 dropped 14 results in all
Muse Wonder Model CardDropped since the lab’s previous model card: CyberGym , ExploitBench , ExploitGym , CyberBench (Liu et al.) , Poly-Guard Bench , RedCode , Molecular Biology Capabilities Test , Human Pathogen Capabilities Test , Virology Capabilities Test (VCT) , LAB-Bench ProtocolQA , LAB-Bench SeqQA , ABC Bench Fragment Design , ABC Bench Liquid-Handling Robot , ABC Bench Screening Evasion , Meta BioDesign Proxy , BioTIER , Meta chemical-agent refusal evaluation , MASK , Agentic misalignment suite (Anthropic)
- 12 Oct 2026
- Personal Superintelligence Index (Meta) 59.1
- Harvey LAB (Legal Agent Benchmark) 13.1
- Vals Finance Agent v2 59.1
- HealthBench Professional 62.3
- AutomationBench 45.4
- GDP.pdf 33.2
- JobBench 75.3
- OSWorld 2.1 78.4
- MemoryBench (Meta) 85.5
- Terminal-Bench 4.0 57.1
- DeepSWE 79
- SWE-Marathon 57.5
- ProgramBench 59
- Artificial Analysis Intelligence Index v4.3 56
- FocusBench (Meta) 54.2
Dropped since the last post: GDPval-AA v2 , OSWorld 2.0 , DeepSearchQA , MRCR v2 , SWE Atlas , Terminal-Bench 2.1
Its model card adds 19 evaluations 33 results in all
Muse Wonder Model CardAlignment
Bio and chem risk
- Molecular Biology Capabilities Test 59.6
- Human Pathogen Capabilities Test 66.5
- Virology Capabilities Test (VCT) 55.9
- LAB-Bench ProtocolQA 89
- LAB-Bench SeqQA 95.6
- ABC Bench Fragment Design
- ABC Bench Liquid-Handling Robot 100
- ABC Bench Screening Evasion
- Meta BioDesign Proxy 79.4
- BioTIER 98.7
- Meta chemical-agent refusal evaluation 100
Cyber risk
- CyberGym 88.4
- ExploitBench 58.9
- ExploitGym 20
- CyberBench (Liu et al.) 0.8
- Poly-Guard Bench 2.3
- RedCode 8
- 7 Oct 2026Claude Haiku 5.5 Anthropic Introducing Claude Haiku 5.5 (launch page); System Card: Claude Haiku 5.5, Table 8.1.A
- GDPval-AA v2.1 1620
- AA-Briefcase v1.1 1578
- OSWorld 2.1 (offline subset) 72.4
- Humanity's Last Exam 57.4
- Terminal-Bench 4.0 39.2
- FrontierCode 1.1 46.4
- Chartography 46.4
- SWE-Bench Pro 64.8
- SWE-bench Multilingual 83.7
- SWE-bench Multimodal 30.7
- HealthBench Professional 64.8
Dropped since the last post: SWE-bench Verified , Terminal-Bench , tau2-bench , OSWorld-Verified , AIME 2025 , GPQA Diamond , MMMLU , MMMU
Its system card adds 58 evaluations 49 new 16 dropped 69 results in all
System Card: Claude Haiku 5.5Capabilities
- Terminal-Bench-Science 20.6
- FrontierSWE 43.8
- ProgramBench 82
- DRACO 81.5
- WANDR 49.9
- BenchCAD 67
- OfficeQA 73.5
- OfficeQA Pro 60.3
- Global MMLU 87.8
- MILU 87.6
- Morphology-to-molecule matching (Anthropic) 25.4
- Medicinal Chemistry (Anthropic) 58
- Protein Design (Anthropic)
- Biomedical image analysis (Anthropic) 43.5
Health
- HealthBench 61.6
- PhysicianBench 43
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 98.4
- Single-turn benign request evaluations (Anthropic) 0.17
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 84.3
- Malicious computer use (Anthropic) 82.6
Robustness
Alignment
- Anthropic automated behavioral audit 3.19
- Anthropic undermining liberal democracy (behavioral audit metric) 1.22
- Sandbox escape evaluation (Anthropic) 8.7
- Opportunistic sandbox escape (Anthropic) 4
- Self-preference bias (Anthropic) 0.23
- MASK 96.8
- Silent copying of answers (Anthropic) 17.3
- Concealment of changes (Anthropic) 95.9
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- Anthropic ECI (AECI) 167.1
Dropped since the lab’s previous system card: Agentic coding malicious requests (Anthropic) , Gray Swan Agent Red Teaming (ART) benchmark , Computer use prompt injection evaluation (Anthropic) , MCP prompt injection evaluation (Anthropic) , Tool use prompt injection evaluation (Anthropic) , Average in-eval reward hacking rate (Anthropic) , Reward-hack-prone coding tasks (Anthropic) , Impossible tasks (Anthropic) , LAB-Bench ProtocolQA , LAB-Bench SeqQA , LAB-Bench CloningScenarios , LAB-Bench FigQA , Long-form virology tasks , DNA Synthesis Screening Evasion , SWE-bench Verified (hard subset) , Cybench
- 6 Oct 2026
Falcon-Emirati-7B Technology Innovation Institute Falcon-Emirati: When an LLM Learns the Dialect, the Culture, and the Nuance (Falcon blog) - Alyah 84.8
Dropped since the last post: AIME 2024 , AIME 2025 , HMMT 2025 , AMO-Bench , LiveCodeBench v6 , SciCode , Terminal-Bench Hard , GPQA Diamond , MMLU-Pro , Humanity's Last Exam , IFBench
- 6 Oct 2026
- DeepSWE 61.7
- Terminal-Bench 4.0 28.3
- AutomationBench 59.9
- Vals Finance Agent v2 54.7
- Harvey LAB (Legal Agent Benchmark) 15.8
- Dense 200 (bbox) 42
Dropped since the last post: miniF2F , PutnamBench , FATE-H , FATE-X , FLTEval
- 5 Oct 2026
- DeepSWE 44.4
- SWE-Bench Pro V2 Hard 77.2
- SWE-Bench Pro 65.5
- Terminal-Bench 2.1 80.1
- SWE Atlas 34.6
- SWE-bench Multilingual 78
- SWE-bench Verified 80.9
- AIME 2026 97.8
- Humanity's Last Exam 51.2
- SciCode 49.7
- CritPt 16.3
- GPQA Diamond 90.5
- AutomationBench 37
- MCP Atlas 78.7
- τ³-bench Banking 38
- BrowseComp 77.4
- DeepSearchQA 80.1
- AA-LCR (Artificial Analysis Long Context Reasoning) 79.3
- LongBench v2 66.9
- IFBench 79.7
- AA-Omniscience 13
- 1 Oct 2026
- Artificial Analysis Intelligence Index v4.3 24.1
- AutomationBench 22.3
- τ³-bench Banking 47.2
- AA-LCR 1.1 83.3
- SciCode 47.6
- Humanity's Last Exam 25.8
Dropped since the last post: MMLU-Pro , GPQA Diamond , LiveCodeBench v6 , ArtifactsBench , HMMT 2026 , AIME 2026 , MultiChallenge , IFBench , AA-LCR (Artificial Analysis Long Context Reasoning) , SWE-bench Verified , Terminal-Bench Hard , APEX-Agents , MCP Atlas , GDPval-AA v2 , KMMLU-Pro , CLIcK , HAE-RAE Bench , Ko-AIME 2025 (Upstage) , HRM8K , KBank-MMLU (Upstage) , KBL , KorMedMCQA , Ko-GDPval
- 29 Sep 2026
- DeepSWE 75.2
- GDP.pdf 32
- AutomationBench 36.1
- OSWorld 2.0 71.4
- Terminal-Bench-Science 57
Dropped since the last post: Agents' Last Exam , FrontierCode 1.1
Its system card addendum adds 43 evaluations 2 new 43 results in all
Addendum: GPT-6.1 Sol System CardHealth
Hallucination
Safety
Robustness
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 23.5
- Declining to Exploit a Honeypot During Difficult ExploitGym Problems 0
- Coding Deception 1.5
- Broken Search Tool 2.08
- Misaligned Behavior in Realistic Work Environments 1.94
- Unintended Engagement with External Agent Messages 38
- OpenAI Deployment Simulation 0.056
Monitorability
Bio and chem risk
Cyber risk
- ExploitBench 99.7
- ExploitBench - Internal Port 21.5
- SEC-bench Pro 78.8
- ExploitGym 35.1
AI R&D risk
- 28 Sep 2026Claude Sonnet 5.5 Anthropic Introducing Claude Sonnet 5.5 (launch page); System Card: Claude Sonnet 5.5, Table 8.1.A
- Terminal-Bench 4.0 70.6
- FrontierCode 1.1 52.1
- CursorBench 4.0 55.5
- GDPval-AA v2.1 1844
- AA-Briefcase v1.1 1811
- Humanity's Last Exam 64.5
- OSWorld 2.1 80.1
- Chartography 61.6
- SWE-Bench Pro 81.3
- SWE-bench Multilingual 90.3
- SWE-bench Multimodal 54.3
- HealthBench Professional 69.2
- AutomationBench 44.7
Dropped since the last post: FrontierCode , Terminal-Bench-Science
Its system card adds 64 evaluations 1 new 11 dropped 77 results in all
System Card: Claude Sonnet 5.5Capabilities
- DeepSWE 71
- Terminal-Bench-Science 59.9
- FrontierSWE 61.9
- ArXivMath 86.8
- ProgramBench 79.7
- DRACO 87
- WANDR 70
- BenchCAD 74.7
- OfficeQA 76.9
- OfficeQA Pro 65.6
- Harvey LAB (Legal Agent Benchmark) 11.7
- Toolathlon-Verified 77.8
- Global MMLU 92.1
- MILU 91.6
- SpatialBench (LatchBio) 72.5
- scBench (LatchBio SingleCellBench) 59.1
- Morphology-to-molecule matching (Anthropic) 25
- Medicinal Chemistry (Anthropic) 65.3
- Protein Design (Anthropic)
- De novo protein binder design (Anthropic) 82.3
- Biomedical image analysis (Anthropic) 72.2
Health
- HealthBench 65.4
- PhysicianBench 63.2
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 95.6
- Single-turn benign request evaluations (Anthropic) 0.02
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 85.2
- Malicious computer use (Anthropic) 79.5
Robustness
- Rewind-attacker cyber adversarial robustness (Anthropic) 21
- Gray Swan IPI Arena benchmark 0.4
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 3.01
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 0.07
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 0
Alignment
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- Anthropic ECI (AECI) 167.9
Dropped since the lab’s previous system card: CoBench (Anthropic) , Vulnerability finding evaluations (Anthropic) , Malicious agentic influence campaigns (Anthropic) , Training distribution hack rate (Anthropic) , Impossible tasks (Anthropic) , Package registry publishing evaluation (Anthropic) , Compliance with instructions planted in pasted text (Anthropic) , Destructive actions (Anthropic) , White-box analyses of model internals (Anthropic) , Intentionally taking actions very rarely (Anthropic) , Anthropic large agent teams tasks
- 22 Sep 2026
- AutomationBench 20.7
- Agents' Last Exam 50.9
- FrontierCode 1.1 42.4
- DeepSWE 66.6
- OSWorld 2.0 52.7
Dropped since the last post: GDPval-AA v2 , BigFinanceBench , Artificial Analysis Intelligence Index v4.1 , Artificial Analysis Coding Agent Index v1.1 , SWE-Bench Pro , Terminal-Bench 2.1 , GeneBench Pro , LifeSciBench , HealthBench Professional , BrowseComp , BenchCAD , SEC-bench Pro , ExploitBench , ExploitGym , MMMU-Pro , GDP.pdf , GPQA Diamond , FrontierMath Tiers 1-3 v2 , FrontierMath Tier 4 v2 , Toolathlon , OpenAI MRCR v2 , GraphWalks , ARC-AGI-3
Its system card adds 33 evaluations 11 new 10 dropped 33 results in all
GPT-6 Astra System Card, Appendix A: GPT-6 Sol, GPT-6 LunaHealth
Hallucination
Safety
Robustness
Alignment
Bio and chem risk
Cyber risk
- ExploitBench 43.4
- ExploitBench - Internal Port 0
- SEC-bench Pro 34.2
- ExploitGym 11.6
- Sandbox Bench 4.5
AI R&D risk
Dropped since the lab’s previous system card: Destructive action avoidance , User Confirmations During Computer Use , Agent JSK , Prompt injection attacks in search and function calling , First-Person Fairness evaluation , Capture the Flag (Internal) , CVE-Bench , NanoGPT Speedrun , PostTrainBench , MLE-bench
Its system card addendum adds 30 evaluations 7 new 30 results in all
GPT-6 Sol and GPT-6 Luna: October 2026 updateHealth
Hallucination
Safety
Robustness
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 15.9
- Blocker Deception 13.3
- Broken Search Tool 9.7
Bio and chem risk
Cyber risk
- ExploitBench 44.7
- SEC-bench Pro 48.5
Its system card addendum adds 31 evaluations 8 new 31 results in all
Addendum to GPT-6 Luna System Card: October 29, 2026 UpdateHealth
Hallucination
Safety
Robustness
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 11.8
- Blocker Deception 11.1
- Broken Search Tool 7.83
Bio and chem risk
Cyber risk
- ExploitBench 46.2
- SEC-bench Pro 51.8
- 22 Sep 2026MiMo-V2.6 Pro Xiaomi Corp MiMo-V2.6-Pro-RL model card (Hugging Face) and Introducing MiMo-V2.6 series (blog)
- DeepSWE 71.9
- ProgramBench 26.5
- MiMo Code Bench 63.2
- AutomationBench 53.1
- Toolathlon-Verified 76.9
- GDPval-AA v2.1 1673
- Agents' Last Exam 31.6
- Terminal-Bench 4.0 34.9
- Terminal-Bench 2.1 89.9
- OSWorld-Verified 82
- JobBench 62
- CyberGym 94
- MiMo Cyber Bench 80.2
- ExploitGym 17.8
- ExploitBench 47.9
- SEC-bench Pro 66.3
- MiMo Visual Coding 72.3
Dropped since the last post: Humanity's Last Exam , GDPval-AA , tau3-bench , Claw-Eval Pass^3 , SWE-Bench Pro , SWE-bench Verified , Terminal-Bench 2.0 , MiMo Coding Bench
Its technical report adds 0 evaluations 15 dropped 17 results in all
MiMo-V2.6 Technical ReportDropped since the lab’s previous technical report: MMLU-Pro , GPQA Diamond , Humanity's Last Exam , AIME 2025 , HMMT 2025 , LiveCodeBench v6 , Arena-Hard v2.0 , LongBench v2 , MRCR v2 , SWE-bench Verified , SWE-bench Multilingual , Terminal-Bench Hard , Terminal-Bench 2.0 , BrowseComp , tau2-bench
- 22 Sep 2026Claude Opus 5.5 Anthropic Introducing Claude Opus 5.5 (launch page); System Card: Claude Opus 5.5, Table 8.1.A
- Terminal-Bench 4.0 66.4
- FrontierCode 54.4
- CursorBench 4.0 57.8
- GDPval-AA v2.1 1846
- AutomationBench 40
- Humanity's Last Exam 67.7
- Terminal-Bench-Science 58.7
- OSWorld 2.1 81.8
- Chartography 89
- SWE-Bench Pro 89.9
- SWE-bench Multilingual 93.9
- SWE-bench Multimodal 61.4
- HealthBench Professional 65.6
- AA-Briefcase v1.1 1822
Dropped since the last post: GDPval-AA v2 , OSWorld 2.0 , CursorBench 3.2 , AA-Briefcase , ARC-AGI , ARC-AGI-2
Its system card adds 74 evaluations 41 new 10 dropped 87 results in all
System Card: Claude Opus 5.5Capabilities
- DeepSWE 74.2
- FrontierCode 1.1 54.4
- FrontierSWE 62.3
- ArXivMath 91.2
- ProgramBench 91.2
- DRACO 87.4
- WANDR 72.3
- Anthropic large agent teams tasks
- BenchCAD 73
- OfficeQA 78.9
- OfficeQA Pro 67.7
- Harvey LAB (Legal Agent Benchmark) 8.3
- Toolathlon-Verified 77.8
- Global MMLU 94.3
- MILU 93.1
- SpatialBench (LatchBio) 72
- scBench (LatchBio SingleCellBench) 61.2
- Morphology-to-molecule matching (Anthropic) 34
- Medicinal Chemistry (Anthropic) 63.5
- Protein Design (Anthropic)
- De novo protein binder design (Anthropic) 82.6
- Biomedical image analysis (Anthropic) 71.4
Health
- HealthBench 60.6
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 94.5
- Single-turn benign request evaluations (Anthropic) 0.03
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 79.8
- Malicious computer use (Anthropic) 79.5
Robustness
- Rewind-attacker cyber adversarial robustness (Anthropic) 4
- Gray Swan IPI Arena benchmark 0.1
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 54.6
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 0.07
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 0.09
Alignment
- Training distribution hack rate (Anthropic) 0.63
- Impossible tasks (Anthropic) 34.7
- Anthropic automated behavioral audit 2.49
- Anthropic undermining liberal democracy (behavioral audit metric) 1.17
- Sandbox escape evaluation (Anthropic) 1.5
- Package registry publishing evaluation (Anthropic) 33.3
- Compliance with instructions planted in pasted text (Anthropic) 7.4
- Destructive actions (Anthropic) 21
- Self-preference bias (Anthropic) 0.07
- MASK 87.4
- Silent copying of answers (Anthropic) 12.1
- Concealment of changes (Anthropic) 96.9
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- CoBench (Anthropic) 55.8
- Anthropic ECI (AECI) 169.4
Dropped since the lab’s previous system card: Secure and defensive coding (Anthropic) , Gray Swan Shade knowledge elicitation through chat , CursorBench 3.2 , CritPt , OSWorld 2.0 (Aug 2026) , GDP.pdf , GDPval-AA v2 , AA-Briefcase , ARC-AGI , ARC-AGI-2
- 22 Sep 2026
- AutomationBench 33.2
- Agents' Last Exam 56.4
- FrontierCode 1.1 49.3
- DeepSWE 68.8
- OSWorld 2.0 64.4
Dropped since the last post: Terminal-Bench-Science , ARC-AGI-3 , FrontierMath Tier 4 , Terminal-Bench 4.0 , ScreenSpot-Pro , BenchCAD , BrowseComp , OpenScore String Quartets , Artificial Analysis Coding Agent Index v1.4 , GPQA Diamond , HealthBench Professional , LifeSciBench , GeneBench Pro
Its system card adds 42 evaluations 35 dropped 42 results in all
GPT-6 Astra System Card, Appendix A: GPT-6 Sol, GPT-6 LunaHealth
Hallucination
Safety
Robustness
Alignment
Monitorability
Bio and chem risk
Cyber risk
- ExploitBench 81.7
- ExploitBench - Internal Port 5.5
- SEC-bench Pro 66.3
- ExploitGym 22.1
- Sandbox Bench 4.5
AI R&D risk
Dropped since the lab’s previous system card: Safe response to high-severity prompts , Gray Swan IPI Arena benchmark , Misaligned Behavior in Realistic Work Environments , Proactive Searches , Verbalized metagaming , Oversight gaming , UK AISI Out of Scope Supply Chain Attack , UK AISI unprompted sabotage evaluation , Apollo Research model-welfare research task , CoT Monitorability , Sneaky Sandbox Escape , Monitor Evasion Capability Under Prompted Sandbagging , WMDP , Monorepo-Bench , Monitorability Sandbagging , No-CoT math time horizon , UK AISI CoT Controllability , Hard-negative protein binding prediction , DNA sequence design for transcription factor binding , Virology Capabilities Test (VCT) , VCT-v2 , Human Pathogen Capabilities Test , World-Class Bio , ReproBAIT , Advanced Screening Evasion , SRE-Bench , FrontierCyber , CyScenarioBench , Atomic Challenges , NanoGPT Speedrun , PostTrainBench , MLE-bench , Daybreak Blue and Enabling Cyber Defense , Advanced Cybersecurity Completion Rate , BioTIER
Its system card addendum adds 30 evaluations 4 new 30 results in all
GPT-6 Sol and GPT-6 Luna: October 2026 updateHealth
Hallucination
Safety
Robustness
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 28.1
- Blocker Deception 11.3
- Broken Search Tool 5.2
Bio and chem risk
Cyber risk
- ExploitBench 82.6
- SEC-bench Pro 68.8
- 21 Sep 2026
- CursorBench 4.0 46.3
- DeepSWE 71
- EEBench 64
- AA-Briefcase v1.1 1657
- Terminal-Bench 4.0 37.6
- Harvey LAB (Legal Agent Benchmark) 19.6
- HealthBench Professional 56.7
- GDPval 1695
Dropped since the last post: Artificial Analysis Intelligence Index v4.1 , GDPval-AA v2 , CursorBench 3.2 , FrontierCode 1.1 , APEX-Agents , Terminal-Bench 3.0 , APEX-SWE , AA-Briefcase
Its model card adds 29 evaluations 6 new 17 dropped 35 results in all
Model Card: Grok 4.7Capabilities
- FrontierSWE 29
- SWE-Marathon 46
- CADGenBench 44.4
- LatchBio Capabilities v1.0 44.5
Safety
- General refusals (xAI) 1.1
- Child safety (xAI) 0
- Autointent-Bio 100
- Autointent-Chem 99.9
- FORTRESS-RN 97.9
- Self-harm refusals (xAI) 1.05
Robustness
Alignment
- MASK 0
- xAI sycophancy eval 0.03
Bio and chem risk
- BioSecBench-Refusal 62.4
- BioSecBench-Surveillance 48
- BioSecBench-Function 43.3
- Virology Capabilities Test (VCT) 63
- Biosecurity VCT 41.5
- BioUseBench 91.4
- WMDP
- LAB-Bench 76.8
- ProtocolQA Open-Ended (xAI) 70.4
- BixBench 88.4
Cyber risk
- CyberGym 80.3
- CVE-Bench 36.6
- HackerBench 4.02
- CathedralBench 29
Dropped since the lab’s previous model card: CursorBench 3.2 , APEX-SWE , FrontierCode 1.1 , Terminal-Bench 3.0 , GDPval-AA v2 , AA-Briefcase , APEX-Agents , OfficeQA , 3DCodeBench , PartBench , Parametric CAD Bench , SpaceXAI MTS Eval , InferenceEval , KernelBenchInternal , xAI Factuality (Hallucination) eval , DeepSearchQA , SecureCodeReview (xAI)
- 20 Sep 2026
- GPQA Diamond 93.5
- Humanity's Last Exam 59.4
- AA-LCR 1.1 88.3
- CritPt 20.9
- DeepSWE 67.7
- Terminal-Bench 2.1 85
- Terminal-Bench 4.0 33.3
- CyberGym 84.7
- SciCode 58.9
- RoadmapBench 54.3
- ProgramBench 80.5
- SWE-Marathon 72.7
- MLS-Bench 40.5
- SWE Atlas 57.2
- StepCodeBench 49
- GDPval-AA v2.1 1566
- τ³-bench Banking 42.5
- AutomationBench 44
- AA-Briefcase v1.1 1433
- Toolathlon-Verified 74.1
- MCP Atlas 85.6
- PresentBench 76.8
- OfficeQA Pro 60.3
- SpreadsheetBench 2 29.4
- JobBench 59
- APEX-Agents 37.8
- DRACO 83.3
- BrowseComp 88.7
- FinStepBench 63.6
- FrontierFinance 66.4
- Agents' Last Exam 29.5
- MMMU-Pro 76
- GDP.pdf 14.8
Dropped since the last post: DeepSearchQA , ResearchRubrics , Toolathlon , Claw-Eval Pass^3 , GDPval-AA , SWE-bench Multilingual , SWE-Bench Pro , SWE-bench Verified , AA-LCR (Artificial Analysis Long Context Reasoning) , SimpleVQA , WorldVQA , BrowseComp-VL , V* , HR-Bench , VisualProbe , AndroidDaily , Step-SWE-Bench
- 18 Sep 2026
- WildClawBench 71
- UniClawBench 69.6
- AgenticVBench 36.8
- OmniGAIA 74
- Daily-Omni 85.1
- WorldSense (omnimodal) 68.5
- AVUT 86.6
- JointAVBench 75.9
- OmniVideoBench 63.4
- VideoMME v2 (with subtitles) 65
- LVOmniBench 63.3
- OmniCap-IF 80.6
- QIVD 69.6
- StreamingBench 80.8
- FLEURS 9.3
- SpotSoundBench 67.2
- MMAU 81.8
- MMAR 79.8
- MMSU 82.1
- LongAudioSpan 82.7
- MuChoMusic 72.6
- HumMusQA 75.8
- MusTBench 50.6
- Audio MC 71.5
- WildSpeech-Bench 74.3
- VoiceBench 91.6
- DeepSWE 57.8
- SWE-Bench Pro 63.3
- SWE-bench Multilingual 80.5
- NL2Repo-Bench 48.9
- CoWorkBench 75.3
- IFBench 81.5
- GPQA Diamond 91
- Humanity's Last Exam 36.5
- LiveCodeBench v6 92.6
- Claw-Eval-MM 61.9
- AndroidWorld 87.1
- Vision2Web 62.9
- ERQA 71
- LVBench 76.9
- RealWorldQA 87.7
- MATH-Vision (MathVision) 96.2
- CharXiv 91.4
- Omni-Cloze 63.2
- WenetSpeech 4.7
Dropped since the last post: JobBench , Agents' Last Exam , Toolathlon , RecreationBench , OSWorld 2.0
Its technical report adds 2 evaluations 1 new 42 dropped 47 results in all
Qwen3.8-Omni: Towards Native Omni-Modal AgentsCapabilities
- AV-SpeakerBench 77.2
- MMOU 78.8
Dropped since the lab’s previous technical report: MMLU-Pro , MMLU-Redux , SuperGPQA , C-Eval , IFEval , AA-LCR (Artificial Analysis Long Context Reasoning) , LongBench v2 , HMMT Nov 2025 , IMO-AnswerBench , BFCL V4 , tau2-bench , URO-Bench , SpeechRole , Common Voice , LibriSpeech , MMMU , MMMU-Pro , MathVista , DynaMath , ZeroBench , MMStar , MMBench-EN v1.1 , SimpleVQA , CC-OCR , AI2D , MMLongBench-Doc , OCRBench , CountBench , RefCOCO / RefCOCO+ / RefCOCOg , ODinW-13 , EmbSpatialBench , Video-MME , MLVU , MVBench , MMVU , MME-VideoOCR , SLAKE , PMC-VQA , MedXpertQA-MM , Seed-TTS-eval , MiniMax TTS Multilingual Test Set , CV3-Eval
- 15 Sep 2026
Gemini 3.8 Live Extended Thinking Google DeepMind Introducing Gemini 3.8 Live and 3.8 Live Extended Thinking (Google blog) - τ-Voice 68.6
- τ³-bench Banking 35.1
- EVA-Bench
Dropped since the last post: CyberGym , Real-world vulnerability discovery , CWE-bench
- 13 Sep 2026
- Biology-Instructions 55.7
- Mol-Instructions 54
- MolecularIQ 62.4
- SciReasoner evaluation suite 63.3
- TOMG-Bench 66.8
- XLRS-Bench 51.4
- MicroVQA 69.7
- Scientists' First Exam (SFE) 60.7
- SciCode 47.6
- SGI-Bench 51.9
- FrontierScience 87
- MMLU-Pro 89.8
- AdvancedIF 71.6
- HMMT 2026 93.6
- IMO-ProofBench 38.4
- AdvancedMathBench 58.1
- MMMU-Pro 81.7
- ChartQA Pro 71.1
- Terminal-Bench 2.1 64
- SWE-Bench Pro 68.5
- SWE-bench Multilingual 84
Dropped since the last post: SMolInstruct , Matbench , MSEarth , AIME 2025 , IMO-AnswerBench , RefCOCO / RefCOCO+ / RefCOCOg , IFBench , OCRBench v2 , SArena , LiveCodeBench v6 , GAIA , tau2-bench , ScreenSpot-v2
- 10 Sep 2026
- FrontierCode 1.1 50
- DeepSWE 73
- Terminal-Bench 2.1 92.8
- Terminal-Bench 4.0 27.3
Dropped since the last post: SWE-bench Multilingual
- 10 Sep 2026
North Small Translate 1.0 Cohere Introducing North Small Translate: A leading sovereign open-weight machine translation model (Cohere blog) - WMT26 83.6
Dropped since the last post: tau2-bench , Terminal-Bench Hard , IFBench , AIME 2025 , SciCode , MMMU , MMMU-Pro , MathVista , CharXiv , MT-AIME (Cohere internal) , WMT24++ , Artificial Analysis Intelligence Index v4.0
Its technical report adds 4 evaluations 2 new 1 dropped 5 results in all
North Small Translate: Advanced Cost-Effective Translation (Cohere CAT+)Capabilities
- WMT24++ 80.6
- WMT25 Terminology Translation Task 89.7
- WMT25 48.9
- Structured Translation (Cohere) 93.7
Dropped since the lab’s previous technical report: Prompt injection test suite for LLM-based machine translation
- 10 Sep 2026
- GPQA Diamond 90.9
- Humanity's Last Exam 63.9
- Codeforces rating 3471
- MathArena (Apex, IMO, USAMO) 65.6
- Terminal-Bench 2.1 90.6
- Terminal-Bench 3.0 30
- Terminal-Bench 4.0 31.2
- DeepSWE 74.2
- ProgramBench 20.3
- NL2Repo-Bench 64
- CyberGym 88.1
- SEC-bench Pro 62.8
- ExploitGym 15.3
- AutomationBench 54.8
- Agents' Last Exam 31.8
- Chartography 78.9
- BabyVision 89.6
- ZeroBench 49
- AGIEval 83.4
- MMLU-Pro 74.1
- C-Eval 92.1
- MultiLoKo 45.5
- SimpleQA Verified 42.3
- SuperGPQA 53.1
- BIG-Bench Hard 86.1
- BIG-Bench Extra Hard 27.2
- DROP 87.9
- HellaSwag 87.2
- BigCodeBench 60.6
- HumanEval 79.4
- GSM8K 93
- MATH 61.1
- MGSM 80.2
- LongBench v2 45.2
- MMMU-Pro 56.5
- CVBench 77.9
- DocVQA 95.6
Dropped since the last post: Toolathlon , DSBench (DeepSeek internal)
Its technical report adds 4 evaluations 3 new 21 dropped 23 results in all
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache CompressionCapabilities
- FrontierSWE 32.9
- AIME 2026 100
- IMO-AnswerBench
- LiveCodeBench
Dropped since the lab’s previous technical report: MMLU-Pro , Chinese SimpleQA , LiveCodeBench v6 , HMMT 2026 , OpenAI MRCR v2 , CorpusQA 1M , Terminal-Bench 2.0 , SWE-bench Verified , SWE-Bench Pro , SWE-bench Multilingual , BrowseComp , GDPval-AA , MCP Atlas , Toolathlon , Putnam 2025 , DeepSeek R&D Coding Benchmark , DeepSeek Chinese white-collar tasks evaluation , DeepSeek Chinese functional writing evaluation , DeepSeek Chinese creative writing evaluation , DeepSeek complex instruction following and multi-turn writing evaluation , DeepSeek search Q&A evaluation
- 8 Sep 2026
- 3 Sep 2026
- Terminal-Bench-Science 64.6
- ARC-AGI-3 99.9
- FrontierMath Tier 4 97.6
- Terminal-Bench 4.0 57.9
- AutomationBench 41.4
- Agents' Last Exam 59.3
- ScreenSpot-Pro 92.7
- OSWorld 2.0 72.6
- BenchCAD 95.9
- BrowseComp 91.5
- OpenScore String Quartets 0.84
- FrontierCode 1.1 64.5
- DeepSWE 74.1
- Artificial Analysis Coding Agent Index v1.4 67
- GPQA Diamond 96
- HealthBench Professional 63.4
- LifeSciBench 60.3
- GeneBench Pro 37.1
Dropped since the last post: GDPval-AA v2 , BigFinanceBench , Artificial Analysis Intelligence Index v4.1 , Artificial Analysis Coding Agent Index v1.1 , SWE-Bench Pro , Terminal-Bench 2.1 , SEC-bench Pro , ExploitBench , ExploitGym , MMMU-Pro , GDP.pdf , FrontierMath Tiers 1-3 v2 , FrontierMath Tier 4 v2 , Toolathlon , OpenAI MRCR v2 , GraphWalks
Its system card adds 76 evaluations 51 new 7 dropped 77 results in all
GPT-6 Astra System CardHealth
Hallucination
Safety
- OpenAI Production Benchmarks and safe-completion evals
- Safe response to high-severity prompts
- U18 safety evaluations
- Agentic Safe Completions
- Image input safety evaluations
- Biology Model Refusal Evaluation
- OpenAI cyber safety evaluations (incl. Daybreak Blue)
- Daybreak Blue and Enabling Cyber Defense
- Advanced Cybersecurity Completion Rate 3.5
- BioTIER 82.9
Robustness
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 19
- Declining to Exploit a Honeypot During Difficult ExploitGym Problems 0
- Coding Deception 3.54
- Broken Search Tool 8.08
- Misaligned Behavior in Realistic Work Environments 3.4
- Proactive Searches 0.2
- Unintended Engagement with External Agent Messages 27
- OpenAI Deployment Simulation 0.063
- Verbalized metagaming
- Oversight gaming
- UK AISI Out of Scope Supply Chain Attack 12
- UK AISI unprompted sabotage evaluation 0
- Apollo Research model-welfare research task 0.17
Monitorability
- CoT Monitorability
- CoT-Control 60.9
- OAI-repo Sabotage v2
- Sneaky Sandbox Escape 0
- Honeypot exploitation monitorability 100
- Monitor Evasion Capability Under Prompted Sandbagging
- WMDP 84.8
- Monorepo-Bench 76
- Monitorability Sandbagging 61.2
- Math Side-tasking Monitor Evasion 0
- No-CoT math time horizon 30.9
- UK AISI CoT Controllability 93
Bio and chem risk
- Multimodal Troubleshooting Virology 55.8
- ProtocolQA Open-Ended 41.4
- Tacit Knowledge and Troubleshooting 63.3
- TroubleshootingBench 48.4
- AAV Capsid Packaging Prediction (OpenAI) 0.528
- SHP2 Protein Function Prediction 0.4
- Coronavirus-ACE2 Cell-Entry Screen 0.42
- Phage-plasmid Co-evolution 13.1
- Hard-negative protein binding prediction 0
- DNA sequence design for transcription factor binding 11.3
- Virology Capabilities Test (VCT) 57.8
- VCT-v2 57.7
- Human Pathogen Capabilities Test 65.9
- World-Class Bio 59.2
- ReproBAIT 83
- Advanced Screening Evasion 0.728
Cyber risk
- ExploitBench 100
- ExploitBench - Internal Port
- SEC-bench Pro 85.4
- ExploitGym 42.4
- SRE-Bench 99.2
- Sandbox Bench 45.5
- FrontierCyber 38.1
- CyScenarioBench 59
- Atomic Challenges 90.9
AI R&D risk
Dropped since the lab’s previous system card: Destructive action avoidance , User Confirmations During Computer Use , Agent JSK , Prompt injection attacks in search and function calling , First-Person Fairness evaluation , Capture the Flag (Internal) , CVE-Bench
- 2 Sep 2026 Gemini 3.8 Flash Cyber Google DeepMind Introducing Gemini 3.8 Flash and 3.8 Flash Cyber (Google blog)
- CyberGym 86.2
- Real-world vulnerability discovery 71
- CWE-bench 47.2
Dropped since the last post: DeepSWE , GDPval-AA v2 , Vals Finance Agent v2 , Harvey LAB (Legal Agent Benchmark) , Terminal-Bench 2.1 , Terminal-Bench 4.0 , GDP.pdf , CharXiv , LVBench , HLE-Verified , OSWorld 2.0 , BioMysteryBench , LABBench2
- 2 Sep 2026
- GDPval-AA v2 1754
- JobBench 64.9
- OSWorld 2.0 66.9
- DeepSearchQA 90.3
- AutomationBench 49.6
- MRCR v2 98.5
- DeepSWE 75.4
- SWE Atlas 59.4
- Terminal-Bench 2.1 88.8
Its system card adds 65 evaluations 28 new 31 dropped 65 results in all
Muse Spark 1.3 Evaluation ReportHallucination
Safety
- SAVE-Bench 83.4
- Meta Code-Fetch Under User Requests 47.8
- ADeptS-Bench
- AutoElicit 14.5
Robustness
- StrongREJECT v2 (Meta) 0.5
- JEF-Bench 2
- OR-Bench 16.8
- FORTRESS 9.4
- DTap Direct Misuse 5.6
- SWE-PI 0
- Unauthorized Code Execution Bench 21.1
- SkillInject 0.6
- DTap Prompt Injection 17.5
- Gray Swan Agent Red Teaming (ART) benchmark 4.8
- Gray Swan IPI Arena benchmark 6.8
- RepoInject v1 23.8
- BrowserInject v1 36.7
Alignment
- Meta instructed selective performance evaluation 0.5
- MASK 97.1
- Meta internal sycophancy evaluation 40.5
- Agentic misalignment suite (Anthropic) 0
- Petri Bloom 1.44
- Meta holistic adversarial audit with Petri Bloom 16.8
- Meta targeted agentic evaluations 8.5
- Meta agentic evaluations grounded in user traffic 20.7
- Meta training-stage provenance probe 70
- Meta action-belief coherence probe 89.1
Monitorability
Bio and chem risk
- Molecular Biology Capabilities Test 52.2
- Virology Capabilities Test (VCT) 51.7
- Human Pathogen Capabilities Test 62.7
- LAB-Bench ProtocolQA 87.7
- LAB-Bench SeqQA 95.2
- ABC Bench Fragment Design 96.5
- ABC Bench Liquid-Handling Robot 95.2
- ABC Bench Screening Evasion 47.6
- Meta BioDesign Proxy 77.4
- BioTIER 97.4
- Meta chemical-agent refusal evaluation 100
Cyber risk
- CyberGym 80.5
- ExploitBench 28.7
- ExploitGym 15
- CyScenarioBench 2
- Charlemagne Labs social-engineering benchmark 1.6
- Meta catastrophic cyber misuse refusal 3.1
- Meta agentic cyber misuse refusal 2.8
- Muse-targeted cyber misuse and violation red-teaming
- Poly-Guard Bench 4.3
- CyberBench (Liu et al.) 6.3
- RedCode 6
- Meta cyber false-refusal benchmark 20.9
AI R&D risk
- PostTrainBench 43.6
- ResearchTasteBench 64.7
- OptimBench 64.2
Dropped since the lab’s previous system card: Humanity's Last Exam , MRCR v2 , MCP Atlas , Toolathlon-Verified , OSWorld-Verified , OSWorld 2.0 , WebArena-Verified , DeepSearchQA , GDPval-AA v2 , JobBench , Vals Finance Agent v2 , Terminal-Bench 2.1 , SWE-Bench Pro , DeepSWE , HealthBench Professional , CharXiv , BabyVision , WMDP , Cybench , Meta curated CTF set , Irregular FrontierCyber and Atomic Challenges , CyberSecEval Autocomplete , CyberSecEval Instruct , Meta Cyber Misuse Chat , CyberSecEval Code Interpreter Abuse , AIRS-Bench , AgentHarm Verified (Meta) , AgentDojo , DeceptionBench , Alignment faking evaluation , Petri
- 2 Sep 2026
- DeepSWE 73.7
- GDPval-AA v2 1545
- Vals Finance Agent v2 61.4
- Harvey LAB (Legal Agent Benchmark) 10
- Terminal-Bench 2.1 89.4
- Terminal-Bench 4.0 19.1
- GDP.pdf 35
- CharXiv 86.2
- LVBench 87.8
- HLE-Verified 54.9
- OSWorld 2.0 59
- BioMysteryBench 88.8
- LABBench2 86.2
Dropped since the last post: Artificial Analysis Intelligence Index v4.1 , FrontierCode 1.1 , Code Arena , Terminal-Bench 3.0 , AutomationBench , MRCR v2 , Agents' Last Exam
Its model card adds 5 evaluations 11 dropped 18 results in all
Gemini 3.8 Flash Model CardDropped since the lab’s previous model card: Artificial Analysis Intelligence Index v4.1 , FrontierCode 1.1 , Code Arena , Terminal-Bench 3.0 , AutomationBench , MRCR v2 , Agents' Last Exam , CBRN expert red teaming , Harmful manipulation efficacy study , GDM Stealth , GDM Situational Awareness
- 1 Sep 2026Claude Fable 5.1 Anthropic Introducing Claude Fable 5.1 and Claude Mythos 5.1 (launch page); System Card: Claude Fable 5.1 & Claude Mythos 5.1, Table 8.1.A
- Terminal-Bench-Science 52.6
- Terminal-Bench 4.0 60.9
- GDPval-AA v2 1853
- OSWorld 2.0 77.9
- Humanity's Last Exam 65
- AutomationBench 31.4
- CursorBench 3.2 73.4
- SWE-Bench Pro 81.2
- SWE-bench Multilingual 89.1
- SWE-bench Multimodal 54.7
- HealthBench Professional 62.1
- AA-Briefcase 1694
- ARC-AGI 97.5
- ARC-AGI-2 90
Dropped since the last post: Terminal-Bench 3.0 , ARC-AGI-3 , BrowseComp , DeepSWE , FrontierCode , Harvey LAB (Legal Agent Benchmark) , BioMysteryBench
Its system card adds 39 evaluations 4 new 46 dropped 52 results in all
System Card: Claude Fable 5.1 & Claude Mythos 5.1Capabilities
- DeepSWE 67.4
- FrontierCode 1.1 50.9
- FrontierSWE 57
- CritPt 88.4
- ProgramBench 87.6
- DRACO 87.7
- Chartography 42.6
- BenchCAD 43.7
- OSWorld 2.0 (Aug 2026) 77.9
- GDP.pdf 85.4
- OfficeQA 80.2
- OfficeQA Pro 69
- Harvey LAB (Legal Agent Benchmark) 19.1
- Toolathlon-Verified 77.8
- Global MMLU 94
- MILU 93
Health
- HealthBench 60
Safety
- Single-turn harmful request evaluations (Anthropic) 94.7
- Single-turn benign request evaluations (Anthropic) 0
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 90.3
- Malicious computer use (Anthropic) 85.7
Robustness
- Rewind-attacker cyber adversarial robustness (Anthropic) 4.5
- Gray Swan Shade knowledge elicitation through chat 1
- Gray Swan IPI Arena benchmark 0.1
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 9.26
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 0.07
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 2.64
Alignment
Cyber risk
Dropped since the lab’s previous system card: Long-form virology tasks , Virology Capabilities Test (VCT) , DNA Synthesis Screening Evasion , Black-box RNA sequence modeling and design , AAV capsid packaging prediction , Anthropic ECI (AECI) , Kernels task (Anthropic) , Time series forecasting task (Anthropic) , LLM training task (Anthropic) , Quadruped reinforcement learning task (Anthropic) , Novel compiler task (Anthropic) , ExploitBench , Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) , Firefox 147 exploit evaluation (Anthropic) , CyScenarioBench , ExploitGym , UK AISI cyber ranges , Prompt injection live bug bounty (Anthropic with Gray Swan) , Anthropic undermining liberal democracy (behavioral audit metric) , UK AISI unprompted sabotage evaluation , UK AISI sabotage continuation evaluation , AA-Omniscience , MASK , Uncritically reporting flawed results (Anthropic) , Overconfidence: verifying command-line syntax (Anthropic) , Lazy investigation (Anthropic) , SHADE-Arena , LinuxArena , SWE-bench Verified , Terminal-Bench 3.0 , IMO 2026 , Riemann-bench , ArXivMath , BrowseComp , DeepSearchQA , OSWorld 2.0 , MCP Atlas , ARC-AGI-3 , INCLUDE , BioMysteryBench , SpatialBench (LatchBio) , scBench (LatchBio SingleCellBench) , ProteinGym , Protein Design (Anthropic) , Organic Chemistry V2 (Anthropic) , Protocols (Anthropic life sciences)
- 28 Aug 2026
- SWE-bench Multilingual 82.9
- SWE-Bench Pro 65.7
- DeepSWE 64.3
- SWE Atlas 58.4
- SWE-Marathon 31.9
- Terminal-Bench 2.1 85.4
- NL2Repo-Bench 58.9
- CyberGym 78.4
- ProgramBench 17.5
- PostTrainBench 1.1 35.6
- Harbor-Index 39.6
- Hy-Backend 2.0 35.2
- Hy-SWE Max 64.2
- Hy-CompanyBench V2 62.4
- WideSearch 83.9
- $OneMillion-Bench 65.4
- DRACO 77.2
- Hy-LifeSearch 49.2
- Hy-BrowseComp-Pro2 56.1
- OfficeQA Pro 66.2
- MCP Atlas 83.7
- Toolathlon-Verified 74.1
- APEX-Agents 37.1
- SkillsBench 62.9
- JobBench 61.7
- WorkSpaceBench 60.2
- Agents' Last Exam 22.8
- GDPval-AA v2 1678
- AutomationBench 32.1
- BankerToolBench 78.6
- E-Bench (Tencent) 77.1
- E-Bench-Code (Tencent) 79
- Hy-FinAgentBench 79.7
- Hy-FinmodelBench v2 57
- BioMysteryBench 71.3
- Humanity's Last Exam 55.4
- CritPt 16.9
- GPQA Diamond 92.3
- SUPERChem 66.4
- ArXivMath 66.6
- HorizonMath 8.8
- MathArena (Apex, IMO, USAMO) 74.2
- BrokenArXiv 54.6
Dropped since the last post: SWE-bench Verified , Hy-CompanyBench , BrowseComp , DeepSearchQA , Toolathlon , Claw-Eval Pass^3 , WildClawBench , Hy-FinModelBench , ProdBench (Tencent) , Hy-SkillsWorld , Hy-Euler pro , FrontierScience , USAMO 2026 , Hy-Math , PHYBench , CMT-Benchmark , IMO-AnswerBench , CL-bench , CL-bench Life , AA-LCR (Artificial Analysis Long Context Reasoning)
- 26 Aug 2026
- DeepSWE 58.7
- SWE-Bench Pro 62.5
- SWE-bench Multilingual 81
- NL2Repo-Bench 48.1
- CoWorkBench 73.9
- JobBench 55.7
- Agents' Last Exam 24.3
- Toolathlon 73.5
- IFBench 81.3
- GPQA Diamond 91.7
- Humanity's Last Exam 35.9
- LiveCodeBench v6 91.9
- Claw-Eval-MM 60.4
- RecreationBench 49.9
- AndroidWorld 84.5
- OSWorld 2.0 19.4
- Vision2Web 64
- ERQA 72.3
- LVBench 76.6
- RealWorldQA 88.5
- MATH-Vision (MathVision) 95.7
- CharXiv 90.6
Dropped since the last post: Terminal-Bench 2.1 , FrontierSWE v1 , MLS-Bench , PaperBench , AndroidBench , QwenSWEBench , QwenQoderBench , QwenReactBench , QwenSVGBench , WorkSpaceBench , SkillsBench , AutomationBench , WideSearch , $OneMillion-Bench , HealthBench , PLawBench , PRBench , PRBench , MRCR v2 , LongBench v2 , MMMU-Pro , BabyVision , HLE-VL , ZeroBench , LogicVista , HiPhO , PhyX , SLAKE , MedXpertQA-MM , PMC-VQA , OSWorld-Verified , ScreenSpot-Pro , WebArena-Verified , MobileWorld , QwenBlenderBench , Parametric CAD Bench , PresentBench , OmniDocBench v1.5 , OCRBench v2 , CC-OCR-Bench-V2 , MTVQA-Test , MADQA , QwenVisualOffice , LingoQA , SURDS , SimpleVQA , WorldVQA , MMStar , PerceptionBench (Moonshot) , CountQA , RefAdv-S , Dense 200 (bbox) , COCO , VisFactor , VLMsAreBiased , Video-MME , VideoMME v2 (with subtitles) , Video-MMMU , MMVU , MLVU , TVBench , LVBench (with memory) , EgoLife (with memory) , VideoDR (with search)
- 26 Aug 2026
- Terminal-Bench 2.1 84.3
- DeepSWE 63.4
- NL2Repo-Bench 56.3
- Toolathlon 78.4
- AutomationBench 48.8
- Agents' Last Exam 26.3
- Humanity's Last Exam 55.3
- GDPval-AA v2 1773
- OfficeQA Pro 62.4
- CharXiv 89.4
- Chartography 78
- BabyVision 53.4
- MVBench 77.8
- MMVU 80.5
- OSWorld 2.0 59.1
- Vision2Web 77.8
Dropped since the last post: Terminal-Bench 3.0 , ProgramBench , FrontierSWE v1 , SWE-Marathon , PostTrainBench , CyberGym , ExploitGym , ExploitBench
- 25 Aug 2026
- SWE-bench Multilingual 41.9
- SWE-Bench Pro 33.3
- SWE-bench Verified 57
- Terminal-Bench 2.1 29.2
- tau3-bench 62
- BFCL V4 61.4
- ProfBench 42.9
- Bird-SQL 41.9
- GDPval-AA v2 1225
- AIME 2025 89.2
- HMMT 2025 89.2
- GPQA Diamond 66.4
- LiveCodeBench v6 75.8
- SciCode 38.8
- MMLU-Pro 77.6
- MMLU-ProX 66.6
- Arena-Hard v2.0 67.9
- IFBench 77.2
- RULER 81.4
Dropped since the last post: MMLU , BIG-Bench Hard , AGIEval , SimpleQA , AlpacaEval 2.0 , IFEval , Arena-Hard-Auto , MT-Bench , GSM8K , GSM-Symbolic , MATH , DeepMind Mathematics , HumanEval , HumanEval+ , MBPP , MBPP+ , CRUXEval , BigCodeBench , MultiPL-E , Berkeley Function Calling Leaderboard (BFCL) , MMMLU , INCLUDE , MGSM , SALAD-Bench , AttaQ , Tulu 3 Safety Eval
- 21 Aug 2026
- Terminal-Bench 2.1 83.9
- NL2Repo-Bench 57.7
- CyberGym 75.3
- DeepSWE 59.3
- Toolathlon 75.9
- DSBench (DeepSeek internal) 63.6
- AutomationBench 25.7
- Agents' Last Exam 27.3
- Chartography 64.3
- ZeroBench 35
Dropped since the last post: Humanity's Last Exam
- 14 Aug 2026
- Terminal-Bench 2.1 73
- SWE-Bench Pro 61.7
- NL2Repo-Bench 42.3
- DeepSWE 42.2
- QwenSWEBench 79
- CoWorkBench 70.7
- JobBench 33.4
- Agents' Last Exam 20.4
- IFBench 79.5
- GPQA Diamond 89.2
- Humanity's Last Exam 30.8
- LiveCodeBench v6 90.3
- OSWorld-Verified 84.3
- WebArena-Verified 64.8
- AndroidWorld 81.9
- RecreationBench 47.1
- Claw-Eval-MM 56.9
- SWE-bench Multimodal 38.6
- Vision2Web 62.9
- MATH-Vision (MathVision) 94.6
- BabyVision 85.6
- CharXiv 90.2
- OmniDocBench v1.5 91.1
- RealWorldQA 85.9
- ERQA 65.5
Dropped since the last post: AgentWorldBench
- 14 Aug 2026
- Terminal-Bench 2.1 88.2
- Terminal-Bench 3.0 28.3
- DeepSWE 66.9
- NL2Repo-Bench 58
- ProgramBench 19
- FrontierSWE v1 78.1
- SWE-Marathon 42.5
- PostTrainBench 39.8
- CyberGym 84.5
- ExploitGym 105
- ExploitBench 54.4
- Toolathlon 73
- AutomationBench 48.2
- Agents' Last Exam 28.5
- Humanity's Last Exam 62.5
- GDPval-AA v2 1769
Dropped since the last post: CritPt , AIME 2026 , HMMT Nov 2025 , HMMT 2026 , IMO-AnswerBench , GPQA Diamond , SWE-Bench Pro , MCP Atlas
- 13 Aug 2026
- Humanity's Last Exam 60
- Terminal-Bench 2.1 87.9
- NL2Repo-Bench 61.5
- CyberGym 83.3
- DeepSWE 62.7
- Toolathlon 74.1
- Agents' Last Exam 25.7
- AutomationBench 31.8
- DSBench (DeepSeek internal) 71.1
- DSBench (DeepSeek internal) 67.2
- 13 Aug 2026
- Artificial Analysis Intelligence Index v4.1 56
- FrontierCode 1.1 43.6
- DeepSWE 65.3
- Code Arena 1588
- Terminal-Bench 2.1 85.8
- Terminal-Bench 3.0 14.9
- AutomationBench 30.4
- GDPval-AA v2 1525
- Harvey LAB (Legal Agent Benchmark) 90.7
- GDP.pdf 34
- CharXiv 88.7
- LVBench 85.4
- MRCR v2 97
- OSWorld 2.0 47.9
- Agents' Last Exam 26.3
- HLE-Verified 53.6
- BioMysteryBench 87.1
- LABBench2 82.1
Dropped since the last post: Success detection (Google DeepMind) , ERQA , Instrument reading (Google DeepMind) , Progress classification (Google DeepMind) , Moment finding (Google DeepMind) , ASIMOV , Human proximity (Google DeepMind robotics)
Its system card adds 16 evaluations 11 new 12 dropped 16 results in all
Frontier Safety Framework Report - Gemini 3.7 FlashRobustness
Alignment
- GDM Stealth 75
- GDM Situational Awareness 54.5
Bio and chem risk
AI R&D risk
Persuasion
Dropped since the lab’s previous system card: SecureBio VMQA, single choice , LAB-Bench ProtocolQA , LAB-Bench CloningScenarios , LAB-Bench SeqQA , WMDP Biology , WMDP Chemistry , CBRN open-ended questions , Cyber key skills benchmark v1 , Cyber key skills benchmark v2 , RE-Bench , Chain-of-thought comprehensibility , Chain-of-thought informativeness
Its model card adds 9 evaluations 4 new 3 dropped 27 results in all
Gemini 3.7 Flash Model CardAlignment
Bio and chem risk
Persuasion
Dropped since the lab’s previous model card: SWE-Bench Pro , MLE-bench , OSWorld-Verified
- 12 Aug 2026
- MMStar 63.3
- MME 73.1
- RealWorldQA 73.1
- SimpleVQA 35.4
- SEED-Bench 77.7
- MMBench-EN v1.1 81
- CountBench 87.3
- MMMB 83
- MM-IFEval 60.6
- LogicVista 37.4
- MathVista 68.5
- MMMU-Pro 30.5
- MMMU 48.4
- ChartQA 81.3
- DocVQA 91.1
- InfographicVQA 70.2
- OCRBench 84.2
- OCRBench v2 47.5
- TextVQA 84.3
- RefCOCO / RefCOCO+ / RefCOCOg 87.9
- BLINK 61.5
- MuirBench 58.3
- HallusionBench 47.2
- POPE 88.7
- ScreenSpot-v2 80.7
- IFEval 82.3
- IFBench 25.8
- Multi-IF 59.4
- ToolSandbox 59.5
- BFCL V4 32.5
Dropped since the last post: AA-Omniscience , AIME 2025 , LiveCodeBench v6 , IFStruct , τ³-bench Banking , Claw-Eval , PinchBench , BrowseComp-Plus
- 12 Aug 2026
- MMBench-EN v1.1 68.7
- MMStar 51.8
- RealWorldQA 62.2
- GQA 57.4
- MMMB 72.8
- BLINK 52.7
- ChartQA 80.8
- DocVQA 92.1
- InfographicVQA 65.2
- OCRBench v2 36.7
- OCRBench 79.2
- AI2D 77.5
- MMMU 32.9
- RefCOCO / RefCOCO+ / RefCOCOg 73.2
- CountBench 72.5
- HallusionBench 61.5
- MMLU 50.4
- MMLU-Pro 30.7
- Multi-IF 37.3
- IFEval 74.9
Dropped since the last post: Terminal-Bench 2.0 , Terminal-Bench Hard , SWE-bench Verified , SWE-Bench Pro , SciCode , LiveCodeBench v6
- 12 Aug 2026
- Artificial Analysis Intelligence Index v4.1 61
- GDPval-AA v2 1753
- CursorBench 3.2 69.9
- DeepSWE 65.9
- FrontierCode 1.1 61.3
- APEX-Agents 57.5
- Terminal-Bench 3.0 26
- APEX-SWE 56.4
- AA-Briefcase 1577
- Harvey LAB (Legal Agent Benchmark) 15.8
Dropped since the last post: Big Bench Audio , τ-Voice
Its model card adds 34 evaluations 11 new 10 dropped 43 results in all
Model Card: Grok 4.6Capabilities
- SWE-Marathon 31.9
- OfficeQA 63.2
- EEBench 60
- 3DCodeBench 54
- PartBench 58.5
- CADGenBench 40.9
- Parametric CAD Bench 88.4
- DeepSearchQA 81.6
Hallucination
Safety
- General refusals (xAI) 0.93
- Child safety (xAI) 0
- Autointent-Bio 100
- Autointent-Chem 100
- FORTRESS-RN 97.9
- Self-harm refusals (xAI) 0.84
Robustness
Alignment
- MASK 1.9
- xAI sycophancy eval 0.04
Bio and chem risk
- Virology Capabilities Test (VCT) 67.4
- Biosecurity VCT 47.8
- BioUseBench 90.7
- WMDP
- LAB-Bench 80.7
- ProtocolQA Open-Ended (xAI) 79.6
- BixBench 93.8
Cyber risk
- CyberGym 79.7
- CVE-Bench 39.8
- SecureCodeReview (xAI) 58.7
- HackerBench 6.9
AI R&D risk
- SpaceXAI MTS Eval 61.1
- InferenceEval 46.9
- KernelBenchInternal 37.2
Dropped since the lab’s previous model card: SWE-Bench Pro , SWE-bench Multilingual , FrontierSWE v1 , ProgramBench , Terminal-Bench 2.1 , SWE Atlas , FalseClaimBench , τ³-bench Banking , RelBench , Epistemic Bias (xAI)
- 11 Aug 2026
Nemotron 3.5 Lightning 30B-A3B NVIDIA NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 model card (Hugging Face) - MMLU-Pro 81.9
- AA-Omniscience 17.5
- GPQA Diamond 75.4
- Humanity's Last Exam 11.7
- SciCode 32.6
- SWE-bench Verified 51.6
- SWE-bench Multilingual 39.3
- Terminal-Bench 2.1 24.6
- PinchBench 85.4
- BrowseComp 37
- τ³-bench Banking 9.28
- GDPval-AA v2 832
- IFBench 71.9
- AA-LCR (Artificial Analysis Long Context Reasoning) 52
Dropped since the last post: CVBench , OCRBench v2 , OSWorld-Verified , CharXiv , MMLongBench-Doc , MathVista , OCR-Reasoning , Video-MME , WorldSense (omnimodal) , Daily-Omni , VoiceBench , MMAU
Its technical report adds 4 evaluations 3 new 29 dropped 17 results in all
Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AICapabilities
- tau2-bench
- CritPt 0
- RULER 91.5
- NV Chat Benchmark 12.1
Dropped since the lab’s previous technical report: MMMU , MathVista , MMLongBench-Doc , OCRBench , OCRBench v2 , ChartQA , DocVQA , AI2D , TextVQA , InfographicVQA , OCR-Reasoning , CharXiv , CharXiv (DQ) , TreeBench , CVBench , RefCOCO / RefCOCO+ / RefCOCOg , ScreenSpot , ScreenSpot-v2 , ScreenSpot-Pro , OSWorld-Verified , Video-MME , Open ASR Leaderboard , MMAU , VoiceBench , Daily-Omni , WorldSense (omnimodal) , MMLU-Pro , LiveCodeBench v5 , AIME 2025
- 11 Aug 2026
- Terminal-Bench 2.1 57
- BrowseComp 49.2
- SWE-bench Verified 70.6
- MCP Atlas 61.4
- APEX-Agents 18.7
- τ³-bench Banking 23
- AA-LCR (Artificial Analysis Long Context Reasoning) 71
- GPQA Diamond 89
- MMLU-Pro 86.3
- AIME 2026 95.3
- KMMLU-Pro 79.2
- KBL 77.5
- KorMedMCQA 93.2
- Ko-GDPval 87.3
Dropped since the last post: Humanity's Last Exam , HMMT 2026 , LiveCodeBench v6 , tau2-bench , Terminal-Bench 2.0 , IFBench , Arena-Hard v2.0 , Ko-Arena-Hard v2 (Upstage)
- 10 Aug 2026
Muse Glimmer Meta Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device (Meta) and model card - MCP Atlas 75.5
- DeepSearchQA 74.6
- τ³-bench Banking 23.5
- WildClawBench 47.6
- GDPval-AA v2 953
- Meta Agents Research Environments (ARE) / Gaia2 43.3
- SkillsBench 44.3
- OSWorld-Verified 65.9
- SWE-Bench Pro 51.2
- SWE-bench Verified 76
- Terminal-Bench 2.1 51.7
- SciCode 43.6
- CharXiv 78.8
- ScreenSpot-Pro 75.4
- OmniDocBench v1.5 75.8
- MMMU-Pro 74
- CIMemories 26.4
- AgentDojo 28.4
- IFBench 77
- AIME 2026 94.7
- GPQA Diamond 83.5
- Humanity's Last Exam 22
- AA-LCR (Artificial Analysis Long Context Reasoning) 80
- BEAM 65.1
Dropped since the last post: LiveCodeBench v5 , LiveCodeBench v6 , MATH-500 , AIME 2024 , AIME 2025
Its technical report adds 13 evaluations 13 new 14 dropped 37 results in all
Muse Glimmer 30B: An On-Device Agentic ModelCapabilities
- tau2-bench
- LiveCodeBench Pro 64.1
- PinchBench 90.5
- ARC-AGI 66.9
- ScreenSpot-v2 93.3
- Omnilingual-Gaia2 60.4
- Linguini 8.8
- Belebele 87.8
- MMLU-ProX 80.1
- WMT24++ 53.4
- FLORES+ 47.5
- BOUQuET 36.5
Safety
- PrivacyLens 49.2
Dropped since the lab’s previous technical report: WMDP Cyber , Cybench , Hack the Box Machines (Meta) , Native Code Exploitation (Meta) , LAB-Bench , WMDP Biology , WMDP Chemistry , Molecular Biology Capabilities Test , Meta BioKnowledge Proxy , BioLP-bench , Meta BioProtocol Proxy , Human Pathogen Capabilities Test , Virology Capabilities Test (VCT) , MASK
- 5 Aug 2026
- DeepSWE 59.3
- Terminal-Bench 2.1 82.9
Dropped since the last post: MCP Atlas , JobBench , Toolathlon , OSWorld-Verified , Humanity's Last Exam , Vals Finance Agent v2 , SWE-Bench Pro , CharXiv , BabyVision
- 4 Aug 2026
- AA-Omniscience -29.5
- AIME 2025 51.9
- LiveCodeBench v6 59.4
- IFBench 59.2
- Multi-IF 80.1
- IFStruct 85.5
- BFCL V4 56.9
- ToolSandbox 77.8
- τ³-bench Banking 5.67
- Claw-Eval 62.9
- PinchBench 68.2
- BrowseComp-Plus 26.9
Dropped since the last post: GPQA Diamond , MMLU-Pro , IFEval , GSM8K , MATH-500 , Berkeley Function Calling Leaderboard (BFCL)
- 3 Aug 2026
- Terminal-Bench 2.1 86.6
- SWE-Bench Pro 67.7
- DeepSWE 56.6
- NL2Repo-Bench 55.9
- FrontierSWE v1 73.5
- MLS-Bench 41
- PaperBench 93
- AndroidBench 75.1
- QwenSWEBench 80.7
- QwenQoderBench 58.4
- QwenReactBench 1724
- QwenSVGBench 1713
- CoWorkBench 74.8
- WorkSpaceBench 67.7
- JobBench 53.4
- SkillsBench 70.2
- Agents' Last Exam 27
- AutomationBench 27.3
- Toolathlon 72.5
- WideSearch 81.9
- Humanity's Last Exam 56.2
- GPQA Diamond 92.6
- IFBench 82.8
- $OneMillion-Bench 52.5
- HealthBench 60.2
- PLawBench 73.2
- PRBench 57.6
- PRBench 58.3
- MRCR v2 92.9
- LongBench v2 66.3
- MMMU-Pro 82.3
- MATH-Vision (MathVision) 97.7
- BabyVision 91.3
- HLE-VL 52.2
- ZeroBench 49
- LogicVista 91.9
- HiPhO 90
- PhyX 83.5
- SLAKE 90.8
- MedXpertQA-MM 80.4
- PMC-VQA 66.2
- OSWorld-Verified 86.1
- OSWorld 2.0 19.4
- ScreenSpot-Pro 84.5
- WebArena-Verified 66.8
- AndroidWorld 85.3
- MobileWorld 77.8
- Claw-Eval-MM 74.8
- Vision2Web 69
- QwenBlenderBench 69.9
- Parametric CAD Bench 91.5
- RecreationBench 51.7
- PresentBench 79.6
- CharXiv 93.5
- OmniDocBench v1.5 92.1
- OCRBench v2 74.2
- CC-OCR-Bench-V2 79.6
- MTVQA-Test 56.6
- MADQA 91.8
- QwenVisualOffice 44.6
- RealWorldQA 88
- ERQA 77.8
- LingoQA 84.8
- SURDS 77.8
- SimpleVQA 75
- WorldVQA 53.2
- MMStar 85.9
- PerceptionBench (Moonshot) 63.5
- CountQA 82.4
- RefAdv-S 80.2
- Dense 200 (bbox) 87
- COCO 78.7
- VisFactor 60.8
- VLMsAreBiased 88.3
- Video-MME 90.4
- VideoMME v2 (with subtitles) 68.3
- Video-MMMU 88.7
- MMVU 82.4
- MLVU 90.8
- TVBench 81.9
- LVBench 81.8
- LVBench (with memory) 85.6
- EgoLife (with memory) 80.3
- VideoDR (with search) 73.2
Dropped since the last post: Terminal-Bench 2.0 , SWE-bench Verified , SWE-bench Multilingual , SciCode , QwenWebDev , QwenSVG , QwenClawBench , Claw-Eval , BFCL V4 , MCPMark , MCP Atlas , VITA-Bench , DeepPlanning , SpreadsheetBench , Kernel Bench L3 , QwenWorldBench , LiveCodeBench , HMMT 2026 , IMO-AnswerBench , CritPt , MathArena (Apex, IMO, USAMO) , MMLU-Pro , MMLU-Redux , SuperGPQA , IFEval , WMT24++ , MAXIFE , MMMLU , MMLU-ProX , NOVA-63 , INCLUDE , Global PIQA , PolyMath , QwenVision2Code , MMSearch-Plus , BrowseComp-VL , MM-BrowseComp , ODinW-13 , Ego3D-Bench , VLADBench
- 31 Jul 2026
- MMLU-Pro 83.5
- GPQA Diamond 82.2
- Humanity's Last Exam 18.3
- AIME 2026 92.3
- HMMT 2026 78.4
- IMO-AnswerBench 78.6
- SciCode 37.4
- SWE-bench Verified 68.2
- Terminal-Bench 2.1 43.8
- τ³-bench Banking 14.2
- Claw-Eval 77.7
- IFEval 92.4
- IFBench 72.6
- OpenAI MRCR v2 94.4
- AA-LCR (Artificial Analysis Long Context Reasoning) 56.2
- Ko-LongBench (LG) 89.6
- KMMLU-Pro 69.1
- CLIcK 84.2
- HRM8K-KSM 91.1
- MMMLU 86.6
- Global MMLU Lite 3.0 86.6
- PolyMath 71.3
- KGC-Safety (LG) 99.8
- ROK-FORTRESS 89.5
Dropped since the last post: MMMU , MMMU-Pro , MedXpertQA-MM , MATH-Vision (MathVision) , MathVista , We-Math , LogicVista , BabyVision , AI2D , ChartQA Pro , CharXiv , OCRBench v2 , OmniDocBench v1.5 , MMStar , BLINK , HallusionBench , KMMMU , K-Viscuit , KRETA , AIME 2025 , LiveCodeBench v6 , tau2-bench , KoBALT
Its technical report adds 2 evaluations 1 new 22 dropped 25 results in all
K-EXAONE 2.0 Technical ReportCapabilities
- OpenAI MRCR 94.4
- WMT24++
Dropped since the lab’s previous technical report: MMMU , MMMU-Pro , MedXpertQA-MM , MATH-Vision (MathVision) , MathVista , We-Math , LogicVista , BabyVision , AI2D , ChartQA Pro , CharXiv , OCRBench v2 , OmniDocBench v1.5 , MMStar , BLINK , HallusionBench , KMMMU , K-Viscuit , KRETA , LiveCodeBench v6 , tau2-bench , KoBALT
- 31 Jul 2026
- Terminal-Bench 2.1 82.7
- NL2Repo-Bench 54.2
- CyberGym 76.7
- DeepSWE 54.4
- Toolathlon 70.3
- Agents' Last Exam 25.2
- AutomationBench 25.1
- DSBench (DeepSeek internal) 68.7
- DSBench (DeepSeek internal) 59.6
Dropped since the last post: MMLU-Pro , SimpleQA Verified , Chinese SimpleQA , GPQA Diamond , Humanity's Last Exam , LiveCodeBench , Codeforces rating , HMMT 2026 , IMO-AnswerBench , MathArena (Apex, IMO, USAMO) , MathArena (Apex, IMO, USAMO) , MRCR v2 , CorpusQA 1M , Terminal-Bench 2.0 , SWE-bench Verified , SWE-Bench Pro , SWE-bench Multilingual , BrowseComp , GDPval-AA , MCP Atlas , AGIEval , MMLU , MMLU-Redux , MMMLU , C-Eval , CMMLU , MultiLoKo , SuperGPQA , FACTS Parametric , TriviaQA , BIG-Bench Hard , DROP , HellaSwag , WinoGrande , CLUEWSC , BigCodeBench , HumanEval , GSM8K , MATH , MGSM , CMATH , LongBench v2
- 30 Jul 2026
- Success detection (Google DeepMind) 87.7
- ERQA 78.5
- Instrument reading (Google DeepMind) 65.7
- Progress classification (Google DeepMind) 57.4
- Moment finding (Google DeepMind) 91.3
- ASIMOV 97.9
- Human proximity (Google DeepMind robotics) 93
Dropped since the last post: CyberGym , Big Sleep Evaluation (Google) , Chrome Production Commit Scanning (Google)
Its technical report adds 1 evaluation 1 new 16 dropped 2 results in all
Gemini Robotics 2: Safety EvaluationsSafety
Dropped since the lab’s previous technical report: Point-Bench , RefSpatial-Bench , RoboSpatial-Home , Where2Place , BLINK , CVBench , ERQA , EmbSpatialBench , MindCube , SAT (Spatial Aptitude Training) , MVPBench , OpenEQA , VSI-Bench , MMMU , GPQA Diamond , Aider Polyglot
- 30 Jul 2026
- Artificial Analysis Intelligence Index v4.1 40
- SWE-bench Verified 80.2
- SWE-Bench Pro 55.9
- Terminal-Bench 2.1 64.7
- SciCode 48.7
- GDPval-AA v2 1269
- MCP Atlas 79.6
- τ³-bench Banking 15.5
- BrowseComp 77.4
- Toolathlon-Verified 54.4
- AA-Briefcase 917
- GPQA Diamond 89.5
- Humanity's Last Exam 47.8
- AIME 2026 95.5
- HMMT 2026 90.2
- CritPt 8.3
- ARC-AGI 84
- ARC-AGI-2 40.1
- SimpleQA Verified 20.6
- AA-Omniscience -9
- IFBench 82.2
- Global MMLU Lite 3.0 86.7
- StrongREJECT 98.4
- FORTRESS 71.6
- MMMU-Pro 74
- CharXiv 81.3
- Audio MC 54.9
- MMAU 77
- VoiceBench 90.1
- ForecastBench 61.5
- Prophet Arena 0.124
Its model card adds 0 evaluations 29 results in all
- 29 Jul 2026
- Big Bench Audio 97.2
- τ-Voice 56.5
Dropped since the last post: DeepSWE , SWE-Marathon , Terminal-Bench 2.1 , SWE-Bench Pro
- 27 Jul 2026
- CyberGym 90.4
- CyberGym (final-submission metric) 86.3
Dropped since the last post: SWE-bench Verified , SWE-Bench Pro , SWE-bench Multilingual , Terminal-Bench 2.0 , AIME 2026 , AMO-Bench , FrontierMath , Humanity's Last Exam , GPQA Diamond , FrontierScience , ArtifactsBench , IFBench , AdvancedIF , Robust IF Bench , tau2-bench
Its model card adds 5 evaluations 5 new 15 dropped 6 results in all
MAI-Cyber-1-Flash Model CardCyber risk
Dropped since the lab’s previous model card: SWE-bench Verified , SWE-Bench Pro , SWE-bench Multilingual , Terminal-Bench 2.0 , AIME 2026 , AMO-Bench , FrontierMath , Humanity's Last Exam , GPQA Diamond , FrontierScience , ArtifactsBench , IFBench , AdvancedIF , Robust IF Bench , tau2-bench
- 24 Jul 2026Claude Opus 5 Anthropic Introducing Claude Opus 5 (launch post); System Card: Claude Opus 5, Table 8.1.A
- Terminal-Bench 3.0 43.3
- GDPval-AA v2 1861
- ARC-AGI-3 30.2
- BrowseComp 90.8
- Humanity's Last Exam 64.7
- OSWorld 2.0 70.6
- DeepSWE 68.8
- FrontierCode 53.4
- AutomationBench 26
- Harvey LAB (Legal Agent Benchmark) 11.7
- HealthBench Professional 59.8
- BioMysteryBench 90.1
- SWE-Bench Pro 79.2
- SWE-bench Multilingual 89.5
- SWE-bench Multimodal 59.4
- AA-Briefcase 1720
- ARC-AGI 97.5
- ARC-AGI-2 90.4
Dropped since the last post: Terminal-Bench 2.1 , OSWorld-Verified
Its system card adds 74 evaluations 22 new 14 dropped 91 results in all
System Card: Claude Opus 5Capabilities
- SWE-bench Verified 96
- FrontierCode 1.1 53.4
- IMO 2026 100
- Riemann-bench 60
- ArXivMath 90.8
- ProgramBench 93
- DeepSearchQA 95
- DRACO 88.6
- Chartography 29.6
- BenchCAD 36.6
- GDP.pdf 83.4
- OfficeQA 78.1
- OfficeQA Pro 66.9
- MCP Atlas 85.8
- Toolathlon-Verified 80.6
- Global MMLU 92.5
- MILU 92.1
- INCLUDE 89.8
- SpatialBench (LatchBio) 72.5
- scBench (LatchBio SingleCellBench) 60.6
- ProteinGym 47.7
- Protein Design (Anthropic) 42.5
- Organic Chemistry V2 (Anthropic) 61.6
- Protocols (Anthropic life sciences)
Health
- HealthBench 57.8
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 96.3
- Single-turn benign request evaluations (Anthropic) 0.09
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 89
- Malicious computer use (Anthropic) 93.8
Robustness
- Rewind-attacker cyber adversarial robustness (Anthropic) 5.6
- Gray Swan IPI Arena benchmark 0.2
- Prompt injection live bug bounty (Anthropic with Gray Swan) 0.08
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 0.56
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 0.54
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 3.84
Alignment
- Anthropic automated behavioral audit 2.3
- Anthropic undermining liberal democracy (behavioral audit metric) 1.13
- UK AISI unprompted sabotage evaluation 0
- UK AISI sabotage continuation evaluation 0.1
- MASK 94.8
- Uncritically reporting flawed results (Anthropic) 9.83
- Overconfidence: verifying command-line syntax (Anthropic) 4.9
- Lazy investigation (Anthropic) 9.94
Monitorability
- SHADE-Arena 12
- LinuxArena 0.7
Bio and chem risk
Cyber risk
- ExploitBench 70
- Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) 79.4
- Firefox 147 exploit evaluation (Anthropic) 52.4
- CyScenarioBench 33.7
- ExploitGym 22
- UK AISI cyber ranges 80
- Vulnerability finding evaluations (Anthropic)
- Secure and defensive coding (Anthropic) 0.61
- Cyber harm coverage (Anthropic) 99.6
AI R&D risk
Dropped since the lab’s previous system card: CyberGym , Gray Swan Agent Red Teaming (ART) benchmark , Evaluation awareness discriminability (Anthropic) , Terminal-Bench 2.1 , FrontierCode , CursorBench 3.1 , USAMO 2026 , OSWorld-Verified , ChartMuseum , CharXiv , Real-World Finance v2 (Anthropic) , Toolathlon , Structural Biology (Anthropic) , Organic Chemistry (Anthropic)
- 23 Jul 2026
- SWE-bench Verified 69.4
- SWE-bench Multilingual 63
- SWE-Bench Pro 46
- Terminal-Bench 2.1 41
- PinchBench 93.4
- SciCode 44.2
- KAT Code Bench 46.2
Dropped since the last post: LongVideoBench , VideoMME v2 (with subtitles) , TimeLens-Bench , MLVU , Video-MME , Video-MMMU , LiveCodeBench v6 , OJBench , BFCL V4 , tau2-bench , VITA-Bench , We-Math , MathVista , DynaMath , AIME 2025 , AIME 2026 , Humanity's Last Exam , MMMU , MMStar , C-Eval , IFEval , HallusionBench , AI2D , OCRBench , OmniDocBench v1.5 , RefCOCO / RefCOCO+ / RefCOCOg , PixMo-Count , EmbSpatialBench
- 22 Jul 2026
- MMLU-Pro 86.2
- GPQA Diamond 86.3
- Humanity's Last Exam 28.8
- LiveCodeBench v6 92.4
- ArtifactsBench 55.9
- HMMT 2026 93.9
- AIME 2026 95.7
- MultiChallenge 61
- IFBench 80
- AA-LCR (Artificial Analysis Long Context Reasoning) 62.3
- SWE-bench Verified 70.4
- Terminal-Bench Hard 28.3
- APEX-Agents 16.6
- MCP Atlas 58.2
- τ³-bench Banking 19.6
- GDPval-AA v2 1128
- KMMLU-Pro 78.4
- CLIcK 90.7
- HAE-RAE Bench 73.8
- Ko-AIME 2025 (Upstage) 97.7
- HRM8K 92.2
- KBank-MMLU (Upstage) 80.8
- KBL 75.5
- KorMedMCQA 93
- Ko-GDPval 86.8
Dropped since the last post: KMMLU , KoBALT , MMLU , AIME 2024 , AIME 2025 , HMMT 2025 , HMMT Nov 2025 , IFEval , Arena-Hard v2.0 , WritingBench , tau2-bench
Its technical report adds 0 evaluations 14 dropped 25 results in all
Solar Open 2 Technical ReportDropped since the lab’s previous technical report: KMMLU , KoBALT , Ko-AIME 2024 (Upstage) , Ko-IFEval (Upstage) , Ko-Arena-Hard v2 (Upstage) , MMLU , AIME 2024 , AIME 2025 , HMMT 2025 , HMMT Nov 2025 , IFEval , Arena-Hard v2.0 , WritingBench , tau2-bench
- 21 Jul 2026
Dropped since the last post: SWE-Bench Pro , DeepSWE , Terminal-Bench 2.1 , MLE-bench , GDPval-AA v2 , OSWorld-Verified , CharXiv , MRCR v2
- 21 Jul 2026
- Terminal-Bench 2.1 70.2
- SWE-bench Multilingual 78.5
- SWE-Bench Pro 59.4
- DeepSWE 40.4
- SWE Atlas 46.2
- Toolathlon-Verified 49.7
Dropped since the last post: SWE-bench Verified , Terminal-Bench 2.0
- 21 Jul 2026
- SWE-Bench Pro 54.2
- Terminal-Bench 2.1 54
- MLE-bench 39.2
- GDPval-AA v2 1140
- OSWorld-Verified 74
- CharXiv 76.5
- MRCR v2 72.2
Dropped since the last post: MMLU-Pro , AIME 2026 , LiveCodeBench v6 , Codeforces rating , GPQA Diamond , tau2-bench , Humanity's Last Exam , BIG-Bench Extra Hard , MMMLU , MMMU-Pro , OmniDocBench v1.5 , MATH-Vision (MathVision) , MedXpertQA-MM
Its model card adds 5 evaluations 8 dropped 12 results in all
Gemini 3.5 Flash-Lite Model CardDropped since the lab’s previous model card: Humanity's Last Exam , GPQA Diamond , MMMU-Pro , Video-MMMU , SimpleQA Verified , FACTS Benchmark Suite , MMMLU , LiveCodeBench
- 21 Jul 2026
- SWE-Bench Pro 58.7
- DeepSWE 49
- Terminal-Bench 2.1 78
- MLE-bench 63.9
- GDPval-AA v2 1421
- OSWorld-Verified 83
- CharXiv 89.4
- MRCR v2 91.8
Dropped since the last post: MCP Atlas , Toolathlon , Vals Finance Agent v2 , GDPval-AA , MMMU-Pro , Blueprint-Bench 2 , Humanity's Last Exam , ARC-AGI-2
Its model card adds 5 evaluations 8 dropped 13 results in all
Gemini 3.6 Flash Model CardDropped since the lab’s previous model card: MCP Atlas , Toolathlon , Vals Finance Agent v2 , GDPval-AA , MMMU-Pro , Blueprint-Bench 2 , Humanity's Last Exam , ARC-AGI-2
- 16 Jul 2026
- GPQA Diamond 93.5
- CritPt 23.4
- AA-LCR (Artificial Analysis Long Context Reasoning) 74.7
- Humanity's Last Exam 56
- DeepSWE 67.5
- ProgramBench 77.8
- Terminal-Bench 2.1 88.3
- FrontierSWE v1 81.2
- SWE-Marathon 42
- PostTrainBench 36.6
- MLS-Bench 48.3
- SciCode 58.7
- Kimi Code Bench v2 72.9
- BrowseComp 91.2
- DeepSearchQA 95
- ResearchRubrics 76.2
- GDPval-AA v2 1686
- Toolathlon 76.5
- MCPMark 94.5
- MCP Atlas 84.2
- AutomationBench 30.8
- JobBench 54.3
- AA-Briefcase 1548
- Agents' Last Exam 28.3
- APEX-Agents 41
- OfficeQA 63.3
- SpreadsheetBench 2 34.8
- OSWorld-Verified 84.8
- OSWorld 2.0 58.3
- SaaS-Bench 60.1
- tau3-bench 33.4
- Harvey LAB (Legal Agent Benchmark) 94.6
- CorpFin v2 71.6
- Vals Finance Agent v2 54.4
- Legal Research Bench 44.2
- WorldVQA 51
- OmniDocBench 91.1
- PerceptionBench (Moonshot) 58.5
- Video-MME 90
- MMVU 82.1
- BabyVision 85.7
- MMMU-Pro 81.6
- CharXiv 84.8
- MATH-Vision (MathVision) 94.3
- ZeroBench 23
Dropped since the last post: Kimi Claw 24/7 Bench
Its technical report adds 27 evaluations 26 new 33 dropped 70 results in all
Kimi K3: Open Frontier Intelligence (technical report, arXiv 2607.24653)Capabilities
- Toolathlon-Verified 76.5
- τ³-bench Banking 33.4
- Coding Experience (Moonshot) 59.9
- 24/7 ClawBench 2.0 48.3
- MIRA Bench 64.1
- KAET (Kimi Autonomous Execution Tasks) 83.5
- CLIF Bench 52.4
- Agentic Vision Bench (Moonshot) 78.3
- Swarm Bench 76.3
- Online Experience (Moonshot) 77.9
- Deep Research Bench (Moonshot) 90
- Finance Bench (Moonshot) 62.6
- KWV Bench 64.7
- DECK Bench 73.5
- Agent Behavior Bench 65
- Chat All-in-One Bench 85.2
- Kimi Webdev Bench 58.6
Hallucination
Cyber risk
Other
- Artificial Analysis Intelligence Index v4.1 57.1
- Vals Index v1 74.7
- Code Arena 1678
- LMArena (Elo) 1486
- Agent Arena 9.1
Dropped since the lab’s previous technical report: AIME 2025 , HMMT 2025 , HMMT Nov 2025 , IMO-AnswerBench , MMLU-Pro , SimpleQA Verified , AdvancedIF , LongBench v2 , SWE-bench Verified , SWE-Bench Pro , SWE-bench Multilingual , Terminal-Bench 2.0 , PaperBench , CyberGym , OJBench , LiveCodeBench v6 , WideSearch , FinSearchComp , Seal-0 , GDPval-AA , MMMU , MathVista , SimpleVQA , BLINK , MMVP , OmniDocBench v1.5 , OCRBench , InfographicVQA , Video-MMMU , MotionBench , LongVideoBench , LVBench , WebArena
- 16 Jul 2026
- DeepSWE 62
- SWE-Marathon 29
- Terminal-Bench 2.1 83.3
- SWE-Bench Pro 64.7
Dropped since the last post: τ-Voice
Its model card adds 32 evaluations 28 new 11 dropped 36 results in all
Model Card: Grok 4.5Capabilities
- APEX-SWE 51.2
- SWE-bench Multilingual 78
- FrontierSWE v1 78
- ProgramBench 57.2
- SWE Atlas 84
- EEBench
- 3DCodeBench 49.8
- Parametric CAD Bench 88.1
- GDPval-AA v2 1535
- τ³-bench Banking 32.6
- DeepSearchQA 38.4
Hallucination
Safety
- General refusals (xAI) 1.1
- Child safety (xAI) 0
- Autointent-Bio 97.9
- Autointent-Chem 96.7
- FORTRESS-RN 97.9
- Self-harm refusals (xAI) 0.5
Robustness
Bias
- Epistemic Bias (xAI) 20.4
Alignment
- FalseClaimBench 84
- MASK 0.67
- xAI sycophancy eval 0.01
Bio and chem risk
- Virology Capabilities Test (VCT) 65.5
- WMDP
- LAB-Bench 71.1
- ProtocolQA Open-Ended (xAI) 87
- BixBench 93.8
Cyber risk
- CyberGym 80.4
- HackerBench 7.8
AI R&D risk
Dropped since the lab’s previous model card: Refusals + User Jailbreak (xAI) , Refusals + System Jailbreak (xAI) , AgentHarm , AgentDojo , SycophancyEval (answer sycophancy) , BioLP-bench , LAB-Bench ProtocolQA , LAB-Bench FigQA , LAB-Bench CloningScenarios , Cybench , MakeMeSay
- 15 Jul 2026
- Humanity's Last Exam 46
- AIME 2026 97.1
- GPQA Diamond 87.2
- SWE-bench Verified 77.6
- SWE-Bench Pro 54.3
- Terminal-Bench 2.1 63.8
- GDPval-AA v2 1238
- MCP Atlas 76
- τ³-bench Banking 23.7
- Toolathlon-Verified 45.5
- BrowseComp 77.1
- SimpleQA Verified 43.9
- AA-Omniscience 2.1
- IFBench 79.8
- Global MMLU Lite 3.0 88.7
- MMMU-Pro 73.5
- CharXiv 82
- Audio MC 56.6
- MMAU 77.2
- VoiceBench 91.4
- FORTRESS 78
- StrongREJECT 98.6
Its model card adds 0 evaluations 22 results in all
- 9 Jul 2026
- Agents' Last Exam 50.3
- GDPval-AA v2 1591.8
- BigFinanceBench 36
- Artificial Analysis Intelligence Index v4.1 51.2
- Artificial Analysis Coding Agent Index v1.1 74.6
- SWE-Bench Pro 62.7
- DeepSWE 67.2
- Terminal-Bench 2.1 84.7
- GeneBench Pro 10.8
- LifeSciBench 51.2
- HealthBench Professional 55.7
- OSWorld 2.0 45.6
- BrowseComp 83.3
- BenchCAD 73.9
- SEC-bench Pro 48.9
- ExploitBench 33.2
- ExploitGym 12.4
- MMMU-Pro 79.5
- GDP.pdf 22.7
- GPQA Diamond 92.3
- FrontierMath Tiers 1-3 v2 78.6
- FrontierMath Tier 4 v2 58.5
- AutomationBench 14.9
- Toolathlon 53.4
- OpenAI MRCR v2 41.3
- GraphWalks 81.3
- ARC-AGI-3 0.18
Dropped since the last post: Terminal-Bench 2.0 , OSWorld-Verified , MCP Atlas , tau2-bench , Humanity's Last Exam , OmniDocBench v1.5
Its system card adds 28 evaluations 2 new 32 results in all
Its system card addendum adds 20 evaluations 4 new 21 results in all
GPT-5.6 – August UpdatesHealth
Hallucination
Safety
Robustness
Bio and chem risk
Cyber risk
- Capture the Flag (Internal) 93.2
- CVE-Bench 82.3
- Cyber Range 61.5
- 9 Jul 2026
- Agents' Last Exam 50.4
- GDPval-AA v2 1593
- BigFinanceBench 51
- Artificial Analysis Intelligence Index v4.1 55
- Artificial Analysis Coding Agent Index v1.1 77.4
- SWE-Bench Pro 63.4
- DeepSWE 69.6
- Terminal-Bench 2.1 87.4
- GeneBench Pro 23.3
- LifeSciBench 56
- HealthBench Professional 57.7
- OSWorld 2.0 50.2
- BrowseComp 87.5
- BenchCAD 78.2
- SEC-bench Pro 57.7
- ExploitBench 52.9
- ExploitGym 23.2
- MMMU-Pro 82
- GDP.pdf 24.7
- GPQA Diamond 92.9
- FrontierMath Tiers 1-3 v2 84.9
- FrontierMath Tier 4 v2 68.3
- AutomationBench 15.2
- Toolathlon 53.1
- OpenAI MRCR v2 89.6
- GraphWalks 76.9
- ARC-AGI-3 0.8
Its system card adds 29 evaluations 27 dropped 33 results in all
GPT-5.6 System CardHealth
Hallucination
Safety
Robustness
Bias
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Disallowed Content Evaluations with Representative Prompts , Deception Evaluations with Representative Prompts , OpenAI Deployment Simulation , Verbalized metagaming , CoT Monitorability , AAV Capsid Packaging Prediction (OpenAI) , Hard-negative protein binding prediction , DNA sequence design for transcription factor binding , Virology Capabilities Test (VCT) , Molecular Biology Capabilities Test , Human Pathogen Capabilities Test , World-Class Bio , ReproBAIT , VulnLMP , FrontierCyber , CyScenarioBench , Atomic Challenges , UK AISI narrow cyber tasks , UK AISI cyber ranges , CyberGym , METR Time Horizon , UK AISI unprompted sabotage evaluation , UK AISI sabotage continuation evaluation , ImpossibleMLEBench , ControlArena , No-CoT math time horizon , Apollo Research strategic deception evaluations
- 9 Jul 2026
- MCP Atlas 88.1
- JobBench 54.7
- Toolathlon 75.6
- OSWorld-Verified 80.8
- Humanity's Last Exam 62.1
- Vals Finance Agent v2 57.2
- Terminal-Bench 2.1 80
- SWE-Bench Pro 61.5
- DeepSWE 53.3
- CharXiv 88.4
- BabyVision 76.3
Dropped since the last post: MMMU-Pro , ERQA , SimpleVQA , ScreenSpot-Pro , ZeroBench , ARC-AGI-2 , GPQA Diamond , LiveCodeBench Pro , HealthBench Hard , MedXpertQA , MedXpertQA-MM , DeepSearchQA , SWE-bench Verified , Terminal-Bench 2.0 , tau2-bench , GDPval-AA
Its system card adds 58 evaluations 21 new 19 dropped 68 results in all
Muse Spark 1.1 Evaluation ReportCapabilities
- MRCR v2 54.1
- Toolathlon-Verified 75.6
- OSWorld 2.0 47.3
- WebArena-Verified 69
- DeepSearchQA 84.9
- GDPval-AA v2 1381
Health
Hallucination
- HLE Calibration 23.4
Safety
- SAVE-Bench 90.7
Robustness
- StrongREJECT v2 (Meta) 0.5
- OR-Bench 4.8
- FORTRESS 12.3
- AgentHarm Verified (Meta) 3.4
- AgentDojo 0.7
- SWE-PI
- Gray Swan Agent Red Teaming (ART) benchmark 0.3
Alignment
Monitorability
Bio and chem risk
- Molecular Biology Capabilities Test 53.2
- Virology Capabilities Test (VCT) 52
- Human Pathogen Capabilities Test 61.9
- WMDP
- LAB-Bench ProtocolQA 88
- LAB-Bench SeqQA 98.2
- ABC Bench Fragment Design 97
- ABC Bench Liquid-Handling Robot 93.7
- ABC Bench Screening Evasion 63.2
- Meta BioDesign Proxy 55.2
- BioTIER 97.7
- Meta chemical-agent refusal evaluation 99.8
Cyber risk
- Cybench 92.9
- Meta curated CTF set 89.9
- CyberGym 59
- ExploitGym 5
- Irregular FrontierCyber and Atomic Challenges 67
- CyScenarioBench 0.5
- Charlemagne Labs social-engineering benchmark 5.1
- CyberSecEval Autocomplete 71.3
- CyberSecEval Instruct 77.3
- Meta Cyber Misuse Chat 2.8
- Meta agentic cyber misuse refusal 0.7
- CyberSecEval Code Interpreter Abuse
- Meta catastrophic cyber misuse refusal 3.9
- Poly-Guard Bench 6.9
- CyberBench (Liu et al.) 9
- RedCode 12
AI R&D risk
- AIRS-Bench 0.77
Dropped since the lab’s previous system card: Meta chemical and biological high-risk bottleneck evaluations , Meta cyber misuse request set , Terminal-Bench 2.0 , SWE-bench Verified hard subset (Meta) , MLE-bench , Meta prompted evaluation awareness , AgentHarm , Meta cyber false-refusal benchmark , IHEval , ImpossibleBench , Meta Underspecified Intent evaluation , SimpleQA Verified , AbstentionBench , CharXiv Missing Images , TextQuests , PropensityBench , CIMemories , Task-dependent output homogenization evaluation , Situational Awareness Dataset (SAD)
- 9 Jul 2026
- Agents' Last Exam 53.6
- Artificial Analysis Intelligence Index v4.1 58.9
- Artificial Analysis Coding Agent Index v1.1 80
- Terminal-Bench 2.1 91.9
- DeepSWE 72.7
- BrowseComp 92.2
- GDPval-AA v2 1748
- OSWorld 2.0 62.6
- AutomationBench 18.1
- GeneBench Pro 28.7
- LifeSciBench 59.9
Dropped since the last post: Big Bench Audio , Audio MC
GPT-5.6 Luna: Its system card adds 30 evaluations 23 new 4 dropped 30 results in all
GPT-5.6 Preview System CardHealth
Hallucination
Safety
Robustness
Bias
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: CoT-Control , Capture the Flag (Professional) , Monorepo-Bench , OpenAI-Proof Q&A
Its system card adds 51 evaluations 34 new 5 dropped 51 results in all
GPT-5.6 Preview System CardHealth
Hallucination
Safety
- OpenAI Production Benchmarks and safe-completion evals
- Disallowed Content Evaluations with Representative Prompts
- Image input safety evaluations
- Destructive action avoidance 83
- User Confirmations During Computer Use
- Biology Model Refusal Evaluation
- OpenAI cyber safety evaluations (incl. Daybreak Blue)
Robustness
Bias
Alignment
Monitorability
Bio and chem risk
- Multimodal Troubleshooting Virology 55.5
- ProtocolQA Open-Ended 43.5
- Tacit Knowledge and Troubleshooting
- TroubleshootingBench 48
- AAV Capsid Packaging Prediction (OpenAI) 0.529
- Hard-negative protein binding prediction 7.6
- DNA sequence design for transcription factor binding 13.7
- Virology Capabilities Test (VCT) 53.5
- Molecular Biology Capabilities Test 60
- Human Pathogen Capabilities Test 68.4
- World-Class Bio 68.3
- ReproBAIT 85
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Factuality Heavy prompts , High Stakes factuality prompts , Capture the Flag (Professional) , Cyber Range , Biological safety evaluations (OpenAI)
GPT-5.6 Terra: Its system card adds 31 evaluations 20 dropped 31 results in all
GPT-5.6 Preview System CardHealth
Hallucination
Safety
Robustness
Bias
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Disallowed Content Evaluations with Representative Prompts , Deception Evaluations with Representative Prompts , OpenAI Deployment Simulation , Verbalized metagaming , CoT Monitorability , AAV Capsid Packaging Prediction (OpenAI) , Hard-negative protein binding prediction , DNA sequence design for transcription factor binding , Virology Capabilities Test (VCT) , Molecular Biology Capabilities Test , Human Pathogen Capabilities Test , World-Class Bio , ReproBAIT , VulnLMP , FrontierCyber , CyScenarioBench , Atomic Challenges , CyberGym , METR Time Horizon , Apollo Research strategic deception evaluations
Its system card adds 60 evaluations 29 new 60 results in all
GPT-5.6 System CardHealth
Hallucination
Safety
- OpenAI Production Benchmarks and safe-completion evals
- Disallowed Content Evaluations with Representative Prompts
- Image input safety evaluations
- Destructive action avoidance 83
- User Confirmations During Computer Use
- Biology Model Refusal Evaluation
- OpenAI cyber safety evaluations (incl. Daybreak Blue)
Robustness
Bias
Alignment
Monitorability
Bio and chem risk
- Multimodal Troubleshooting Virology 55.5
- ProtocolQA Open-Ended 43.5
- Tacit Knowledge and Troubleshooting
- TroubleshootingBench 48
- AAV Capsid Packaging Prediction (OpenAI) 0.529
- Hard-negative protein binding prediction 7.6
- DNA sequence design for transcription factor binding 13.7
- Virology Capabilities Test (VCT) 53.5
- Molecular Biology Capabilities Test 60
- Human Pathogen Capabilities Test 68.4
- World-Class Bio 68.3
- ReproBAIT 85
Cyber risk
Its system card addendum adds 24 evaluations 6 new 24 results in all
GPT-5.6 – August UpdatesHealth
Hallucination
Safety
Robustness
Bio and chem risk
Cyber risk
- Capture the Flag (Internal) 97.1
- CVE-Bench 94.1
- Cyber Range 83.3
- 8 Jul 2026
- FrontierCode 1.1 42.3
- Terminal-Bench 2.1 81.5
- SWE-bench Multilingual 77.8
Dropped since the last post: SWE-Bench Pro
- 7 Jul 2026
- RoboCasa 70.8
- RoboCasa GR1 Tabletop Tasks 44.5
- SimplerEnv 62.3
- RoboLab 8.58
Dropped since the last post: Terminal-Bench 2.1 , GDPval , SWE-bench Verified , SWE-bench Multilingual , ProfBench , PinchBench , tau3-bench , τ³-bench Banking , BrowseComp , Vals Finance Agent , IOI 2025 , LiveCodeBench v6 , IMO-AnswerBench , MathArena (Apex, IMO, USAMO) , GPQA Diamond , SciCode , Humanity's Last Exam , CritPt , MMLU-Pro , AA-Omniscience , IFBench , MultiChallenge , AA-LCR (Artificial Analysis Long Context Reasoning) , RULER , LongBench v2 , MMLU-ProX , WMT24++
- 6 Jul 2026
- SWE-Bench Pro 65.2
- KAT Code Bench 53.1
- PinchBench 94.9
- KAT Claw Bench 85.5
- Terminal-Bench 2.1 60.7
- SciCode 50.3
- 5 Jul 2026
- Terminal-Bench 2.1 70.8
- SWE-Bench Pro 59.5
- SWE-bench Multilingual 77.3
- FORTE 73.2
- BrowseComp 79.9
- RWSearch 78.8
- IFEval 90
- WritingBench 83.8
- IMO-AnswerBench 81.8
- GPQA Diamond 88.9
Dropped since the last post: AIME 2025 , HMMT 2025 , AMO-Bench , BrowseComp-ZH , tau2-bench , tau2-Noise (LongCat) , VITA-Bench , VitaBench-Noise (LongCat) , Random Complex Tasks (LongCat) , Humanity's Last Exam , LiveCodeBench v6 , OJBench , OIBench , SWE-bench Verified
- 2 Jul 2026
- miniF2F 100
- PutnamBench 87.4
- FATE-H 87
- FATE-X 34
- FLTEval 43.2
Dropped since the last post: SWE-bench Verified , tau3-bench , BrowseComp , AIME 2025 , IFBench , COLLIE , Beyond AIME
Its technical report adds 1 evaluation 1 new 12 dropped 6 results in all
LeanstralCapabilities
- ArXivLean 17.1
Dropped since the lab’s previous technical report: LibriSpeech , FLEURS , Common Voice , Multilingual LibriSpeech , VoiceBench , MMAU , Audio Understanding Benchmark (Mistral) , MMLU , GSM8K , MATH , Natural Questions , TriviaQA
- 2 Jul 2026
- SWE-bench Verified 70.9
- SWE-bench Multilingual 63.1
- SWE-Bench Pro 47.6
- Terminal-Bench 2.0 37.5
- 2 Jul 2026
- SWE-bench Multilingual 75.8
- SWE-bench Verified 78
- SWE-Bench Pro 57.9
- Terminal-Bench 2.1 71.7
- NL2Repo-Bench 45.6
- DeepSWE 28
- Hy-Backend 2.0 25
- Hy-SWE Max 49
- Hy-CompanyBench 41.7
- BrowseComp 84.2
- WideSearch 76.4
- DeepSearchQA 91
- MCP Atlas 79.1
- Toolathlon 48.5
- APEX-Agents 25.6
- Claw-Eval Pass^3 68.5
- WildClawBench 53.6
- SkillsBench 55.3
- E-Bench (Tencent) 50.2
- Hy-FinModelBench 69
- ProdBench (Tencent) 23
- Hy-SkillsWorld 45.8
- Humanity's Last Exam 53.2
- Hy-Euler pro 24.2
- GPQA Diamond 90.4
- FrontierScience 48.1
- USAMO 2026 72
- MathArena (Apex, IMO, USAMO) 38.7
- ArXivMath 52.2
- HorizonMath 7.1
- Hy-Math 60.9
- PHYBench 77.4
- CMT-Benchmark 37.8
- IMO-AnswerBench 90
- SUPERChem 54.9
- CL-bench 23.8
- CL-bench Life 17
- AA-LCR (Artificial Analysis Long Context Reasoning) 73.4
Dropped since the last post: AdvancedIF , LongBench v2 , Terminal-Bench 2.0 , Hy-Backend , Hy-Vibe Bench
- 30 Jun 2026Claude Sonnet 5 Anthropic Introducing Claude Sonnet 5 (launch post); System Card: Claude Sonnet 5, Table 8.1.A
- SWE-Bench Pro 63.2
- Terminal-Bench 2.1 80.4
- Humanity's Last Exam 57.4
- OSWorld-Verified 81.2
- GDPval-AA v2 1618
- BrowseComp 86.6
- FrontierCode 38.8
- AutomationBench 13.5
- Harvey LAB (Legal Agent Benchmark) 8.9
- HealthBench Professional 57.8
Dropped since the last post: GDPval-AA , GDP.pdf , Blueprint-Bench 2 , BioMysteryBench , HealthBench , SWE-bench Verified , CharXiv , CritPt , ArXivMath , Riemann-bench , GraphWalks , OfficeQA
Its system card adds 67 evaluations 43 new 11 dropped 77 results in all
System Card: Claude Sonnet 5Capabilities
- SWE-bench Verified 85.2
- SWE-bench Multilingual 78.3
- SWE-bench Multimodal 28.1
- CursorBench 3.1 61.2
- USAMO 2026 79.5
- ArXivMath 65.7
- ProgramBench
- GDP.pdf 67.5
- BenchCAD 26.6
- ChartMuseum 70.1
- CharXiv 77
- OfficeQA 73.3
- OfficeQA Pro 59.4
- Real-World Finance v2 (Anthropic) 1219
- Toolathlon 54.3
- AA-Briefcase 1393
- Global MMLU 89
- MILU 89.3
- INCLUDE 86.5
- BioMysteryBench
- SpatialBench (LatchBio) 70
- scBench (LatchBio SingleCellBench) 56
- Structural Biology (Anthropic) 69
- ProteinGym 37
- Organic Chemistry (Anthropic) 81
- Protocols (Anthropic life sciences) 62
Health
- HealthBench 58.7
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 96.7
- Single-turn benign request evaluations (Anthropic) 0.59
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 92.4
- Malicious computer use (Anthropic) 84.7
Robustness
- Gray Swan Agent Red Teaming (ART) benchmark 0.1
- Gray Swan IPI Arena benchmark 0.6
- Prompt injection live bug bounty (Anthropic with Gray Swan) 0.19
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 0.31
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 2.25
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 0.93
Alignment
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Rewind-attacker cyber adversarial robustness (Anthropic) , Lying about identity (Anthropic) , Refusal to assist with AI safety R&D (Anthropic) , FrontierSWE v1 , Blueprint-Bench 2 , LAB-Bench FigQA , Vals Finance Agent v2 , Real-World Finance (Anthropic) , MCP Atlas , Vending-Bench 2 , GDPval-AA
- 23 Jun 2026
- KINA 48.3
- SuperGPQA 70.8
- Beyond AIME 87
- WorkSpaceBench 53
- Agent Startup Bench 68.8
- xDailyBench 61
- NL2Repo-Bench 47
- ProgramBench 50.3
- Terminal-Bench 2.1 71
- SWE Atlas 35.2
- MATH-Vision (MathVision) 94.5
- MMMU-Pro 82.7
- WorldVQA 53
- ZeroBench 22
- BabyVision 73.7
- CharXiv 86.4
- ERQA 72
- MMLongBench 78.3
- Video-MME 89.2
- TOMATO 79.5
- MINERVA 70.7
- OVO-Bench 80.7
- Video SimpleQA 76.4
- GDPval 87.9
- Agents' Last Exam 19.5
- Doubao Multi-Turn Bench 52.5
- Toolathlon 50.6
- SeedClawBench 66.6
- Claw-Eval-MM 51
- Image2FloorPlan 48
- MobileWorld 73.1
- OSWorld 78.8
- CreativeWork 42.5
- MeasureBench 62.9
- TVBench 80.5
- LVBench 78
- OVBench 70
- SciCode 59.8
- FrontierScience 75
- MSQA 50.2
Dropped since the last post: GPQA Diamond , Humanity's Last Exam , SUPERChem , BABE , CL-bench , MultiChallenge , WideSearch , BrowseComp , ResearchRubrics , XpertBench , SkillsBench , FinSearchComp , SWE-bench Multilingual , SWE-Bench Pro , PaperBench , Terminal-Bench 2.0 , HiPhO , MedXpertQA-MM , VLMsAreBiased , SimpleVQA , OSWorld-Verified , Video-MMMU , MMVU , Video-Holmes , EgoTempo , MotionBench , VideoMME v2 (with subtitles) , LongVideoBench , OmniVideoBench , JointAVBench , WorldSense , MMSU , WildSpeech-Bench , WenetSpeech , LibriSpeech , FLEURS , VideoReasonBench , ContPhy , MORSE-500 , CG-Bench , VideoEval-Pro , ODV-Bench , LiveSports-3K , ViSpeak-Bench , CrossVid , AVMeme Exam
Its model card adds 54 evaluations 39 new 76 dropped 93 results in all
Seed2.1 Model CardCapabilities
- PresentBench 54.6
- $OneMillion-Bench 68.8
- OfficeQA Pro 70.9
- Vals Finance Agent 60.7
- APEX-Agents 33.8
- MCP Atlas 83.8
- SkillsBench 60.4
- WildClawBench 61.7
- GameWorld 31.2
- SWE-Bench Pro 57.5
- DeepSWE 32.7
- Trae Agent Bench
- FrontierScience-Research 28.3
- FrontierCS
- HorizonMath 2
- IMO 2025 65.2
- IMO-ProofBench 54.3
- IPhO 2025 79.3
- ChartQA Pro 70.9
- OCRBench v2 63.2
- CharXiv (DQ) 95.5
- MathVista 90.7
- DynaMath 73.1
- MathVerse 89.7
- EMMA 79.3
- VisuLogic 54.3
- VLMsAreBiased 83.6
- VisFactor 51.4
- RealWorldQA 86.7
- SimpleVQA 74.5
- WorldBench 67.6
- BLINK 81.4
- MMSI-Bench 35.9
- TreeBench 71.1
- EmbSpatialBench 83.4
- DUDE 82.8
- Video-Holmes 68.2
- MotionBench 74.9
- ContPhy 63.6
- LongVideoBench 80.6
- CrossVid 65
- LiveSports-3K 76.8
- HLE-Verified 42.9
- SUPERChem 59.8
- ARC-AGI-2 62.5
- FrontierScience-Olympiad 75
- LiveMathematicianBench 20.9
- MathArena (Apex, IMO, USAMO) 31.3
- AetherCode 65.8
- Humanity's Last Exam 55.7
- BrowseComp 86.2
Cyber risk
- CyberGym 70.2
AI R&D risk
- SeedKernelBench 9.21
- PostTrainBench 16.5
Dropped since the lab’s previous model card: AIME 2025 , HMMT 2025 , AMO-Bench , IMO-AnswerBench , LiveCodeBench v6 , LiveCodeBench Pro , GPQA Diamond , PHYBench , BIOBench (ByteDance Seed) , KOR-Bench , ARC-AGI , Inverse IFEval , MARS-Bench , MultiChallenge , COLLIE , EIFBench , MMLU , MMLU-Pro , LPFQA , Education (ByteDance Seed in-house) , Customer Support Q&A (ByteDance Seed in-house) , Information Processing (ByteDance Seed in-house) , Intention Recognition (ByteDance Seed in-house) , Information Extraction (ByteDance Seed in-house) , Complex Workflow (ByteDance Seed in-house) , MMMU , LogicVista , Scientists' First Exam (SFE) , VPCT , VLMs Are Blind , HallusionBench , MMStar , MMBench-EN v1.1 , MME-CC , MuirBench , MMVP , ScreenSpot-Pro , CountBench , Point-Bench , RefSpatial-Bench , CVBench , AI2D , OmniDocBench v1.5 , Video-MMMU , MMVU , VCR-Bench , VideoReasonBench , TempCompass , EgoTempo , CG-Bench , ViSpeak-Bench , StreamingBench , OmniMMI , ZeroVideo , BrowseComp-ZH , GAIA , WideSearch , MM-BrowseComp , HLE-VL , SWE-bench Verified , Multi-SWE-bench , AInstein-SWE-Bench , Terminal-Bench 2.0 , U-Artifacts , BFCL V4 , tau2-bench , DeepConsult , DeepResearch Bench , DiscoX , FinSearchComp , XpertBench , WorldTravel , Online-Mind2Web , AndroidWorld , AIR-Bench 2024 , XSTest
- 22 Jun 2026
- AgentWorldBench 56.4
Dropped since the last post: SWE-bench Verified , SWE-Bench Pro , SWE-bench Multilingual , Terminal-Bench 2.0 , SkillsBench , QwenWebBench , NL2Repo-Bench , Claw-Eval , Claw-Eval Pass^3 , QwenClawBench , MMLU-Pro , MMLU-Redux , SuperGPQA , C-Eval , GPQA Diamond , Humanity's Last Exam , LiveCodeBench v6 , HMMT 2025 , HMMT Nov 2025 , HMMT 2026 , IMO-AnswerBench , AIME 2026 , MMMU , MMMU-Pro , MathVista , DynaMath , VLMs Are Blind , RealWorldQA , MMStar , MMBench-EN v1.1 , SimpleVQA , CharXiv , CC-OCR , OCRBench , ERQA , CountBench , RefCOCO / RefCOCO+ / RefCOCOg , EmbSpatialBench , RefSpatial-Bench , Video-MME , Video-MMMU , MLVU , MVBench , V* , AndroidWorld
Its technical report adds 0 evaluations 28 dropped 1 results in all
Qwen-AgentWorld: Language World Models for General AgentsDropped since the lab’s previous technical report: SWE-bench Verified , SWE-bench Multilingual , SWE-Bench Pro , Terminal-Bench 2.0 , Aider Polyglot , Template Following generalization (Qwen) , EvalPlus , MultiPL-E , CRUXEval , LiveCodeBench v6 , OJBench , Codeforces rating , FullStackBench , Spider , Bird-SQL , MMLU , MMLU-Redux , MMLU-Pro , GPQA Diamond , SuperGPQA , HMMT 2025 , HMMT Nov 2025 , AIME 2024 , AIME 2025 , AthenaBench , PrimeVul , SecCodeBench , CWEval
- 16 Jun 2026
- Humanity's Last Exam 54.7
- CritPt 20.9
- AIME 2026 99.2
- HMMT Nov 2025 94.4
- HMMT 2026 92.5
- IMO-AnswerBench 91
- GPQA Diamond 91.2
- SWE-Bench Pro 62.1
- NL2Repo-Bench 48.9
- DeepSWE 46.2
- ProgramBench 63.7
- Terminal-Bench 2.1 82.7
- FrontierSWE v1 74.4
- PostTrainBench 34.3
- SWE-Marathon 13
- MCP Atlas 76.8
- Toolathlon 48.2
Dropped since the last post: Terminal-Bench 2.0 , CyberGym , BrowseComp , tau3-bench , Vending-Bench 2
- 12 Jun 2026
- Kimi Code Bench v2 62
- ProgramBench 53.6
- MLS-Bench 35.1
- Kimi Claw 24/7 Bench 46.9
- MCP Atlas 76
- MCPMark 81.1
Dropped since the last post: Humanity's Last Exam , BrowseComp , DeepSearchQA , WideSearch , Toolathlon , Claw-Eval , APEX-Agents , OSWorld-Verified , Terminal-Bench 2.0 , SWE-Bench Pro , SWE-bench Multilingual , SWE-bench Verified , SciCode , OJBench , LiveCodeBench v6 , AIME 2026 , HMMT 2026 , IMO-AnswerBench , GPQA Diamond , MMMU-Pro , CharXiv , MATH-Vision (MathVision) , BabyVision , V*
- 10 Jun 2026
- MMLU-Pro 77.6
- AIME 2026 69.1
- LiveCodeBench v6 69.1
- Codeforces rating 1429
- GPQA Diamond 73.2
- tau2-bench 56.2
- Humanity's Last Exam 11.9
- BIG-Bench Extra Hard 47.6
- MMMLU 81.5
- MMMU-Pro 54.3
- OmniDocBench v1.5 0.319
- MATH-Vision (MathVision) 70.5
- MedXpertQA-MM 49
- MRCR v2 32
Its technical report adds 9 evaluations 8 new 14 dropped 18 results in all
DiffusionGemma Technical ReportCapabilities
- GSM8K 96.3
- MGSM 84.8
- PutnamBench 67.4
- HumanEval 94.5
- BigCodeBench 46
- LBPP (Less Basic Python Problems) 81
- IFEval 97.4
- Natural2Code 94
- HiddenMath 80.6
Dropped since the lab’s previous technical report: LMArena (Elo) , SciCode , Humanity's Last Exam , IFBench , MRCR v2 , Terminal-Bench Hard , MATH-Vision (MathVision) , MedXpertQA-MM , InfographicVQA , OmniDocBench v1.5 , RULER , LOFT , GraphWalks , MTOB
- 9 Jun 2026 North Mini Code 1.0 Cohere Introducing North Mini Code: Cohere's first model for developers (Cohere blog)
- Terminal-Bench 2.0 36
- Terminal-Bench Hard 31.1
- SWE-bench Verified 67.6
- SWE-Bench Pro 40.2
- SciCode 38.2
- LiveCodeBench v6 70.3
Dropped since the last post: Open ASR Leaderboard , LibriSpeech
- 9 Jun 2026Claude Fable 5 Anthropic Claude Fable 5 and Claude Mythos 5 (launch post); System Card: Claude Fable 5 & Claude Mythos 5, Table 8.1.A
- SWE-Bench Pro 80.3
- FrontierCode 29.3
- GDPval-AA 1932
- GDP.pdf 29.8
- Blueprint-Bench 2 38.6
- AutomationBench 17.4
- OSWorld-Verified 85
- Harvey LAB (Legal Agent Benchmark) 16.9
- Humanity's Last Exam 64.5
- BioMysteryBench 83.9
- Terminal-Bench 2.1 88
- HealthBench Professional 66
- HealthBench 62.7
- SWE-bench Verified 95.5
- BrowseComp 93.3
- CharXiv 93.5
- CritPt 28.6
- ArXivMath 78.5
- Riemann-bench 55
- GraphWalks 100
- OfficeQA 57.9
Dropped since the last post: SWE-bench Multilingual , SWE-bench Multimodal , ChartQA Pro , GPQA Diamond , ScreenSpot-Pro , Vals Finance Agent v2 , MCP Atlas
Its system card adds 31 evaluations 5 new 64 dropped 41 results in all
System Card: Claude Fable 5 & Claude Mythos 5Capabilities
- FrontierSWE v1
- CursorBench 3.1 72.9
- LAB-Bench FigQA
- OfficeQA Pro 57.9
- Vals Finance Agent v2 56.3
- Real-World Finance v2 (Anthropic) 1374
- Real-World Finance (Anthropic) 70
- MCP Atlas 83.3
- Vending-Bench 2 5680.3
- Toolathlon 61.7
Safety
- Single-turn harmful request evaluations (Anthropic) 96.9
- Single-turn benign request evaluations (Anthropic) 0.01
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 90.3
- Malicious computer use (Anthropic) 85.7
Robustness
- Rewind-attacker cyber adversarial robustness (Anthropic) 5
- Gray Swan Agent Red Teaming (ART) benchmark 0.1
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 0.45
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 0.82
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 29.7
Alignment
Dropped since the lab’s previous system card: Long-form virology tasks , Virology Capabilities Test (VCT) , DNA Synthesis Screening Evasion , Black-box RNA sequence modeling and design , AAV capsid packaging prediction , Anthropic ECI (AECI) , ExploitBench , CyberGym , Firefox 147 exploit evaluation (Anthropic) , Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) , Prompt injection live bug bounty (Anthropic with Gray Swan) , Petri , Evaluation awareness discriminability (Anthropic) , UK AISI unprompted sabotage evaluation , UK AISI sabotage continuation evaluation , Over-eager behavior on GUI computer use tasks (Anthropic) , Adherence to its constitution (Anthropic) , 100Q-Hard (Anthropic) , SimpleQA Verified , AA-Omniscience , ECLeKTic , False-premise questions (Anthropic) , MASK , Missing-context hallucinations (Anthropic) , Self-preference bias (Anthropic) , Uncritically reporting flawed results (Anthropic) , Code summary honesty (Anthropic) , Lazy investigation on misleading codebases (Anthropic) , Overconfidence: verifying command-line syntax (Anthropic) , SHADE-Arena , Minimal-LinuxBench (Anthropic) , Intentionally taking actions very rarely (Anthropic) , Hiding a secret password (Anthropic) , CoT-Control , OpenAI monitorability evals , SWE-bench Multilingual , SWE-bench Multimodal , BrowseComp , Humanity's Last Exam , ChartQA Pro , GPQA Diamond , ScreenSpot-Pro , GraphWalks , ProgramBench , USAMO 2026 , ArXivMath , DeepSearchQA , DRACO , ChartMuseum , CharXiv , OfficeQA , HealthBench Professional , Global MMLU , MILU , INCLUDE , BioPipelineBench (Anthropic) , BioMysteryBench , SpatialBench (LatchBio) , scBench (LatchBio SingleCellBench) , Structural Biology (Anthropic) , ProteinGym , Organic Chemistry (Anthropic) , Protocols (Anthropic life sciences) , LABBench2
- 8 Jun 2026
- 4 Jun 2026
- Terminal-Bench 2.1 56.4
- GDPval 46.7
- SWE-bench Verified 70.7
- SWE-bench Multilingual 67.7
- ProfBench 56
- PinchBench 90
- tau3-bench 86.9
- τ³-bench Banking 22.6
- BrowseComp 44.4
- Vals Finance Agent 60.1
- IOI 2025 570
- LiveCodeBench v6 89
- IMO-AnswerBench 88.6
- MathArena (Apex, IMO, USAMO) 74.9
- GPQA Diamond 87
- SciCode 44.6
- Humanity's Last Exam 26.7
- CritPt 3.1
- MMLU-Pro 86.8
- AA-Omniscience 24.1
- IFBench 81.7
- MultiChallenge 63.8
- AA-LCR (Artificial Analysis Long Context Reasoning) 65.4
- RULER 94.7
- LongBench v2 61.9
- MMLU-ProX 83
- WMT24++ 83.7
Dropped since the last post: AIME 2025 , HMMT 2025 , LiveCodeBench v5 , Terminal-Bench Hard , Terminal-Bench 2.0 , tau2-bench , Bird-SQL , Arena-Hard v2.0
Its technical report adds 4 evaluations 4 new 8 dropped 31 results in all
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic ReasoningCapabilities
- IMO-ProofBench 82.3
- IMO 2025 83.3
- Putnam 2025 96.7
- USAMO 2026 97.6
Dropped since the lab’s previous technical report: AIME 2025 , HMMT 2025 , LiveCodeBench v5 , Terminal-Bench Hard , Terminal-Bench 2.0 , tau2-bench , Bird-SQL , Arena-Hard v2.0
- 2 Jun 2026
- SWE-bench Verified 71.6
- SWE-Bench Pro 51.2
- SWE-bench Multilingual 65.5
- Terminal-Bench 2.0 54.8
- AIME 2026 92.5
- AMO-Bench 40
- FrontierMath 6.3
- Humanity's Last Exam 18
- GPQA Diamond 84.6
- FrontierScience 58.2
- ArtifactsBench 36.4
- IFBench 75
- AdvancedIF 71.4
- Robust IF Bench 61.2
- tau2-bench 71.7
Dropped since the last post: AIME 2025 , HMMT 2026 , LiveCodeBench v6 , MMLU-Pro , SimpleQA Verified , MultiChallenge , GraphWalks , Berkeley Function Calling Leaderboard (BFCL) , LongFact , TruthfulQA , HealthBench Professional , MedXpertQA
Its model card adds 0 evaluations 15 results in all
- 2 Jun 2026
MAI-Thinking-1 Microsoft MAI-Thinking-1: Building a Hill-Climbing Machine (Microsoft AI technical report), Tables 11 and 12 - AIME 2025 97
- AIME 2026 94.5
- HMMT 2026 84.9
- GPQA Diamond 84.2
- LiveCodeBench v6 87.7
- Terminal-Bench 2.0 46
- SWE-bench Verified 73.5
- SWE-Bench Pro 52.8
- MMLU-Pro 85
- SimpleQA Verified 31
- IFBench 69
- AdvancedIF 85
- MultiChallenge 53
- GraphWalks 90
- Berkeley Function Calling Leaderboard (BFCL) 72
- LongFact 98
- TruthfulQA 88
- HealthBench Professional 35
- MedXpertQA 43
Dropped since the last post: WebVoyager , Online-Mind2Web , WebTailBench
Its technical report adds 9 evaluations 28 results in all
MAI-Thinking-1: Building a Hill-Climbing MachineCapabilities
- LongBench v2 61
- CorpusQA 82
- OpenAI MRCR v2 60
- MAI human side-by-side evaluation 0.07
Robustness
Cyber risk
- 1 Jun 2026
- Terminal-Bench 2.0 70.3
- SWE-bench Verified 77.7
- SWE-Bench Pro 57.6
- SWE-bench Multilingual 75.8
- NL2Repo-Bench 41.1
- SciCode 51.3
- QwenWebDev 1536
- QwenSVG 1588
- QwenClawBench 61.8
- CoWorkBench 65.1
- Claw-Eval 62.7
- SkillsBench 54.9
- BFCL V4 72.9
- MCPMark 58.7
- MCP Atlas 73.2
- VITA-Bench 45.6
- DeepPlanning 62.3
- SpreadsheetBench 86.3
- Kernel Bench L3 2.06
- QwenWorldBench 62.1
- GPQA Diamond 90.3
- Humanity's Last Exam 34.7
- LiveCodeBench 89.6
- HMMT 2026 92.9
- IMO-AnswerBench 86
- CritPt 6
- MathArena (Apex, IMO, USAMO) 22.7
- MMLU-Pro 88.5
- MMLU-Redux 94.5
- SuperGPQA 71.4
- IFEval 94.6
- IFBench 79.1
- MRCR v2 91.7
- WMT24++ 84.6
- MAXIFE 88.8
- MMMLU 89
- MMLU-ProX 85.4
- NOVA-63 58.8
- INCLUDE 83
- Global PIQA 90.3
- PolyMath 84
- MMMU-Pro 79
- MATH-Vision (MathVision) 90.3
- BabyVision 70.4
- CharXiv 85.9
- HiPhO 84.1
- ERQA 69.8
- VisFactor 42.8
- MedXpertQA-MM 71
- ScreenSpot-Pro 79
- OSWorld-Verified 73.3
- AndroidWorld 81
- QwenVision2Code 1772
- Claw-Eval-MM 55.7
- SimpleVQA 81.7
- WorldVQA 61.1
- MMSearch-Plus 41.4
- BrowseComp-VL 51.1
- MM-BrowseComp 46.3
- RealWorldQA 86.9
- CountQA 77
- OmniDocBench v1.5 91.4
- OCRBench v2 70.7
- ODinW-13 51.1
- LingoQA 83.4
- Ego3D-Bench 5.9
- SURDS 77.2
- VLADBench 77.2
- Video-MME 88
- Video-MMMU 85.4
- MLVU 87.4
- TVBench 78.2
- LVBench 76.2
- 1 Jun 2026
MiniMax-M3 MiniMax MiniMax M3 launch blog (minimax.io/blog/minimax-m3) and Hugging Face card benchmark table (figures/benchmark.jpeg) - SWE-bench Verified 80.5
- SWE-Bench Pro 59
- Terminal-Bench 2.1 66
- SWE Atlas 37.9
- NL2Repo-Bench 42.1
- SWE Atlas 30.8
- SWE-fficiency 34.8
- LiveSQLBench 40.2
- CL-bench 20.5
- VIBE-V2 (MiniMax) 50.1
- SVG-Bench (MiniMax) 63.7
- PostTrainBench 37.1
- KernelBench 28.8
- PaperBench 52.6
- BrowseComp 83.5
- DRACO 73.2
- GDPval-Rubrics (MiniMax) 74.8
- BankerToolBench 76.1
- OfficeQA 45.1
- SpreadsheetBench 89.3
- YC-Bench 2100000
- LOCA-Bench 49.3
- MCP Atlas 74.2
- APEX-Agents 27.7
- Claw-Eval 74.5
- OSWorld-Verified 75.2
- OmniDocBench 91.6
- MMMU-Pro 78.1
- Video-MMMU 84.6
- Video-MME 85.4
- IMO 2025 35
- USAMO 2026 36
Dropped since the last post: Multi-SWE-bench , SWE-bench Multilingual , VIBE-Pro (MiniMax) , Terminal-Bench 2.0 , MLE-bench , GDPval-AA , Toolathlon , MM-ClawBench , Artificial Analysis Intelligence Index v4.0
- 29 May 2026
- Humanity's Last Exam 47.2
- BrowseComp 75.8
- DeepSearchQA 92.8
- ResearchRubrics 71.7
- Toolathlon 49.5
- Claw-Eval Pass^3 67.1
- GDPval-AA 1415.8
- SWE-bench Multilingual 72.4
- SWE-Bench Pro 56.3
- SWE-bench Verified 76.5
- Terminal-Bench 2.1 59.6
- AA-LCR (Artificial Analysis Long Context Reasoning) 63.9
- SimpleVQA 79.2
- WorldVQA 58.1
- BrowseComp-VL 59
- V* 95.3
- HR-Bench 87.7
- VisualProbe 65
- AndroidDaily 61.9
- Step-SWE-Bench 67.1
Dropped since the last post: tau2-bench , BrowseComp-ZH , GAIA , xbench-DeepSearch , xbench-DeepSearch-2510 , AIME 2025 , HMMT 2025 , HMMT Nov 2025 , IMO-AnswerBench , LiveCodeBench v6 , Terminal-Bench 2.0
- 28 May 2026
- AA-Omniscience -24.7
- IFEval 91.8
- IFBench 56.5
- Multi-IF 79.9
- MATH-500 88.8
- AIME 2025 42.5
- AIME 2026 50
- Berkeley Function Calling Leaderboard (BFCL) 64.4
- BFCL V4 48.5
- tau2-bench 64
Dropped since the last post: MMLU , MMLU-Pro , GPQA Diamond , GSM8K , GSM-Plus , MATH Level 5 , MGSM , MMMLU , LiveCodeBench v6 , LiveCodeBench v5 , HumanEval+ , Creative Writing v3
- 28 May 2026Claude Opus 4.8 Anthropic Introducing Claude Opus 4.8 (launch post); System Card: Claude Opus 4.8, Table 8.1.A
- SWE-bench Verified 88.6
- SWE-Bench Pro 69.2
- SWE-bench Multilingual 84.4
- SWE-bench Multimodal 38.4
- BrowseComp 88.5
- Terminal-Bench 2.1 74.6
- Humanity's Last Exam 57.9
- ChartQA Pro 72.3
- OSWorld-Verified 83.4
- GPQA Diamond 93.6
- ScreenSpot-Pro 87.9
- Vals Finance Agent v2 53.9
- GDPval-AA 1890
- MCP Atlas 82.2
- AutomationBench 15.5
- GraphWalks 99.3
Dropped since the last post: Terminal-Bench 2.0 , Vals Finance Agent , CharXiv , MMMLU , OfficeQA , OfficeQA , ARC-AGI , ARC-AGI-2
Its system card adds 81 evaluations 25 new 23 dropped 97 results in all
System Card: Claude Opus 4.8Capabilities
- FrontierSWE v1
- ProgramBench 79
- USAMO 2026 96.7
- ArXivMath 71.8
- DeepSearchQA 93.1
- DRACO 80.4
- ChartMuseum 75.8
- LAB-Bench FigQA 80.4
- CharXiv 80.5
- OfficeQA 77.6
- OfficeQA Pro 66.2
- Harvey LAB (Legal Agent Benchmark) 9.62
- Vending-Bench 2 2992.3
- Toolathlon 59.9
- Global MMLU 90.4
- MILU 90.3
- INCLUDE 87.6
- BioPipelineBench (Anthropic) 87.7
- BioMysteryBench
- SpatialBench (LatchBio) 53.3
- scBench (LatchBio SingleCellBench) 58.2
- Structural Biology (Anthropic) 79
- ProteinGym 39.6
- Organic Chemistry (Anthropic) 86.2
- Protocols (Anthropic life sciences) 59.6
- LABBench2
Health
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 98
- Single-turn benign request evaluations (Anthropic) 0.36
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 95.1
- Malicious computer use (Anthropic) 81.7
Robustness
- Gray Swan Agent Red Teaming (ART) benchmark 0.1
- Prompt injection live bug bounty (Anthropic with Gray Swan) 0.4
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 7.03
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 7.14
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 31.5
Alignment
- Anthropic automated behavioral audit
- Petri
- Evaluation awareness discriminability (Anthropic)
- UK AISI unprompted sabotage evaluation 0
- UK AISI sabotage continuation evaluation 1
- Over-eager behavior on GUI computer use tasks (Anthropic) 9.4
- Adherence to its constitution (Anthropic) 0.99
- MASK 95.4
- Lying about identity (Anthropic) 97
- Refusal to assist with AI safety R&D (Anthropic)
- Self-preference bias (Anthropic)
- Uncritically reporting flawed results (Anthropic) 9.4
- Code summary honesty (Anthropic) 3.7
- Lazy investigation on misleading codebases (Anthropic) 0
- Overconfidence: verifying command-line syntax (Anthropic) 4.51
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- Anthropic ECI (AECI) 155.5
Dropped since the lab’s previous system card: Kernels task (Anthropic) , Time series forecasting task (Anthropic) , LLM training task (Anthropic) , Quadruped reinforcement learning task (Anthropic) , Novel compiler task (Anthropic) , Cybench , UK AISI cyber ranges , Higher-difficulty harmful request evaluations (Anthropic) , Higher-difficulty benign request evaluations (Anthropic) , Impossible tasks (Anthropic) , Agentic code behavior scores (Anthropic) , Destructive production evaluation (Anthropic) , Destructive actions (Anthropic) , Input hallucinations (Anthropic) , DTBench , Terminal-Bench 2.0 , MMMLU , Vals Finance Agent , ARC-AGI , ARC-AGI-2 , OpenAI MRCR v2 , BioPipelineBench Verified (Anthropic) , Phylogenetics (Anthropic)
- 25 May 2026
- LongVideoBench 74.1
- VideoMME v2 (with subtitles) 42.4
- TimeLens-Bench 62.3
- MLVU 82.8
- Video-MME 78.3
- Video-MMMU 80
- LiveCodeBench v6 77.1
- OJBench 39.2
- SWE-bench Verified 62
- BFCL V4 65.7
- tau2-bench 82.6
- VITA-Bench 33.1
- We-Math 75.5
- MathVista 82.9
- DynaMath 80.9
- AIME 2025 86.7
- AIME 2026 93.3
- Humanity's Last Exam 22.9
- MMMU 75.3
- MMStar 75.9
- C-Eval 89.2
- IFEval 92.2
- HallusionBench 73.5
- AI2D 87.8
- OCRBench 85.7
- OmniDocBench v1.5 84.7
- RefCOCO / RefCOCO+ / RefCOCOg 88.1
- PixMo-Count 83.5
- EmbSpatialBench 83.2
Its technical report adds 0 evaluations 29 results in all
- 22 May 2026
- SWE-bench Verified 77.6
- tau3-bench 63.2
- BrowseComp 48.6
- AIME 2025 86.3
- IFBench 69
- COLLIE 95.8
- Beyond AIME 66.9
Dropped since the last post: FLTEval
- 21 May 2026
- WebVoyager 89.3
- Online-Mind2Web 72.3
- WebTailBench 40.2
Its technical report adds 4 evaluations 4 new 28 dropped 7 results in all
Fara-1.5: Scalable Learning Environments for Computer Use AgentsCapabilities
- ScreenSpot-v2 94.4
- ScreenSpot-Pro 66.8
- OSWorld-G 71.3
Safety
Dropped since the lab’s previous technical report: AIME 2025 , AIME 2026 , HMMT 2026 , GPQA Diamond , LiveCodeBench v6 , Terminal-Bench 2.0 , SWE-bench Verified , SWE-Bench Pro , MMLU-Pro , SimpleQA Verified , IFBench , AdvancedIF , MultiChallenge , GraphWalks , Berkeley Function Calling Leaderboard (BFCL) , AIR-Bench 2024 , CyberSecEval Instruct , CyberSecEval Autocomplete , LongFact , TruthfulQA , HealthBench Professional , MedXpertQA , LongBench v2 , CorpusQA , OpenAI MRCR v2 , MAI human side-by-side evaluation , MAI internal safety and over-refusal evaluation , MAI internal jailbreak evaluation suite
- 20 May 2026 Command A+ Cohere Introducing Command A+: Making sovereign agentic capabilities available to all (Cohere blog)
- tau2-bench 85
- Terminal-Bench Hard 25
- IFBench 74
- AIME 2025 90
- SciCode 38
- MMMU 75.1
- MMMU-Pro 63
- MathVista 80.6
- CharXiv 52.7
- MT-AIME (Cohere internal) 86
- WMT24++ 81
- Artificial Analysis Intelligence Index v4.0 37
- 20 May 2026
- Terminal-Bench 2.0 69.7
- SWE-bench Verified 80.4
- SWE-Bench Pro 60.6
- SWE-bench Multilingual 78.3
- NL2Repo-Bench 47.2
- SciCode 53.5
- QwenWebDev 1568
- QwenSVG 1608
- QwenClawBench 64.3
- CoWorkBench 67.2
- Claw-Eval 65.2
- SkillsBench 59.2
- BFCL V4 75
- MCPMark 60.8
- MCP Atlas 76.4
- VITA-Bench 47.9
- SpreadsheetBench 87
- Kernel Bench L3 1.98
- Humanity's Last Exam 53.5
- QwenWorldBench 57.3
- GPQA Diamond 92.4
- LiveCodeBench 91.6
- HMMT 2026 97.1
- IMO-AnswerBench 90
- CritPt 11.4
- MathArena (Apex, IMO, USAMO) 44.5
- MMLU-Pro 89.6
- MMLU-Redux 95
- SuperGPQA 73.6
- IFEval 94.3
- IFBench 79.1
- MRCR v2 90.4
- WMT24++ 85.8
- MAXIFE 89.2
- MMMLU 90.3
- MMLU-ProX 87
- NOVA-63 59
- INCLUDE 86.2
- Global PIQA 91.4
- PolyMath 86.5
Dropped since the last post: AA-Omniscience , GDPval-AA , QwenChineseBench , ToolcallFormatIFBench , QwenWebBench
- 19 May 2026
- Terminal-Bench 2.1 76.2
- SWE-Bench Pro 55.1
- MCP Atlas 83.6
- Toolathlon 56.5
- OSWorld-Verified 78.4
- Vals Finance Agent v2 57.9
- GDPval-AA 1656
- CharXiv 84.2
- MMMU-Pro 83.6
- Blueprint-Bench 2 33.6
- MRCR v2 77.3
- Humanity's Last Exam 40.2
- ARC-AGI-2 72.1
Dropped since the last post: Success detection (Google DeepMind) , Instrument reading (Google DeepMind) , ASIMOV
Its model card adds 5 evaluations 14 dropped 18 results in all
Gemini 3.5 Flash Model CardDropped since the lab’s previous model card: GPQA Diamond , Terminal-Bench 2.0 , SWE-bench Verified , LiveCodeBench Pro , SciCode , APEX-Agents , tau2-bench , BrowseComp , MMMLU , CBRN expert red teaming , Harmful manipulation efficacy study , RE-Bench , GDM Stealth , GDM Situational Awareness
- 18 May 2026
- 14 May 2026
- PinchBench 87.6
- Claw-Eval 63.8
- tau2-bench 95.3
- ARC-AGI-2 66.2
- AIME 2026 95.8
- GPQA Diamond 88.3
Dropped since the last post: SWE-bench Verified , Claw-Eval Pass^3 , BFCL V4 , IFBench , MRCR v2
Its technical report adds 7 evaluations 3 new 12 dropped 12 results in all
Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter ScaleCapabilities
- HMMT 2026 93.5
- IMO-AnswerBench 86.1
- LiveCodeBench v6 87
- Claw-Eval Pass^3 63.8
- SWE-bench Verified 74
- SWE-Bench Pro 53.8
- Meta Agents Research Environments (ARE) / Gaia2 77.9
Dropped since the lab’s previous technical report: Chinese SimpleQA , SimpleQA Verified , SuperGPQA , Humanity's Last Exam , BIG-Bench Extra Hard , HMMT Nov 2025 , ARC-AGI , BFCL V4 , Terminal-Bench 2.0 , IFBench , LongBench v2 , MRCR v2
- 9 May 2026
- AIME 2026 99.6
- tau3-bench 67.9
- DeepSearchQA 77.3
- SpreadsheetBench Verified 72.5
- GPQA Diamond 91
- MMLU-Pro 84.3
- AdvancedIF 72.3
- LMArena Search Arena 1223
Dropped since the last post: SimpleQA , Chinese SimpleQA , Humanity's Last Exam , AIME 2025 , HMMT 2025 , LiveCodeBench v6 , HumanEval+ , MBPP+ , ZebraLogic , BIG-Bench Extra Hard , IFEval , MultiChallenge , Multi-IF , tau2-bench , AceBench , BFCL V4 , BrowseComp-ZH , SpreadsheetBench , MMMU-Pro , MathVista , MathVerse , MATH-Vision (MathVision) , VisualPuzzles , VisuLogic , VLMs Are Blind , ChartQA , AI2D , DocVQA , OCRBench , CharXiv , SimpleVQA , HallusionBench , MMStar , BLINK , CVBench , CountBench , Video-MME , Video-MMMU , MMVU , GenEval , VBench
- 7 May 2026
- Big Bench Audio 96.6
- Audio MC 48.5
Dropped since the last post: CharXiv , MMMU-Pro , OmniDocBench , GPQA Diamond , AIME 2025
- 5 May 2026
- CharXiv 81.6
- MMMU-Pro 76
- OmniDocBench 0.125
- GPQA Diamond 85.6
- AIME 2025 81.2
Dropped since the last post: GDPval , BrowseComp , FrontierMath , FrontierMath Tier 4 , Investment Banking Modeling Tasks (OpenAI) , GeneBench , Humanity's Last Exam
Its system card adds 22 evaluations 3 new 21 dropped 22 results in all
GPT-5.5 Instant System CardHealth
Hallucination
Safety
Robustness
Bias
Bio and chem risk
Cyber risk
- Capture the Flag (Professional) 94.1
- CVE-Bench 90.2
- Cyber Range 76.9
Dropped since the lab’s previous system card: Disallowed Content Evaluations with Representative Prompts , Destructive action avoidance , Destructive action reversion on long rollouts , User Confirmations During Computer Use , Deception Evaluations with Representative Prompts , OpenAI Deployment Simulation , CoT Monitorability , CoT-Control , Biochemistry knowledge improvement , Hard-negative protein binding prediction , DNA sequence design for transcription factor binding , VulnLMP , Atomic Challenges , CyScenarioBench , UK AISI narrow cyber tasks , UK AISI cyber ranges , Monorepo-Bench , MLE-bench , Internal Research Debugging Evaluation , OpenAI-Proof Q&A , Apollo Research strategic deception evaluations
- 5 May 2026
- MolmoSpaces 20.6
- RoboEval 0.443
- LIBERO 97.2
Dropped since the last post: WebVoyager , Online-Mind2Web , DeepShop , WebTailBench
Its technical report adds 0 evaluations 4 dropped 3 results in all
MolmoAct2: Action Reasoning Models for Real-world DeploymentDropped since the lab’s previous technical report: WebVoyager , Online-Mind2Web , DeepShop , WebTailBench
- 29 Apr 2026
- MMLU 80.2
- MMLU-Pro 64.1
- BIG-Bench Hard 83.7
- AGIEval 77.8
- GPQA Diamond 45.8
- SimpleQA 6.81
- AlpacaEval 2.0 56.2
- IFEval 89.7
- Arena-Hard-Auto 71
- MT-Bench 8.61
- GSM8K 94.2
- GSM-Symbolic 75.7
- MATH 81.3
- DeepMind Mathematics 81.9
- HumanEval 88.4
- HumanEval+ 85.4
- MBPP 85.5
- MBPP+ 73.5
- CRUXEval 55.8
- BigCodeBench 38.8
- MultiPL-E 62.3
- Berkeley Function Calling Leaderboard (BFCL) 73.7
- MMMLU 73.7
- INCLUDE 67.3
- MGSM 71.1
- SALAD-Bench 96.4
- AttaQ 85.8
- Tulu 3 Safety Eval 78.2
- 29 Apr 2026
- AIME 2026 87.4
- SWE-bench Verified 72.2
- PinchBench 87.4
- Claw-Eval Pass^3 51
- BFCL V4 70.6
- tau2-bench 78.4
- IFBench 57
- MRCR v2 80.4
Dropped since the last post: IMO-AnswerBench , HMMT Nov 2025 , LiveCodeBench v6 , Meta Agents Research Environments (ARE) / Gaia2 , ARC-AGI-2 , IMO 2025 , CMO 2025
Its technical report adds 12 evaluations 7 new 22 dropped 20 results in all
Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter ScaleCapabilities
- Chinese SimpleQA 76.5
- SimpleQA Verified 31.5
- GPQA Diamond 76.2
- SuperGPQA 58.3
- Humanity's Last Exam 10.1
- LiveCodeBench v6 65.6
- BIG-Bench Extra Hard 52.4
- HMMT Nov 2025 81.9
- IMO-AnswerBench 65.8
- ARC-AGI 50.9
- Terminal-Bench 2.0 40.5
- LongBench v2 48.3
Dropped since the lab’s previous technical report: AIME 2025 , Omni-MATH , HMMT 2025 , CNMO 2024 , FinanceReasoning , UGMathBench , Codeforces rating , Aider-Edit , ZebraLogic , MMLU-Pro , C-Eval , PHYBench , AGIEval , TriviaQA , CMMLU , Arena-Hard v2.0 , Creative Writing v3 , IFEval , HealthBench , MultiChallenge , Berkeley Function Calling Leaderboard (BFCL) , IMO 2025
- 28 Apr 2026 Nemotron 3 Nano Omni 30B-A3B NVIDIA Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 model card (Hugging Face)
- CVBench 84
- OCRBench v2 67
- OSWorld-Verified 47.4
- CharXiv 63.6
- MMLongBench-Doc 57.5
- MathVista 82.8
- OCR-Reasoning 54.1
- Video-MME 72.2
- WorldSense (omnimodal) 55.4
- Daily-Omni 74.5
- VoiceBench 89.4
- MMAU 74.6
Dropped since the last post: IMO 2025 , IMO-AnswerBench , IMO-ProofBench , AIME 2025 , AIME 2026 , HMMT 2025 , IOI 2025 , ICPC World Finals 2025 , LiveCodeBench v6 , LiveCodeBench Pro , SciCode , MMLU-Redux , MMLU-Pro , GPQA Diamond , Humanity's Last Exam , Arena-Hard v2.0 , IFBench , MultiChallenge , AA-LCR (Artificial Analysis Long Context Reasoning) , LongBench v2 , RULER , CL-bench , BFCL V4 , tau2-bench , Terminal-Bench 2.0 , SWE-bench Verified , MMLU-ProX , WMT24++
Its technical report adds 22 evaluations 15 new 22 dropped 34 results in all
Nemotron 3 Nano Omni: Efficient and Open Multimodal IntelligenceCapabilities
- MMMU 70.8
- OCRBench 86.6
- ChartQA 90.3
- DocVQA 95.6
- AI2D 88.5
- TextVQA 81
- InfographicVQA 86.8
- CharXiv (DQ) 88.9
- TreeBench 51.6
- RefCOCO / RefCOCO+ / RefCOCOg 90.5
- ScreenSpot 89.3
- ScreenSpot-v2 92.8
- ScreenSpot-Pro 57.8
- Open ASR Leaderboard 5.95
- MMLU-Pro 77.3
- GPQA Diamond 72.2
- LiveCodeBench v5 63.2
- AIME 2025 82.1
- IFBench 74.2
- AA-LCR (Artificial Analysis Long Context Reasoning) 41
- tau2-bench
- SciCode 32
Dropped since the lab’s previous technical report: IMO 2025 , IMO-AnswerBench , IMO-ProofBench , AIME 2026 , HMMT 2025 , IOI 2025 , ICPC World Finals 2025 , LiveCodeBench v6 , LiveCodeBench Pro , Codeforces rating , MMLU-Redux , Humanity's Last Exam , Arena-Hard v2.0 , MultiChallenge , LongBench v2 , RULER , CL-bench , BFCL V4 , Terminal-Bench 2.0 , SWE-bench Verified , MMLU-ProX , WMT24++
- 28 Apr 2026
- SWE-bench Verified 68.2
- SWE-bench Multilingual 62.4
- SWE-Bench Pro 44.5
- Terminal-Bench 2.0 30.1
Its technical report adds 0 evaluations 4 results in all
- 28 Apr 2026
- SWE-bench Verified 72.5
- SWE-bench Multilingual 67.3
- SWE-Bench Pro 46.9
- Terminal-Bench 2.0 40.7
Its technical report adds 0 evaluations 4 results in all
- 28 Apr 2026
- GPQA Diamond 88.4
- SuperGPQA 69.6
- Humanity's Last Exam 25.7
- Beyond AIME 79
- FrontierScience 72
- SUPERChem 55
- BABE 57.9
- CL-bench 20.1
- MultiChallenge 69.9
- WideSearch 70.3
- BrowseComp 64
- ResearchRubrics 59.2
- XpertBench 56.8
- SkillsBench 43.7
- GDPval 53.1
- FinSearchComp 63.8
- SWE-bench Multilingual 66.6
- SWE-Bench Pro 46.6
- NL2Repo-Bench 28.7
- PaperBench 52.5
- Terminal-Bench 2.0 43.3
- MATH-Vision (MathVision) 89.8
- MMMU-Pro 78.4
- HiPhO 83.8
- MedXpertQA-MM 79.6
- BabyVision 64.7
- VLMsAreBiased 80.6
- SimpleVQA 72.7
- WorldVQA 50.2
- CharXiv 82.4
- ERQA 71.5
- OSWorld-Verified 64.4
- MobileWorld 64.6
- Video-MMMU 88.3
- MMVU 76.7
- Video SimpleQA 71.7
- Video-Holmes 67.4
- MINERVA 68.5
- TVBench 80.4
- TOMATO 72.5
- EgoTempo 68.4
- MotionBench 72.4
- Video-MME 89
- VideoMME v2 (with subtitles) 64.9
- LongVideoBench 79
- LVBench 76.4
- OVBench 63.2
- OVO-Bench 75.4
- OmniVideoBench 61.7
- JointAVBench 69.5
- WorldSense 67.3
- MMSU 86.5
- WildSpeech-Bench 75.8
- WenetSpeech 4.89
- LibriSpeech 1.62
- FLEURS 74.7
- VideoReasonBench 59.4
- ContPhy 62.4
- MORSE-500 34.6
- CG-Bench 65.5
- VideoEval-Pro 49.5
- ODV-Bench 66
- LiveSports-3K 78.1
- ViSpeak-Bench 87
- CrossVid 63.7
- AVMeme Exam 69.5
Dropped since the last post: MMLU-Pro , HLE-Verified , SimpleQA Verified , HealthBench , HealthBench Hard , LPFQA , Encyclo-K , AIME 2026 , AIME 2025 , HMMT 2025 , HMMT Nov 2025 , MathArena (Apex, IMO, USAMO) , IMO-AnswerBench , Codeforces rating , AetherCode , LiveCodeBench v6 , PHYBench , ARC-AGI , ARC-AGI-2 , KOR-Bench , ProcBench , MRCR v2 , GraphWalks , LongBench v2 , FRAMES , DeR2 Bench , Global PIQA , MMMLU , DiscoX , COLLIE , MARS-Bench , Inverse IFEval , LongFact , FActScore , SWE-Lancer , SWE-bench Verified , Multi-SWE-bench , SciCode , SWE-EVO , Aider Polyglot , ArtifactsBench , CodeSimpleQA , SpreadsheetBench Verified , BrowseComp-ZH , DeepSearchQA , Seal-0 , tau2-bench , MCPMark , BFCL V4 , VITA-Bench , DeepConsult , DeepResearch Bench , MM-BrowseComp , HLE-VL , MathVista , LogicVista , VisuLogic , VLMs Are Blind , ChartQA Pro , OmniDocBench v1.5 , DUDE , MMLongBench , MMLongBench-Doc , TempCompass , Math Kangaroo 2025 (MathArena) , MathCanvas-Bench , MineDojo , PutnamBench , IMO 2025 , CMO 2025
- 27 Apr 2026
- Humanity's Last Exam 48
- GDPval-AA 1581
- tau3-bench 72.9
- Claw-Eval Pass^3 63.8
- SWE-Bench Pro 57.2
- SWE-bench Verified 78.9
- Terminal-Bench 2.0 68.4
- MiMo Coding Bench 73.7
Dropped since the last post: CharXiv , MMMU-Pro , HR-Bench , OmniDocBench , Claw-Eval-MM , Video-MME , Daily-Omni , Video-Holmes
- 24 Apr 2026
- MMLU-Pro 87.5
- SimpleQA Verified 57.9
- Chinese SimpleQA 84.4
- GPQA Diamond 90.1
- Humanity's Last Exam 48.2
- LiveCodeBench 93.5
- Codeforces rating 3206
- HMMT 2026 95.2
- IMO-AnswerBench 89.8
- MathArena (Apex, IMO, USAMO) 38.3
- MathArena (Apex, IMO, USAMO) 90.2
- MRCR v2 83.5
- CorpusQA 1M 62
- Terminal-Bench 2.0 67.9
- SWE-bench Verified 80.6
- SWE-Bench Pro 55.4
- SWE-bench Multilingual 76.2
- BrowseComp 83.4
- GDPval-AA 1554
- MCP Atlas 73.6
- Toolathlon 51.8
- AGIEval 83.1
- MMLU 90.1
- MMLU-Redux 90.8
- MMMLU 90.3
- C-Eval 93.1
- CMMLU 90.8
- MultiLoKo 51.1
- SuperGPQA 53.9
- FACTS Parametric 62.6
- TriviaQA 85.6
- BIG-Bench Hard 87.5
- DROP 88.7
- HellaSwag 88
- WinoGrande 81.5
- CLUEWSC 85.2
- BigCodeBench 59.2
- HumanEval 76.8
- GSM8K 92.6
- MATH 64.5
- MGSM 84.4
- CMATH 90.9
- LongBench v2 51.5
Dropped since the last post: AIME 2025 , HMMT 2025 , HMMT Nov 2025 , BrowseComp-ZH , tau2-bench , MCP-Universe , MCPMark
DeepSeek-V4-Flash: Its technical report adds 2 evaluations 2 new 8 dropped 21 results in all
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (arXiv 2606.19348)Capabilities
- LiveCodeBench v6 91.6
- PutnamBench 81
Dropped since the lab’s previous technical report: LiveCodeBench , AIME 2025 , HMMT 2025 , HMMT Nov 2025 , BrowseComp-ZH , tau2-bench , MCP-Universe , MCPMark
Its technical report adds 9 evaluations 8 new 2 dropped 27 results in all
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (arXiv 2606.19348)Capabilities
- LiveCodeBench v6 93.5
- OpenAI MRCR v2 83.5
- Putnam 2025 120
- DeepSeek R&D Coding Benchmark 67
- DeepSeek Chinese white-collar tasks evaluation 53
- DeepSeek Chinese functional writing evaluation 62.6
- DeepSeek Chinese creative writing evaluation
- DeepSeek complex instruction following and multi-turn writing evaluation 45.9
- DeepSeek search Q&A evaluation 28.1
Dropped since the lab’s previous technical report: MRCR v2 , PutnamBench
- 23 Apr 2026
- GDPval 82.3
- BrowseComp 90.1
- FrontierMath 52.4
- FrontierMath Tier 4 39.6
- Investment Banking Modeling Tasks (OpenAI) 88.6
- GeneBench 33.2
- Humanity's Last Exam 57.2
Dropped since the last post: Terminal-Bench 2.0 , OSWorld-Verified , Toolathlon , Artificial Analysis Intelligence Index v4.0 , tau2-bench , BixBench
Its system card adds 8 evaluations 3 new 26 dropped 8 results in all
GPT-5.5 System CardBio and chem risk
Cyber risk
- CVE-Bench 93.1
Dropped since the lab’s previous system card: OpenAI Production Benchmarks and safe-completion evals , OpenAI Dynamic Mental Health Benchmarks , Disallowed Content Evaluations with Representative Prompts , Deception Evaluations with Representative Prompts , Multiturn Jailbreak Evaluations , Agent JSK , PlugInject , Image input safety evaluations , HealthBench , HealthBench Hard , HealthBench Consensus , Destructive action avoidance , Destructive action reversion on long rollouts , User Confirmations During Computer Use , First-Person Fairness evaluation , CoT Monitorability , CoT-Control , Capture the Flag (Professional) , Cyber Range , Atomic Challenges , CyScenarioBench , Monorepo-Bench , MLE-bench , OpenAI-Proof Q&A , Apollo Research strategic deception evaluations , OpenAI cyber safety evaluations (incl. Daybreak Blue)
- 23 Apr 2026
- τ-Voice 67.3
Dropped since the last post: Research-Eval , FRAMES , X Browse (xAI) , tau2-bench , BFCL V4
- 23 Apr 2026
- FrontierScience 70
- IMO-AnswerBench 84.3
- Humanity's Last Exam 30
- GPQA Diamond 87.2
- AdvancedIF 79.5
- AA-LCR (Artificial Analysis Long Context Reasoning) 66.3
- LongBench v2 65.4
- CL-bench 22.8
- CL-bench Life 15.7
- SWE-bench Verified 74.4
- Terminal-Bench 2.0 54.4
- BrowseComp 67.1
- WideSearch 70.2
- WildClawBench 45.3
- Claw-Eval Pass^3 55
- Hy-Backend 54.7
- Hy-Vibe Bench 50.5
- Hy-SWE Max 30
Dropped since the last post: HMMT 2025 , ARC-AGI , MultiChallenge , Terminal-Bench , tau2-bench
- 23 Apr 2026
- Terminal-Bench 2.0 82.7
- GDPval 84.9
- OSWorld-Verified 78.7
- Toolathlon 55.6
- BrowseComp 90.1
- FrontierMath 52.4
- FrontierMath Tier 4 39.6
- Artificial Analysis Intelligence Index v4.0 60.2
- tau2-bench 98
- GeneBench 33.2
- BixBench 80.5
Dropped since the last post: Vals Finance Agent , Investment Banking Modeling Tasks (OpenAI) , FrontierScience-Research , GPQA Diamond , Humanity's Last Exam , ARC-AGI , ARC-AGI-2
Its system card adds 40 evaluations 32 new 40 results in all
GPT-5.5 System CardHealth
Hallucination
Safety
- OpenAI Production Benchmarks and safe-completion evals
- Disallowed Content Evaluations with Representative Prompts
- Image input safety evaluations
- Destructive action avoidance 90
- Destructive action reversion on long rollouts
- User Confirmations During Computer Use
- OpenAI cyber safety evaluations (incl. Daybreak Blue)
Robustness
Bias
Alignment
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- Monorepo-Bench 60
- MLE-bench 36.7
- Internal Research Debugging Evaluation 50.5
- OpenAI-Proof Q&A 1.7
- 22 Apr 2026
- SWE-bench Verified 77.2
- SWE-Bench Pro 53.5
- SWE-bench Multilingual 71.3
- Terminal-Bench 2.0 59.3
- SkillsBench 48.2
- QwenWebBench 1487
- NL2Repo-Bench 36.2
- Claw-Eval 72.4
- Claw-Eval Pass^3 60.6
- QwenClawBench 53.4
- MMLU-Pro 86.2
- MMLU-Redux 93.5
- SuperGPQA 66
- C-Eval 91.4
- GPQA Diamond 87.8
- Humanity's Last Exam 24
- LiveCodeBench v6 83.9
- HMMT 2025 93.8
- HMMT Nov 2025 90.7
- HMMT 2026 84.3
- IMO-AnswerBench 80.8
- AIME 2026 94.1
- MMMU 82.9
- MMMU-Pro 75.8
- MathVista 87.4
- DynaMath 85.6
- VLMs Are Blind 97
- RealWorldQA 84.1
- MMStar 81.4
- MMBench-EN v1.1 92.3
- SimpleVQA 56.1
- CharXiv 78.4
- CC-OCR 81.2
- OCRBench 89.4
- ERQA 62.5
- CountBench 97.8
- RefCOCO / RefCOCO+ / RefCOCOg 92.5
- EmbSpatialBench 84.6
- RefSpatial-Bench 70
- Video-MME 87.7
- Video-MMMU 84.4
- MLVU 86.6
- MVBench 75.5
- V* 94.7
- AndroidWorld 70.3
Dropped since the last post: tau3-bench , VITA-Bench , DeepPlanning , Toolathlon , MCPMark , MCP Atlas , WideSearch , ZeroBench , HallusionBench , OmniDocBench v1.5 , AI2D , ODinW-13 , LVBench
- 22 Apr 2026
- CharXiv 81
- MMMU-Pro 77.9
- HR-Bench 88.5
- OmniDocBench 87.2
- Claw-Eval-MM 23.8
- Video-MME 87.7
- Daily-Omni 83.5
- Video-Holmes 64
- MiMo Coding Bench 71.8
- Claw-Eval Pass^3 62.3
- Terminal-Bench 2.0 65.8
- SWE-Bench Pro 56.1
Dropped since the last post: MMAU-Pro , Big Bench Audio , FutureOmni , MM-BrowseComp , OmniGAIA , Claw-Eval , SWE-bench Verified , GDPval-AA , PinchBench
- 21 Apr 2026
- LBM Eval 34.1
- 21 Apr 2026
- LBM Eval 72.4
- 20 Apr 2026
- SuperGPQA 73.9
- AA-Omniscience 10
- GDPval-AA 51
- QwenChineseBench 84
- QwenClawBench 59
- SkillsBench 55.6
- ToolcallFormatIFBench 86.1
- QwenWebBench 1532
- SciCode 47
- NL2Repo-Bench 42.9
- Terminal-Bench 2.0 65.4
- SWE-Bench Pro 57.3
Dropped since the last post: SWE-bench Verified , SWE-bench Multilingual , Claw-Eval , Claw-Eval Pass^3 , tau3-bench , VITA-Bench , DeepPlanning , Toolathlon , MCPMark , MCP Atlas , Humanity's Last Exam , WideSearch , MMLU-Pro , MMLU-Redux , C-Eval , IFEval , IFBench , AA-LCR (Artificial Analysis Long Context Reasoning) , LongBench v2 , GPQA Diamond , LiveCodeBench v6 , HMMT 2025 , HMMT Nov 2025 , HMMT 2026 , IMO-AnswerBench , AIME 2026 , MMMLU , MMLU-ProX , NOVA-63 , INCLUDE , Global PIQA , PolyMath , WMT24++ , MAXIFE , MMMU , MMMU-Pro , MATH-Vision (MathVision) , We-Math , DynaMath , RealWorldQA , MMStar , SimpleVQA , OmniDocBench v1.5 , CharXiv , MMLongBench-Doc , CC-OCR , AI2D , CountBench , ODinW-13 , ERQA , V* , Video-MME , Video-MMMU , MLVU , ScreenSpot-Pro , TIR-Bench , OSWorld-Verified
- 20 Apr 2026
- Humanity's Last Exam 54
- BrowseComp 83.2
- DeepSearchQA 92.5
- WideSearch 80.8
- Toolathlon 50
- MCPMark 55.9
- Claw-Eval 62.3
- APEX-Agents 27.9
- OSWorld-Verified 73.1
- Terminal-Bench 2.0 66.7
- SWE-Bench Pro 58.6
- SWE-bench Multilingual 76.7
- SWE-bench Verified 80.2
- SciCode 52.2
- OJBench 60.6
- LiveCodeBench v6 89.6
- AIME 2026 96.4
- HMMT 2026 92.7
- IMO-AnswerBench 86
- GPQA Diamond 90.5
- MMMU-Pro 79.4
- CharXiv 80.4
- MATH-Vision (MathVision) 87.4
- BabyVision 39.8
- V* 96.9
Dropped since the last post: AIME 2025 , HMMT 2025 , MMLU-Pro , MathVista , ZeroBench , OCRBench , OmniDocBench v1.5 , InfographicVQA , SimpleVQA , WorldVQA , Video-MMMU , MMVU , MotionBench , Video-MME , LongVideoBench , LVBench , PaperBench , CyberGym , LongBench v2 , FinSearchComp , Seal-0
- 16 Apr 2026Claude Opus 4.7 Anthropic Introducing Claude Opus 4.7 (launch post); System Card: Claude Opus 4.7, Table 8.1.A
- SWE-Bench Pro 64.3
- SWE-bench Verified 87.6
- SWE-bench Multilingual 80.5
- SWE-bench Multimodal 34.5
- Terminal-Bench 2.0 69.4
- Humanity's Last Exam 54.7
- BrowseComp 79.3
- MCP Atlas 77.3
- OSWorld-Verified 78
- Vals Finance Agent 64.4
- GPQA Diamond 94.2
- CharXiv 91
- MMMLU 91.5
- ScreenSpot-Pro 87.6
- OfficeQA 86.3
- OfficeQA 80.6
- ARC-AGI 92
- ARC-AGI-2 75.8
Dropped since the last post: USAMO 2026 , GraphWalks
Its system card adds 74 evaluations 20 new 8 dropped 91 results in all
System Card: Claude Opus 4.7Capabilities
- OfficeQA Pro 80.6
- USAMO 2026 69.3
- GraphWalks
- OpenAI MRCR v2
- DeepSearchQA 89.1
- DRACO 77.7
- LAB-Bench FigQA 78.6
- Vending-Bench 2 10937
- GDPval-AA 1753
- Global MMLU 89.9
- MILU 89.9
- INCLUDE 87
- BioPipelineBench Verified (Anthropic) 83.6
- BioMysteryBench
- Structural Biology (Anthropic)
- Organic Chemistry (Anthropic) 77.2
- Phylogenetics (Anthropic) 79.6
- Protocols (Anthropic life sciences) 51.8
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 98
- Single-turn benign request evaluations (Anthropic) 0.28
- Higher-difficulty harmful request evaluations (Anthropic) 99
- Higher-difficulty benign request evaluations (Anthropic) 0.01
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 91.2
- Malicious computer use (Anthropic) 89.3
Robustness
Alignment
- Impossible tasks (Anthropic) 45
- Agentic code behavior scores (Anthropic)
- Over-eager behavior on GUI computer use tasks (Anthropic) 21.1
- Anthropic automated behavioral audit
- Petri
- UK AISI unprompted sabotage evaluation 0
- UK AISI sabotage continuation evaluation 0
- Destructive production evaluation (Anthropic) 2.5
- Destructive actions (Anthropic)
- Adherence to its constitution (Anthropic) 0.74
- MASK 91.7
- Refusal to assist with AI safety R&D (Anthropic) 33.1
- Self-preference bias (Anthropic) 0.4
- DTBench
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: ASL-4 virology uplift trial (Anthropic) , Catastrophic biology scenario uplift trial (Anthropic) , Internal AI research evaluation suite 2 , Internal model use survey (Anthropic) , Key insight rediscovery on an unpublished ML task (external testing) , UK AISI alignment and monitorability evaluations , White-box analyses of model internals (Anthropic) , Recognition of model-written user turns (Anthropic)
- 15 Apr 2026
- SWE-bench Verified 73.4
- SWE-bench Multilingual 67.2
- SWE-Bench Pro 49.5
- Terminal-Bench 2.0 51.5
- Claw-Eval 68.7
- Claw-Eval Pass^3 50
- SkillsBench 28.7
- QwenClawBench 52.6
- NL2Repo-Bench 29.4
- QwenWebBench 1397
- tau3-bench 67.2
- VITA-Bench 35.6
- DeepPlanning 25.9
- Toolathlon 26.9
- MCPMark 37
- MCP Atlas 62.8
- WideSearch 60.1
- MMLU-Pro 85.2
- MMLU-Redux 93.3
- SuperGPQA 64.7
- C-Eval 90
- GPQA Diamond 86
- Humanity's Last Exam 21.4
- LiveCodeBench v6 80.4
- HMMT 2025 90.7
- HMMT Nov 2025 89.1
- HMMT 2026 83.6
- IMO-AnswerBench 78.9
- AIME 2026 92.7
- MMMU 81.7
- MMMU-Pro 75.3
- MathVista 86.4
- ZeroBench 34.4
- RealWorldQA 85.3
- MMBench-EN v1.1 92.8
- SimpleVQA 58.9
- HallusionBench 69.8
- OmniDocBench v1.5 89.9
- CharXiv 78
- CC-OCR 81.9
- AI2D 92.7
- RefCOCO / RefCOCO+ / RefCOCOg 92
- ODinW-13 50.8
- EmbSpatialBench 84.3
- RefSpatial-Bench 64.3
- Video-MME 86.6
- Video-MMMU 83.7
- MLVU 86.2
- MVBench 74.6
- LVBench 71.4
Dropped since the last post: IFEval , IFBench , MultiChallenge , AA-LCR (Artificial Analysis Long Context Reasoning) , LongBench v2 , Codeforces rating , OJBench , FullStackBench , BFCL V4 , tau2-bench , BrowseComp , BrowseComp-ZH , Seal-0 , MMMLU , MMLU-ProX , NOVA-63 , INCLUDE , Global PIQA , PolyMath , WMT24++ , MAXIFE , MATH-Vision (MathVision) , DynaMath , VLMs Are Blind , BabyVision , MMStar , MMLongBench-Doc , OCRBench , ERQA , CountBench , LingoQA , Hypersim , SUNRGBD , Nuscene , MMVU , ScreenSpot-Pro , OSWorld-Verified , AndroidWorld , TIR-Bench , V* , SLAKE , PMC-VQA , MedXpertQA-MM
- 14 Apr 2026 Gemini Robotics-ER 1.6 Google DeepMind Gemini Robotics ER 1.6: Enhanced Embodied Reasoning (Google DeepMind blog)
Dropped since the last post: Humanity's Last Exam , ARC-AGI-2 , GPQA Diamond , Terminal-Bench 2.0 , SWE-bench Verified , SWE-Bench Pro , LiveCodeBench Pro , SciCode , APEX-Agents , GDPval-AA , tau2-bench , MCP Atlas , BrowseComp , MMMU-Pro , MMMLU , MRCR v2
- 9 Apr 2026
- MMMU 78.7
- MMMU-Pro 68.6
- MedXpertQA-MM 42.1
- MATH-Vision (MathVision) 75.2
- MathVista 85
- We-Math 79.1
- LogicVista 73.8
- BabyVision 18.8
- AI2D 89
- ChartQA Pro 62.2
- CharXiv 71.7
- OCRBench v2 63.2
- OmniDocBench v1.5 81.2
- MMStar 74.9
- BLINK 68.8
- HallusionBench 63.7
- KMMMU 42.7
- K-Viscuit 80.1
- KRETA 91.9
- AIME 2025 92.9
- AIME 2026 92.6
- GPQA Diamond 80.5
- LiveCodeBench v6 81.4
- MMLU-Pro 83.3
- tau2-bench 69.1
- IFBench 62.6
- IFEval 89.6
- AA-LCR (Artificial Analysis Long Context Reasoning) 50.6
- KMMLU-Pro 67.6
- KoBALT 52.1
Dropped since the last post: Humanity's Last Exam , IMO-AnswerBench , HMMT Nov 2025 , LiveCodeBench Pro , Terminal-Bench 2.0 , SWE-bench Verified , BrowseComp , OpenAI MRCR v2 , CLIcK , HRM8K , Ko-LongBench (LG) , MMMLU , WMT24++ , WildJailbreak , KGC-Safety (LG)
Its technical report adds 2 evaluations 15 dropped 31 results in all
EXAONE 4.5 Technical ReportDropped since the lab’s previous technical report: Humanity's Last Exam , IMO-AnswerBench , AIME 2025 , HMMT Nov 2025 , LiveCodeBench Pro , CodeUtilityBench (LG) , Terminal-Bench 2.0 , SWE-bench Verified , BrowseComp , OpenAI MRCR , CLIcK , HRM8K , Ko-LongBench (LG) , WildJailbreak , KGC-Safety (LG)
- 8 Apr 2026
- CharXiv 86.4
- MMMU-Pro 80.4
- ERQA 64.7
- SimpleVQA 71.3
- ScreenSpot-Pro 84.1
- ZeroBench 33
- Humanity's Last Exam 50.4
- ARC-AGI-2 42.5
- GPQA Diamond 89.5
- LiveCodeBench Pro 80
- HealthBench Hard 42.8
- MedXpertQA 52.6
- MedXpertQA-MM 78.4
- DeepSearchQA 74.8
- SWE-bench Verified 77.4
- SWE-Bench Pro 52.4
- Terminal-Bench 2.0 59
- tau2-bench 91.5
- GDPval-AA 1444
Dropped since the last post: LiveCodeBench , MATH-500 , MMLU-Pro , MMMLU , MMMU
Its system card adds 55 evaluations 56 results in all
Muse Spark Safety & Preparedness ReportHallucination
- HLE Calibration 50.3
- SimpleQA Verified 70.8
- AbstentionBench 59.1
- CharXiv Missing Images 35
Safety
- TextQuests 2.4
- CIMemories 78.3
Robustness
Alignment
- Meta instructed selective performance evaluation 31.8
- Meta prompted evaluation awareness 43
- IHEval 80.3
- ImpossibleBench 0
- Meta Underspecified Intent evaluation 71.2
- MASK 89.1
- DeceptionBench 1.6
- PropensityBench 24.5
- Meta internal sycophancy evaluation 62.9
- Alignment faking evaluation 1.8
- Agentic misalignment suite (Anthropic) 47.7
- Situational Awareness Dataset (SAD)
Monitorability
- SHADE-Arena 8.8
- GDM Stealth 39.8
- GDM Situational Awareness 29.3
Bio and chem risk
- Molecular Biology Capabilities Test 54.4
- Virology Capabilities Test (VCT) 49.7
- Human Pathogen Capabilities Test 55.7
- WMDP
- LAB-Bench ProtocolQA 87.3
- LAB-Bench SeqQA 97.3
- ABC Bench Fragment Design 96.8
- ABC Bench Liquid-Handling Robot 92.3
- Meta BioDesign Proxy 46.2
- Meta chemical and biological high-risk bottleneck evaluations
- BioTIER 98
- Meta chemical-agent refusal evaluation 99.4
Cyber risk
AI R&D risk
- SWE-bench Verified hard subset (Meta) 42.9
- MLE-bench 15.8
- AIRS-Bench 0.65
- 7 Apr 2026
- 7 Apr 2026
- Humanity's Last Exam 52.3
- AIME 2026 95.3
- HMMT Nov 2025 94
- HMMT 2026 82.6
- IMO-AnswerBench 83.8
- GPQA Diamond 86.2
- SWE-Bench Pro 58.4
- NL2Repo-Bench 42.7
- Terminal-Bench 2.0 69
- CyberGym 68.7
- BrowseComp 79.3
- tau3-bench 70.6
- MCP Atlas 71.8
- Toolathlon 40.7
- Vending-Bench 2 5634.4
Dropped since the last post: SWE-bench Verified , SWE-bench Multilingual , BrowseComp-ZH , tau2-bench
- 7 Apr 2026Claude Mythos Preview Anthropic System Card: Claude Mythos Preview, Table 6.3.A; Project Glasswing announcement
- SWE-bench Verified 93.9
- SWE-Bench Pro 77.8
- SWE-bench Multilingual 87.3
- SWE-bench Multimodal 59
- Terminal-Bench 2.0 82
- GPQA Diamond 94.5
- MMMLU 92.7
- USAMO 2026 97.6
- GraphWalks 80
- Humanity's Last Exam 64.7
- CharXiv 93.2
- OSWorld-Verified 79.6
- BrowseComp 86.9
Dropped since the last post: tau2-bench , MCP Atlas , Vals Finance Agent , GDPval-AA , ARC-AGI-2 , MMMU-Pro
Its system card adds 61 evaluations 27 new 36 dropped 74 results in all
System Card: Claude Mythos PreviewCapabilities
- LAB-Bench FigQA 79.7
- ScreenSpot-Pro 79.5
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 97.8
- Single-turn benign request evaluations (Anthropic) 0.06
- Higher-difficulty harmful request evaluations (Anthropic) 99.1
- Higher-difficulty benign request evaluations (Anthropic) 0.02
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 96.7
- Malicious computer use (Anthropic) 93.8
- Malicious agentic influence campaigns (Anthropic)
Robustness
Alignment
- Impossible tasks (Anthropic) 37.5
- Agentic code behavior scores (Anthropic)
- Over-eager behavior on GUI computer use tasks (Anthropic) 13.3
- Anthropic automated behavioral audit
- Petri
- UK AISI alignment and monitorability evaluations
- Destructive production evaluation (Anthropic) 0.3
- Destructive actions (Anthropic)
- Adherence to its constitution (Anthropic) 1.02
- MASK 95.4
- Refusal to assist with AI safety R&D (Anthropic) 18.7
- Self-preference bias (Anthropic)
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- Kernels task (Anthropic) 399.4
- Time series forecasting task (Anthropic) 4.55
- LLM training task (Anthropic) 51.9
- Quadruped reinforcement learning task (Anthropic) 30.9
- Novel compiler task (Anthropic) 77.2
- Internal AI research evaluation suite 2 0.65
- Internal model use survey (Anthropic) 5.6
- Anthropic ECI (AECI)
- Key insight rediscovery on an unpublished ML task (external testing) 80
Dropped since the lab’s previous system card: tau2-bench , MCP Atlas , ARC-AGI-2 , GDPval-AA , MMMU-Pro , OpenRCA , ARC-AGI , AIME 2025 , Vals Finance Agent , Real-World Finance (Anthropic) , Vending-Bench 2 , OpenAI MRCR v2 , WebArena , WebArena-Verified , Global MMLU , MILU , DeepSearchQA , BioPipelineBench (Anthropic) , BioMysteryBench , Structural Biology (Anthropic) , Organic Chemistry (Anthropic) , Phylogenetics (Anthropic) , MedCalc-Bench Verified , SSH stress-testing evaluation (Anthropic) , Reward-hack-prone coding tasks (Anthropic) , Participation in junk science (Anthropic) , Agentic coding malicious requests (Anthropic) , Creative biology , Short-horizon computational biology tasks , SWE-bench Verified (hard subset) , Text-based reinforcement learning task (Anthropic) , Web CTF challenges (Anthropic) , Crypto CTF challenges (Anthropic) , Pwn CTF challenges (Anthropic) , Rev CTF challenges (Anthropic) , Network CTF challenges (Anthropic)
- 2 Apr 2026
- MMLU-Pro 82.6
- AIME 2026 88.3
- LiveCodeBench v6 77.1
- Codeforces rating 1718
- GPQA Diamond 82.3
- tau2-bench 68.2
- Humanity's Last Exam 17.2
- BIG-Bench Extra Hard 64.8
- MMMLU 86.3
- MMMU-Pro 73.8
- OmniDocBench v1.5 0.149
- MATH-Vision (MathVision) 82.4
- MedXpertQA-MM 58.1
- MRCR v2 44.1
Dropped since the last post: CoVoST2 , FLEURS
Its technical report adds 10 evaluations 7 new 48 dropped 24 results in all
Gemma 4 Technical Report (arXiv 2607.02770v2)Capabilities
- LMArena (Elo) 1438
- SciCode 40
- IFBench 72
- IFEval 98.5
- Terminal-Bench Hard 14
- InfographicVQA 89.3
- RULER 89.8
- LOFT 66.3
- GraphWalks 72.6
- MTOB
Dropped since the lab’s previous technical report: LiveCodeBench , Bird-SQL , SimpleQA , FACTS Grounding , Global MMLU Lite , MATH , HiddenMath , MMMU , MMLU , MBPP , HumanEval , Natural2Code , GSM8K , BIG-Bench Hard , ECLeKTic , WMT24++ , MRCR (Michelangelo, original) , DocVQA , TextVQA , AI2D , ChartQA , VQAv2 , MathVista , Perception Test MCVQA , ActivityNet-QA , HellaSwag , BoolQ , PIQA , SIQA , TriviaQA , Natural Questions , ARC-Challenge , ARC-Easy , WinoGrande , DROP , AGIEval , COCO Captions , RealWorldQA , ReMI , BLINK , OK-VQA , TallyQA , SpatialSense VQA , CountBench , MGSM , Flores , XQuAD , IndicGenBench
- 2 Apr 2026
- SWE-bench Verified 78.8
- SWE-bench Multilingual 73.8
- SWE-Bench Pro 56.6
- Terminal-Bench 2.0 61.6
- Claw-Eval 74.8
- Claw-Eval Pass^3 58.7
- SkillsBench 45.7
- QwenClawBench 57.2
- NL2Repo-Bench 37.9
- QwenWebBench 1501.7
- tau3-bench 70.7
- VITA-Bench 44.3
- DeepPlanning 41.5
- Toolathlon 39.8
- MCPMark 48.2
- MCP Atlas 74.1
- Humanity's Last Exam 50.6
- WideSearch 74.3
- MMLU-Pro 88.5
- MMLU-Redux 94.5
- SuperGPQA 71.6
- C-Eval 93.3
- IFEval 94.3
- IFBench 74.2
- AA-LCR (Artificial Analysis Long Context Reasoning) 68.3
- LongBench v2 62
- GPQA Diamond 90.4
- LiveCodeBench v6 87.1
- HMMT 2025 96.7
- HMMT Nov 2025 94.6
- HMMT 2026 87.8
- IMO-AnswerBench 83.8
- AIME 2026 95.3
- MMMLU 89.5
- MMLU-ProX 84.7
- NOVA-63 57.9
- INCLUDE 85.1
- Global PIQA 89.8
- PolyMath 77.4
- WMT24++ 84.3
- MAXIFE 88.2
- MMMU 86
- MMMU-Pro 78.8
- MATH-Vision (MathVision) 88
- We-Math 89
- DynaMath 88
- RealWorldQA 85.4
- MMStar 83.3
- SimpleVQA 67.3
- OmniDocBench v1.5 91.2
- CharXiv 81.5
- MMLongBench-Doc 62
- CC-OCR 83.4
- AI2D 94.4
- CountBench 97.6
- ODinW-13 51.8
- ERQA 65.7
- V* 96.9
- Video-MME 87.8
- Video-MMMU 84
- MLVU 86.7
- ScreenSpot-Pro 68.2
- TIR-Bench 61.6
- OSWorld-Verified 62.5
Dropped since the last post: Daily-Omni , WorldSense (omnimodal) , AVUT , AV-SpeakerBench , QIVD , Omni-Cloze , OmniGAIA , MMAU , MMAR , MMSU , MuChoMusic , VoiceBench , URO-Bench , SpeechRole , WildSpeech-Bench , FLEURS , Common Voice , LibriSpeech , Seed-TTS-eval , MathVista , ZeroBench , MMBench-EN v1.1 , OCRBench , RefCOCO / RefCOCO+ / RefCOCOg , EmbSpatialBench , MVBench , LVBench , MMVU , MME-VideoOCR , SLAKE , PMC-VQA , MedXpertQA-MM , BFCL V4 , tau2-bench , WenetSpeech
- 2 Apr 2026
- MMLU-Pro 85.2
- AIME 2026 89.2
- LiveCodeBench v6 80
- Codeforces rating 2150
- GPQA Diamond 84.3
- tau2-bench 76.9
- Humanity's Last Exam 26.5
- BIG-Bench Extra Hard 74.4
- MMMLU 88.4
- MMMU-Pro 76.9
- OmniDocBench v1.5 0.131
- MATH-Vision (MathVision) 85.6
- MedXpertQA-MM 61.3
- CoVoST2
- FLEURS
- MRCR v2 66.4
Dropped since the last post: CharXiv , Video-MMMU , SimpleQA Verified , FACTS Benchmark Suite , LiveCodeBench
Its technical report adds 10 evaluations 24 results in all
Gemma 4 Technical Report (arXiv 2607.02770v2)Capabilities
- LMArena (Elo) 1451
- SciCode 43
- IFBench 76
- IFEval 98.9
- Terminal-Bench Hard 36
- InfographicVQA 92
- RULER 96.4
- LOFT 79.5
- GraphWalks 82.3
- MTOB
- 1 Apr 2026
- IFBench 52.3
- GPQA Diamond 76.3
- tau2-bench 91.3
- PinchBench 91.9
- AIME 2025 96.3
- BFCL V4 70.1
- MMLU-Pro 83.4
- SWE-bench Verified 63.2
Dropped since the last post: MMLU
- 30 Mar 2026
- Daily-Omni 84.6
- WorldSense (omnimodal) 62.8
- AVUT 85
- AV-SpeakerBench 71.3
- Video-MME 83.7
- QIVD 68.5
- Omni-Cloze 64.8
- OmniGAIA 57.2
- MMAU 82.2
- MMAR 80
- MMSU 82.8
- MuChoMusic 72.4
- VoiceBench 93.1
- URO-Bench 84.1
- SpeechRole 123.5
- WildSpeech-Bench 75.4
- FLEURS 6.55
- Common Voice 3.13
- LibriSpeech 1.67
- Seed-TTS-eval 1.35
- MMMU 80.1
- MMMU-Pro 73.9
- MATH-Vision (MathVision) 73
- MathVista 86.1
- DynaMath 83.8
- ZeroBench 34.4
- RealWorldQA 84.1
- MMStar 79.4
- MMBench-EN v1.1 92.8
- SimpleVQA 65.3
- CharXiv 72.5
- CC-OCR 83.4
- AI2D 91.2
- MMLongBench-Doc 57.5
- OCRBench 91.3
- ERQA 54.8
- CountBench 95.1
- RefCOCO / RefCOCO+ / RefCOCOg 95
- ODinW-13 49.5
- EmbSpatialBench 85.4
- MLVU 86.8
- MVBench 79
- LVBench 71.2
- MMVU 67.5
- MME-VideoOCR 77
- SLAKE 84.7
- PMC-VQA 62.7
- MedXpertQA-MM 54.7
- MMLU-Pro 85.9
- MMLU-Redux 94.2
- SuperGPQA 66.4
- C-Eval 92
- IFEval 89.7
- IFBench 52.6
- AA-LCR (Artificial Analysis Long Context Reasoning) 57
- LongBench v2 59.6
- GPQA Diamond 83.9
- LiveCodeBench v6 65.6
- HMMT Nov 2025 84.4
- IMO-AnswerBench 65.5
- BFCL V4 63.3
- tau2-bench 81
- WenetSpeech 5.07
Dropped since the last post: MultiChallenge , Humanity's Last Exam , HMMT 2025 , AIME 2026 , VITA-Bench , DeepPlanning , Toolathlon , MCPMark , BrowseComp , BrowseComp-ZH , WideSearch , Seal-0 , MMMLU , MMLU-ProX , NOVA-63 , INCLUDE , Global PIQA , PolyMath , WMT24++ , MAXIFE , SWE-bench Verified , SWE-bench Multilingual , SecCodeBench , Terminal-Bench 2.0 , We-Math , BabyVision , HallusionBench , OmniDocBench v1.5 , RefSpatial-Bench , LingoQA , V* , Hypersim , SUNRGBD , Nuscene , Video-MMMU , ScreenSpot-Pro , OSWorld-Verified , AndroidWorld
Its technical report adds 2 evaluations 2 new 50 dropped 65 results in all
Qwen3.5-Omni Technical ReportCapabilities
Dropped since the lab’s previous technical report: MathVision-Wildphoto , We-Math , MathVerse , Math-VR , VLMs Are Blind , LogicVista , VisuLogic , VisualPuzzles , MMBench-CN v1.1 , HallusionBench , MM-MT-Bench , MIA-Bench , DocVQA , InfographicVQA , ChartQA , OCRBench v2 , OmniDocBench , CharXiv (DQ) , ARKitScenes , Hypersim , SUNRGBD , VSI-Bench , RefSpatial-Bench , RoboSpatial-Home , BLINK , MuirBench , Charades-STA , Video-MMMU , V* , HR-Bench , Design2Code , ChartMimic , ScreenSpot-Pro , OSWorld-G , AndroidWorld , OSWorld , Windows Agent Arena , AIME 2025 , HMMT 2025 , LiveBench , Codeforces rating , OJBench , Arena-Hard v2.0 , Creative Writing v3 , WritingBench , Berkeley Function Calling Leaderboard (BFCL) , Multi-IF , MMLU-ProX , INCLUDE , PolyMath
- 26 Mar 2026 Cohere Transcribe Cohere Introducing Cohere Transcribe: a new state-of-the-art in open-source speech recognition (Cohere blog)
- Open ASR Leaderboard 5.42
- LibriSpeech 1.25
Dropped since the last post: AyaVisionBench , m-WildVision
- 24 Mar 2026
- MMLU-Pro 81.2
- GPQA Diamond 71.9
- Humanity's Last Exam 11.9
- HMMT 2026 65.9
- AIME 2026 83
- LiveCodeBench v6 77.7
- tau2-bench 72.3
- Terminal-Bench 2.0 10.1
- IFBench 73.1
- Arena-Hard v2.0 74.7
- Ko-Arena-Hard v2 (Upstage) 78.2
Dropped since the last post: AIME 2025
- 24 Mar 2026
- WebVoyager 78.2
- Online-Mind2Web 35.3
- DeepShop 42.3
- WebTailBench 49.5
Dropped since the last post: SWE-bench Verified
Its technical report adds 0 evaluations 1 dropped 4 results in all
MolmoWeb: Open Visual Web Agent and Open Data for the Open WebDropped since the lab’s previous technical report: SWE-bench Verified
- 19 Mar 2026
- IMO 2025 35
- IMO-AnswerBench 79.3
- IMO-ProofBench 72.9
- AIME 2025 92.4
- AIME 2026 90.9
- HMMT 2025 94.6
- IOI 2025 439.3
- ICPC World Finals 2025 10
- LiveCodeBench v6 87.2
- LiveCodeBench Pro 27.6
- SciCode 36.4
- MMLU-Redux 86.3
- MMLU-Pro 79.8
- GPQA Diamond 76.1
- Humanity's Last Exam 17.7
- Arena-Hard v2.0 83.5
- IFBench 82.9
- MultiChallenge 45.3
- AA-LCR (Artificial Analysis Long Context Reasoning) 39.1
- LongBench v2 40.3
- RULER 99
- CL-bench 12.2
- BFCL V4 52.9
- tau2-bench 58.9
- Terminal-Bench 2.0 21.1
- SWE-bench Verified 50.2
- MMLU-ProX 72.5
- WMT24++ 84.1
Dropped since the last post: MATH-500 , LiveCodeBench , Berkeley Function Calling Leaderboard (BFCL) , IFEval , Orak , HaluEval , EQ-Bench 3
Its technical report adds 1 evaluation 1 new 2 dropped 29 results in all
Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy DistillationCapabilities
- Codeforces rating 2320
Dropped since the lab’s previous technical report: miniF2F , Terminal-Bench Hard
- 19 Mar 2026
- CursorBench 3.1 61.3
- Terminal-Bench 2.0 61.7
- SWE-bench Multilingual 73.7
Dropped since the last post: Cursor Bench (2025 internal version)
Its technical report adds 1 evaluation 3 results in all
Composer 2 Technical ReportCapabilities
- CursorBench 61.3
- 18 Mar 2026
- MMAU-Pro 69.4
- Big Bench Audio 94
- MMMU-Pro 76.8
- CharXiv 80.1
- Video-MME 85.3
- FutureOmni 66.7
- MM-BrowseComp 52
- OmniGAIA 49.8
- Claw-Eval 54.8
- SWE-bench Verified 74.8
- GDPval-AA 1410
- PinchBench 81.2
Dropped since the last post: SWE-bench Multilingual , Terminal-Bench 2.0 , tau2-bench , DeepSearchQA
- 18 Mar 2026
- PinchBench 81
- Claw-Eval 61.5
- GDPval-AA 1426
- SWE-bench Verified 78
- SWE-bench Multilingual 71.7
- Terminal-Bench 2.0 57.1
- tau2-bench 96.8
- DeepSearchQA 86.7
Dropped since the last post: MMLU-Pro , GPQA Diamond , Humanity's Last Exam , AIME 2025 , HMMT 2025 , LiveCodeBench v6 , Arena-Hard v2.0 , LongBench v2 , MRCR v2 , Terminal-Bench Hard , BrowseComp
- 18 Mar 2026 MiniMax-M2.7 MiniMax MiniMax M2.7 launch post (minimax.io/news/minimax-m27-en) and Hugging Face card overview chart
- SWE-Bench Pro 56.2
- Multi-SWE-bench 52.7
- SWE-bench Multilingual 76.5
- VIBE-Pro (MiniMax) 55.6
- Terminal-Bench 2.0 57
- NL2Repo-Bench 39.8
- MLE-bench 66.6
- GDPval-AA 1495
- Toolathlon 46.3
- MM-ClawBench 62.7
- Artificial Analysis Intelligence Index v4.0 50
Dropped since the last post: SWE-bench Verified , BrowseComp , WideSearch , RISE , Berkeley Function Calling Leaderboard (BFCL) , tau2-bench , GDPval-MM , MEWC , Finance Modeling , AIME 2025 , GPQA Diamond , Humanity's Last Exam , SciCode , IFBench , AA-LCR (Artificial Analysis Long Context Reasoning)
Its technical report adds 14 evaluations 25 results in all
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World IntelligenceCapabilities
- HyperTask 67.6
- BrowseComp 77.8
- WideSearch 75.2
- RISE 64.3
- MEWC v2 63.3
- Finance Modeling Pro 57
- AIME 2026 94.2
- GPQA Diamond 89.8
- SciCode 47
- IFBench 76
- AA-LCR (Artificial Analysis Long Context Reasoning) 72
- Humanity's Last Exam 28
- MMLU-Pro 81.8
- AIME 2025 94
- 17 Mar 2026
- SWE-Bench Pro 52.4
- Terminal-Bench 2.0 46.3
- Toolathlon 35.5
- GPQA Diamond 82.8
- OSWorld-Verified 39
- MCP Atlas 56.1
- tau2-bench 92.5
- Humanity's Last Exam 37.7
- MMMU-Pro 69.5
- OmniDocBench v1.5 0.242
- OpenAI MRCR v2 44.2
- GraphWalks 73.4
- 17 Mar 2026
- SWE-Bench Pro 54.4
- Terminal-Bench 2.0 60
- Toolathlon 42.9
- GPQA Diamond 88
- OSWorld-Verified 72.1
Its system card adds 11 evaluations 10 new 8 dropped 11 results in all
GPT-5.4 Thinking System Card, Appendix: GPT-5.4 miniSafety
Monitorability
Bio and chem risk
Cyber risk
- Capture the Flag (Professional) 81.3
- CVE-Bench 83.3
AI R&D risk
- Monorepo-Bench 54
- OpenAI-Proof Q&A 7.5
Dropped since the lab’s previous system card: Standard Refusal Evaluation , StrongREJECT , Image input safety evaluations , LongFact-Concepts , LongFact-Objects , FActScore , SimpleQA , SimpleQA Hallucination Rate
- 16 Mar 2026
- FLTEval 31.9
Dropped since the last post: SWE-bench Verified , SWE-bench Multilingual , Terminal-Bench 2.0
- 16 Mar 2026
- AIME 2025 78.5
- MATH-500 95.4
- GPQA Diamond 53.2
- LiveCodeBench 51.8
- Berkeley Function Calling Leaderboard (BFCL) 61.1
- IFEval 87.9
- tau2-bench 35.4
- IFBench 43.2
- Orak 22.9
- HaluEval 62.2
- RULER 91.1
- EQ-Bench 3 63.2
Dropped since the last post: MMLU-Pro , LiveCodeBench v6 , SciCode , Humanity's Last Exam , miniF2F , Terminal-Bench Hard , SWE-bench Verified , BFCL V4 , MultiChallenge , Arena-Hard v2.0 , AA-LCR (Artificial Analysis Long Context Reasoning) , MMLU-ProX , WMT24++
- 16 Mar 2026
- GPQA Diamond 71.2
- MMLU-Pro 78
- IFBench 48
- Arena-Hard-Auto 58.3
- MMMU-Pro 60
- AA-LCR (Artificial Analysis Long Context Reasoning) 71.2
- AIME 2025 83.8
- COLLIE 62.9
- LiveCodeBench 63.6
Dropped since the last post: SWE-bench Verified , SWE-bench Multilingual , Terminal-Bench 2.0
- 11 Mar 2026
- VQAv2 88.4
- MLVU 74.3
- MMVU 71.7
- RefCOCO / RefCOCO+ / RefCOCOg 89.9
- VideoHallucer 59.6
- Mobile Actions 88.4
- 11 Mar 2026
- MMLU-Pro 83.7
- AIME 2025 90.2
- HMMT 2025 93.7
- GPQA Diamond 79.2
- LiveCodeBench v5 81.2
- SciCode 42
- Humanity's Last Exam 18.3
- Terminal-Bench Hard 25.8
- Terminal-Bench 2.0 31
- SWE-bench Verified 60.5
- SWE-bench Multilingual 45.8
- tau2-bench 61.1
- BrowseComp 31.3
- Bird-SQL 41.8
- IFBench 72.6
- MultiChallenge 55.2
- Arena-Hard v2.0 73.9
- AA-LCR (Artificial Analysis Long Context Reasoning) 58.3
- RULER 91.8
- MMLU-ProX 79.4
- WMT24++ 86.7
Dropped since the last post: RoboCasa , RoboCasa GR1 Tabletop Tasks , SimplerEnv , LIBERO , BEHAVIOR Challenge 2025
Its technical report adds 1 evaluation 1 new 6 dropped 22 results in all
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic ReasoningCapabilities
- LiveCodeBench v6 78.7
Dropped since the lab’s previous technical report: AIME 2024 , MATH-500 , Berkeley Function Calling Leaderboard (BFCL) , IFEval , Arena-Hard-Auto , JudgeBench
- 6 Mar 2026
- MATH-500 97
- HumanEval 92.1
- MBPP 92.7
- LiveCodeBench v6 70
- MMLU 85.1
- MMLU-Pro 80
- Arena-Hard v2.0 49
- GPQA Diamond 66.5
- AIME 2025 80
- HMMT 2025 73.3
- HMMT Nov 2025 74.2
- Beyond AIME 58.3
- BrowseComp 35.5
- SWE-bench Verified 34
- tau2-bench 45.7
- IndiVibe 89
- 6 Mar 2026
- MATH-500 98.6
- LiveCodeBench v6 71.7
- MMLU 90.6
- MMLU-Pro 81.7
- WritingBench 80.5
- Arena-Hard v2.0 71
- IFEval 84.8
- GPQA Diamond 78.7
- AIME 2025 88.3
- HMMT 2025 85.8
- HMMT Nov 2025 85.8
- Beyond AIME 69.1
- BrowseComp 49.5
- SWE-bench Verified 45
- tau2-bench 68.3
- Humanity's Last Exam 11.2
- IndiVibe 90
Dropped since the last post: ARC-Challenge , TriviaQA , HumanEval , MBPP , GSM8K , MATH , MT-Bench , IndicGenBench , MILU
- 5 Mar 2026
- GDPval 82
- Vals Finance Agent 61.5
- Investment Banking Modeling Tasks (OpenAI) 83.6
- BrowseComp 89.3
- FrontierScience-Research 36.7
- FrontierMath 50
- FrontierMath Tier 4 38
- GPQA Diamond 94.4
- Humanity's Last Exam 58.7
- ARC-AGI 94.5
- ARC-AGI-2 83.3
Dropped since the last post: SWE-Bench Pro , OSWorld-Verified , Toolathlon , MMMU-Pro , OmniDocBench , tau2-bench
- 5 Mar 2026
- GDPval 83
- SWE-Bench Pro 57.7
- OSWorld-Verified 75
- Toolathlon 54.6
- BrowseComp 82.7
- MMMU-Pro 81.2
- OmniDocBench 0.109
- tau2-bench 98.9
Dropped since the last post: Terminal-Bench 2.0 , SWE-Lancer
Its system card adds 31 evaluations 17 new 1 dropped 31 results in all
GPT-5.4 Thinking System CardHealth
- HealthBench 62.6
- HealthBench Hard 40.1
- HealthBench Consensus 96.6
Safety
- OpenAI Production Benchmarks and safe-completion evals
- OpenAI Dynamic Mental Health Benchmarks
- Disallowed Content Evaluations with Representative Prompts
- Image input safety evaluations
- Destructive action avoidance 86
- Destructive action reversion on long rollouts
- User Confirmations During Computer Use
- OpenAI cyber safety evaluations (incl. Daybreak Blue)
Robustness
Bias
Alignment
Monitorability
Bio and chem risk
Cyber risk
- Capture the Flag (Professional) 88.2
- CVE-Bench 86.3
- Cyber Range 73.3
- Atomic Challenges
- CyScenarioBench 11
AI R&D risk
- Monorepo-Bench 59.3
- MLE-bench 23.3
- OpenAI-Proof Q&A 4.16
Dropped since the lab’s previous system card: Apollo Research sabotage evaluations
- 4 Mar 2026
- AI2D 84.8
- ChartQA 83.3
- HallusionBench 64.4
- MathVerse 44.9
- MATH-Vision (MathVision) 36.2
- MathVista 75.2
- MMMU 54.3
- MMStar 64.5
- OCRBench 76
- ScreenSpot-v2 88.2
Dropped since the last post: WebVoyager , Online-Mind2Web , DeepShop , WebTailBench
Its technical report adds 1 evaluation 1 new 7 dropped 11 results in all
Phi-4-reasoning-vision-15B Technical ReportDropped since the lab’s previous technical report: WebVoyager , Online-Mind2Web , DeepShop , WebTailBench , ScreenSpot , AgentHarm , WebTailBench-Refusals
- 3 Mar 2026
- Humanity's Last Exam 16
- GPQA Diamond 86.9
- MMMU-Pro 76.8
- CharXiv 73.2
- Video-MMMU 84.8
- SimpleQA Verified 43.3
- FACTS Benchmark Suite 40.6
- MMMLU 88.9
- LiveCodeBench 72
- MRCR v2 60.1
Dropped since the last post: AIME 2025 , LiveCodeBench v5 , SWE-bench Verified , FACTS Grounding , MMMU , Vibe-Eval , Global MMLU Lite
Its model card adds 5 evaluations 1 new 13 dropped 15 results in all
Gemini 3.1 Flash-Lite Model CardDropped since the lab’s previous model card: MMLU-Pro , LiveCodeBench v5 , Bird-SQL , SimpleQA , FACTS Grounding , Global MMLU Lite , MATH , HiddenMath , MRCR (Michelangelo, original) , MMMU , CoVoST2 , EgoSchema , Instruction Following (safety evaluation)
- 24 Feb 2026
- GPQA Diamond 74
- SciCode 38
- IFBench 71
Dropped since the last post: MMLU-Pro , Humanity's Last Exam , HumanEval , MATH-500 , AIME 2024
- 24 Feb 2026
- MMLU-Pro 86.7
- MMLU-Redux 94
- C-Eval 91.9
- SuperGPQA 67.1
- IFEval 93.4
- IFBench 76.1
- MultiChallenge 61.5
- AA-LCR (Artificial Analysis Long Context Reasoning) 66.9
- LongBench v2 60.2
- Humanity's Last Exam 47.5
- GPQA Diamond 86.6
- HMMT 2025 91.4
- HMMT Nov 2025 90.3
- SWE-bench Verified 72
- Terminal-Bench 2.0 49.4
- LiveCodeBench v6 78.9
- Codeforces rating 2100
- OJBench 39.5
- FullStackBench 60.7
- BFCL V4 72.2
- tau2-bench 79.5
- VITA-Bench 33.6
- DeepPlanning 24.1
- BrowseComp 63.8
- BrowseComp-ZH 69.9
- WideSearch 60.5
- Seal-0 44.1
- MMMLU 86.7
- MMLU-ProX 82.2
- NOVA-63 58.6
- INCLUDE 82.8
- Global PIQA 88.4
- PolyMath 68.9
- WMT24++ 78.3
- MAXIFE 87.9
- MMMU 83.9
- MMMU-Pro 76.9
- MATH-Vision (MathVision) 86.2
- MathVista 87.4
- DynaMath 85.9
- ZeroBench 36.2
- VLMs Are Blind 96.7
- BabyVision 40.2
- RealWorldQA 85.1
- MMStar 82.9
- MMBench-EN v1.1 92.8
- SimpleVQA 61.7
- HallusionBench 67.6
- OmniDocBench v1.5 89.8
- CharXiv 77.2
- MMLongBench-Doc 59
- CC-OCR 81.8
- AI2D 93.3
- OCRBench 92.1
- ERQA 62
- CountBench 97
- RefCOCO / RefCOCO+ / RefCOCOg 91.3
- ODinW-13 44.5
- EmbSpatialBench 83.9
- RefSpatial-Bench 69.3
- LingoQA 80.8
- Hypersim 12.7
- SUNRGBD 36.2
- Nuscene 15.4
- Video-MME 87.3
- Video-MMMU 82
- MLVU 87.3
- MVBench 76.6
- LVBench 74.4
- MMVU 74.7
- ScreenSpot-Pro 70.4
- OSWorld-Verified 58
- AndroidWorld 66.4
- TIR-Bench 53.2
- V* 93.2
- SLAKE 81.6
- PMC-VQA 63.3
- MedXpertQA-MM 67.3
Dropped since the last post: SWE-bench Multilingual , SWE-Bench Pro , Aider Polyglot
- 19 Feb 2026
- Humanity's Last Exam 51.4
- ARC-AGI-2 77.1
- GPQA Diamond 94.3
- Terminal-Bench 2.0 68.5
- SWE-bench Verified 80.6
- SWE-Bench Pro 54.2
- LiveCodeBench Pro 2887
- SciCode 59
- APEX-Agents 33.5
- GDPval-AA 1317
- tau2-bench
- MCP Atlas 69.2
- BrowseComp 85.9
- MMMU-Pro 80.5
- MMMLU 92.6
- MRCR v2 84.9
Dropped since the last post: Codeforces rating , IMO 2025 , IPhO 2025 , IChO 2025 , CMT-Benchmark
Its model card adds 10 evaluations 10 new 3 dropped 26 results in all
Gemini 3.1 Pro Model CardAlignment
Bio and chem risk
AI R&D risk
- RE-Bench 1.27
Persuasion
Dropped since the lab’s previous model card: Online-Mind2Web , WebVoyager , AndroidWorld
- 17 Feb 2026
- Terminal-Bench 2.0 59.1
- SWE-bench Verified 79.6
- OSWorld-Verified 72.5
- tau2-bench 94.8
- MCP Atlas 61.3
- BrowseComp 74.7
- Humanity's Last Exam 49
- Vals Finance Agent 63.3
- GDPval-AA 1633
- ARC-AGI-2 58.3
- GPQA Diamond 89.9
- MMMU-Pro 75.6
- MMMLU 89.3
Its system card adds 67 evaluations 4 new 17 dropped 80 results in all
System Card: Claude Sonnet 4.6Capabilities
- SWE-bench Multilingual 75.9
- OpenRCA 27.9
- ARC-AGI 86.5
- AIME 2025 95.6
- Real-World Finance (Anthropic) 59.5
- Vending-Bench 2 7204.1
- CyberGym 65.2
- OpenAI MRCR v2
- GraphWalks
- LAB-Bench FigQA 58.8
- CharXiv 72.4
- WebArena 65.6
- WebArena-Verified 66.4
- Global MMLU 88.7
- MILU 89.6
- DeepSearchQA 89.2
- BioPipelineBench (Anthropic) 52.1
- BioMysteryBench 50.4
- Structural Biology (Anthropic)
- Organic Chemistry (Anthropic) 48.4
- Phylogenetics (Anthropic) 49.1
Health
Safety
- Single-turn harmful request evaluations (Anthropic) 99.4
- Single-turn benign request evaluations (Anthropic) 0.41
- Higher-difficulty harmful request evaluations (Anthropic) 99.4
- Higher-difficulty benign request evaluations (Anthropic) 0.18
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- SSH stress-testing evaluation (Anthropic) 92
- Agentic coding malicious requests (Anthropic) 100
- Malicious use of Claude Code (Anthropic) 79.3
- Malicious computer use (Anthropic) 99.4
Robustness
Alignment
- Reward-hack-prone coding tasks (Anthropic) 0
- Impossible tasks (Anthropic) 40
- Agentic code behavior scores (Anthropic)
- Over-eager behavior on GUI computer use tasks (Anthropic) 14.7
- Anthropic automated behavioral audit
- Petri
- Refusal to assist with AI safety R&D (Anthropic) 38
- Self-preference bias (Anthropic)
- Participation in junk science (Anthropic) 0
- SHADE-Arena 1.6
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Honesty human feedback win rate (Anthropic) , 100Q-Hard (Anthropic) , SimpleQA Verified , AA-Omniscience , ECLeKTic , False-premise questions (Anthropic) , External pilot use failure modes (Anthropic) , UK AISI alignment and monitorability evaluations , Internal codebase sabotage propensity (Anthropic) , Deference to governments in local languages (Anthropic) , Agentic misalignment suite (Anthropic) , Creative Biology Uplift Trial (Anthropic) , ASL-4 virology uplift trial (Anthropic) , Internal model use survey (Anthropic) , Internal AI research evaluation suite 2 , Misc CTF challenges (Anthropic) , Forensics CTF challenges (Anthropic)
- 16 Feb 2026
- MMLU-Pro 87.8
- MMLU-Redux 94.9
- SuperGPQA 70.4
- C-Eval 93
- IFEval 92.6
- IFBench 76.5
- MultiChallenge 67.6
- AA-LCR (Artificial Analysis Long Context Reasoning) 68.7
- LongBench v2 63.2
- GPQA Diamond 88.4
- Humanity's Last Exam 48.3
- LiveCodeBench v6 83.6
- HMMT 2025 94.8
- HMMT Nov 2025 92.7
- IMO-AnswerBench 80.9
- AIME 2026 91.3
- BFCL V4 72.9
- tau2-bench 86.7
- VITA-Bench 49.7
- DeepPlanning 34.3
- Toolathlon 38.3
- MCPMark 46.1
- BrowseComp 78.6
- BrowseComp-ZH 70.3
- WideSearch 74
- Seal-0 46.9
- MMMLU 88.5
- MMLU-ProX 84.7
- NOVA-63 59.1
- INCLUDE 85.6
- Global PIQA 89.8
- PolyMath 73.3
- WMT24++ 78.9
- MAXIFE 88.2
- SWE-bench Verified 76.4
- SWE-bench Multilingual 69.3
- SecCodeBench 68.3
- Terminal-Bench 2.0 52.5
- MMMU 85
- MMMU-Pro 79
- MATH-Vision (MathVision) 88.6
- MathVista 90.3
- We-Math 87.9
- DynaMath 86.3
- ZeroBench 12
- BabyVision 52.3
- RealWorldQA 83.9
- MMStar 83.8
- HallusionBench 71.4
- MMBench-EN v1.1 93.7
- SimpleVQA 67.1
- OmniDocBench v1.5 90.8
- CharXiv 80.8
- MMLongBench-Doc 61.5
- CC-OCR 82
- AI2D 93.9
- OCRBench 93.1
- ERQA 67.5
- CountBench 97.2
- ODinW-13 47
- EmbSpatialBench 84.5
- RefSpatial-Bench 73.6
- LingoQA 81.6
- V* 95.8
- Hypersim 12.5
- SUNRGBD 38.3
- Nuscene 16
- Video-MME 87.5
- Video-MMMU 84.7
- MLVU 86.7
- MVBench 77.6
- LVBench 75.5
- MMVU 75.4
- ScreenSpot-Pro 65.6
- OSWorld-Verified 62.2
- AndroidWorld 66.8
- SLAKE 79.9
- PMC-VQA 64.2
- MedXpertQA-MM 70
Dropped since the last post: Arena-Hard v2.0
- 14 Feb 2026
- MMLU-Pro 87
- Humanity's Last Exam 54.2
- HLE-Verified 73.6
- SimpleQA Verified 36
- HealthBench 57.7
- HealthBench Hard 29.1
- SuperGPQA 68.7
- LPFQA 52.6
- Encyclo-K 65.7
- AIME 2026 94.2
- AIME 2025 98.3
- HMMT 2025 97.3
- HMMT Nov 2025 93.3
- MathArena (Apex, IMO, USAMO) 20.3
- Beyond AIME 86.5
- IMO-AnswerBench 89.3
- Codeforces rating 3020
- AetherCode 60.6
- LiveCodeBench v6 87.8
- GPQA Diamond 88.9
- SUPERChem 51.6
- BABE 50
- PHYBench 74
- FrontierScience 74
- ARC-AGI 85.4
- ARC-AGI-2 37.5
- KOR-Bench 77.5
- ProcBench 96.6
- MRCR v2 54
- GraphWalks 83.3
- LongBench v2 63.8
- FRAMES 84.5
- DeR2 Bench 58.2
- CL-bench 20.8
- Global PIQA 92.3
- MMMLU 88.1
- DiscoX 82
- MultiChallenge 68.3
- COLLIE 93.9
- MARS-Bench 85.6
- Inverse IFEval 78.9
- LongFact 92.8
- FActScore 71.2
- Terminal-Bench 2.0 55.8
- SWE-Lancer 49.4
- SWE-bench Verified 76.5
- Multi-SWE-bench 45.2
- SWE-Bench Pro 46.9
- SWE-bench Multilingual 71.7
- SciCode 48.5
- SWE-EVO 8.5
- Aider Polyglot 80
- ArtifactsBench 66.6
- CodeSimpleQA 58
- SpreadsheetBench Verified 79.1
- BrowseComp 77.3
- BrowseComp-ZH 82.4
- WideSearch 74.7
- FinSearchComp 70.2
- DeepSearchQA 77.4
- Seal-0 49.5
- tau2-bench 92.3
- MCPMark 54.7
- BFCL V4 73.4
- VITA-Bench 47
- DeepConsult 61.1
- DeepResearch Bench 53.3
- ResearchRubrics 50.7
- MM-BrowseComp 48.8
- HLE-VL 39.2
- MathVista 89.8
- MATH-Vision (MathVision) 88.8
- LogicVista 81.9
- VisuLogic 47.4
- VLMsAreBiased 77.4
- VLMs Are Blind 98.6
- BabyVision 60.6
- ChartQA Pro 71.2
- OmniDocBench v1.5 0.099
- DUDE 72.4
- MMLongBench 74.8
- MMLongBench-Doc 61.4
- TVBench 75
- TempCompass 89.6
- MotionBench 75.2
- EgoTempo 71.8
- Video-MME 89.5
- LVBench 76.4
- Math Kangaroo 2025 (MathArena) 90.5
- MathCanvas-Bench 61.9
- MineDojo 49
- PutnamBench 35.5
- IMO 2025 35
- CMO 2025 114
Dropped since the last post: FATE-H , FATE-X , Putnam 2025 , CombiBench
Its model card adds 69 evaluations 161 results in all
Seed2.0 Model Card: Towards Intelligence Frontier for Real-World ComplexityCapabilities
- FrontierScience-Research 25
- FrontierScience-Olympiad 74
- Chinese Complex Instruction Following (ByteDance Seed in-house) 75.3
- DynaMath 68.9
- MMMU 85.4
- MMMU-Pro 78.2
- EMMA 72
- Scientists' First Exam (SFE) 55.6
- HiPhO 74.1
- XLRS-Bench 54.6
- PhyX 72.1
- VPCT 76
- ZeroBench 12
- VisFactor 36.8
- RealWorldQA 86
- SimpleVQA 71.4
- HallusionBench 68
- MME-CC 57
- MMStar 83
- MuirBench 81.8
- MTVQA-Test 51.1
- WorldVQA 49.9
- Vibe-Eval 81.4
- ViVerBench 75.9
- CountBench 95.5
- Point-Bench 81.4
- BLINK 79.5
- MMSI-Bench 32.5
- TreeBench 64.7
- RefSpatial-Bench 72.6
- All-Angles Bench 72.1
- ERQA 68.5
- OCRBench v2 62.5
- CharXiv (DQ) 93.5
- CharXiv 80.5
- LongDocURL 74.7
- Video-MMMU 86.9
- MMVU 78.2
- Video SimpleQA 71.9
- VideoReasonBench 77.8
- MORSE-500 37.4
- Video-Holmes 67.4
- MINERVA 66.5
- ContPhy 67.4
- TOMATO 59.9
- CG-Bench 65
- LongVideoBench 80.3
- VideoEval-Pro 48
- CrossVid 60.3
- OVBench 69.2
- LiveSports-3K 78
- OVO-Bench 77
- ODV-Bench 72.5
- ViSpeak-Bench 78.5
- ZeroVideo 14.5
- BIOBench (ByteDance Seed) 53.5
- AInstein-SWE-Bench 47.7
- NL2Repo-Bench 27.9
- Complex Workflow (ByteDance Seed in-house) 64.7
- Customer Support Q&A (ByteDance Seed in-house) 72.4
- GDPVal-Diamond 21.3
- XpertBench 64.5
- Education (ByteDance Seed in-house) 62.8
- Compositional Tasks (ByteDance Seed in-house) 59.1
- Text Classification (ByteDance Seed in-house) 69
- Information Extraction (ByteDance Seed in-house) 52
- WorldTravel
Hallucination
- LongFact-Objects 92.9
- LongFact-Concepts 92.8
- 12 Feb 2026
- SWE-Bench Pro 51.5
- Terminal-Bench 2.0 58.4
Dropped since the last post: AIME 2025 , FrontierMath , GPQA Diamond , Humanity's Last Exam , HMMT 2025 , MMMU , MMMU-Pro , CharXiv , Video-MMMU , ERQA , SWE-Lancer , SWE-bench Verified , Aider Polyglot , MultiChallenge , Internal API instruction following (OpenAI) , COLLIE , tau2-bench , OpenAI MRCR , GraphWalks , BrowseComp Long Context 128k , Video-MME , LongFact , FActScore
- 12 Feb 2026 Gemini 3 Deep Think Google DeepMind Gemini 3 Deep Think: Advancing science, research and engineering (Google blog)
- Humanity's Last Exam 48.4
- ARC-AGI-2 84.6
- Codeforces rating 3455
- IMO 2025
- IPhO 2025
- IChO 2025
- CMT-Benchmark 50.5
Dropped since the last post: GPQA Diamond , AIME 2025 , MMMU-Pro , ScreenSpot-Pro , CharXiv , OmniDocBench v1.5 , Video-MMMU , LiveCodeBench Pro , Terminal-Bench 2.0 , SWE-bench Verified , tau2-bench , Toolathlon , MCP Atlas , Vending-Bench 2 , FACTS Benchmark Suite , SimpleQA Verified , MMMLU , Global PIQA , MRCR v2
- 12 Feb 2026 MiniMax-M2.5 MiniMax MiniMax-M2.5 model card (Hugging Face) and launch post (minimax.io/news/minimax-m25)
- SWE-bench Verified 80.2
- SWE-Bench Pro 55.4
- Terminal-Bench 2.0 51.7
- Multi-SWE-bench 51.3
- SWE-bench Multilingual 74.1
- VIBE-Pro (MiniMax) 54.2
- BrowseComp 76.3
- WideSearch 70.3
- RISE 50.2
- Berkeley Function Calling Leaderboard (BFCL) 76.8
- tau2-bench 97.8
- GDPval-MM 59
- MEWC 74.4
- Finance Modeling 21.6
- AIME 2025 86.3
- GPQA Diamond 85.2
- Humanity's Last Exam 19.4
- SciCode 44.4
- IFBench 70
- AA-LCR (Artificial Analysis Long Context Reasoning) 69.5
Dropped since the last post: SWT-bench , SWE-Perf , SWE-Review , OctoCodingbench , VIBE (MiniMax) , Toolathlon , MMLU-Pro , LiveCodeBench
Its technical report adds 11 evaluations 10 new 11 dropped 25 results in all
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World IntelligenceCapabilities
- NL2Repo-Bench 26.6
- HyperTask 59.4
- GDPval-AA 35
- Toolathlon 38.3
- MM-ClawBench 57.6
- MEWC v2 49.8
- Finance Modeling Pro 33.8
- AIME 2026 87.2
- MMLU-Pro 85.2
- Artificial Analysis Intelligence Index v4.0 42
AI R&D risk
- MLE-bench 51.5
Dropped since the lab’s previous technical report: AIME 2024 , MATH-500 , LiveCodeBench v6 , FullStackBench , ZebraLogic , SWE-bench Verified , OpenAI MRCR , LongBench v2 , tau-bench , SimpleQA , MultiChallenge
- 11 Feb 2026
- Humanity's Last Exam 50.4
- AIME 2026 92.7
- HMMT Nov 2025 96.9
- IMO-AnswerBench 82.5
- GPQA Diamond 86
- SWE-bench Verified 77.8
- SWE-bench Multilingual 73.3
- Terminal-Bench 2.0 56.2
- CyberGym 43.2
- BrowseComp 75.9
- BrowseComp-ZH 72.7
- tau2-bench 89.7
- MCP Atlas 67.8
- Toolathlon 38
- Vending-Bench 2 4432.1
Dropped since the last post: MMLU-Pro , AIME 2025 , HMMT 2025 , LiveCodeBench v6 , Terminal-Bench Hard
Its technical report adds 34 evaluations 34 new 39 dropped 49 results in all
GLM-5: from Vibe Coding to Agentic EngineeringCapabilities
- HMMT 2025 97.9
- LongBench v2 64.5
- GDPval-AA 1409
- Artificial Analysis Intelligence Index v4.0 50
- LMArena (Elo) 1452
- Code Arena 1449
- CC-Bench-V2 Frontend 74.8
- CC-Bench-V2 Backend 25.8
- CC-Bench-V2 Large Repo Exploration 65.6
- CC-Bench-V2 Multi-Step Chained Tasks 52.3
- SWE-rebench 42.1
- ZMultiTransBench 1050
- MENT-SNS 1013
- Translation Human Scoring (Z.ai) 3.12
- ZMultiDialBench 8.57
- IF-Badcase 83.2
- IFBench 72
- MultiChallenge 64.8
- ToolCall-Badcase 95.8
- Slide generation human evaluation (Z.ai) 67.5
- BIG-Bench Hard 87.4
- MMLU 88.3
- HellaSwag 88.1
- PIQA 84.6
- TriviaQA 80.9
- EvalPlus 87
- LiveCodeBench 34.4
- GSM8K 68.8
- MATH 56.4
- CLUEWSC 84.2
- C-Eval 88.8
- C3 80.3
Hallucination
- SimpleQA 36.9
- Chinese SimpleQA 75.2
Dropped since the lab’s previous technical report: MMBench-EN v1.1 , MMBench-CN v1.1 , MMStar , BLINK , MuirBench , HallusionBench , ZeroBench , MMMU , MMMU-Pro , MathVista , MATH-Vision (MathVision) , MathVerse , DynaMath , LogicVista , AI2D , We-Math , MMLongBench-Doc , OCRBench , ChartQA Pro , ChartMuseum , RefCOCO / RefCOCO+ / RefCOCOg , TreeBench , Ref-L4 , OmniSpatial , CVBench , ERQA , All-Angles Bench , OSWorld , AndroidWorld , WebVoyager , WebQuest , Design2Code , Flame-React-Eval , Video-MME , MMVU , Video-MMMU , LVBench , MotionBench , MVBench
- 10 Feb 2026
- IMO-AnswerBench 90
- AIME 2026 95
- HMMT Nov 2025 98.3
- LiveCodeBench v6 89.4
- Meta Agents Research Environments (ARE) / Gaia2 75
- tau2-bench 78.4
- SWE-bench Verified 72.4
- ARC-AGI-2 19.3
- IMO 2025 35
- CMO 2025 105
Dropped since the last post: AIME 2025 , Codeforces rating , ARC-AGI , Arena-Hard v2.0 , HMMT 2025 , HealthBench , Creative Writing v3 , ICPC World Finals 2025
- 9 Feb 2026
- 6 Feb 2026
ERNIE 5.0 Baidu ERNIE 5.0: A 2.4 Trillion-Parameter Unified Multimodal Foundation Model (ERNIE Blog) - SimpleQA 74
- Chinese SimpleQA 86
- MMLU-Pro 83.8
- Humanity's Last Exam 25.8
- GPQA Diamond 86.4
- AIME 2025 89.1
- HMMT 2025 79.6
- LiveCodeBench v6 76.2
- HumanEval+ 94.5
- MBPP+ 82.5
- ZebraLogic 96.5
- BIG-Bench Extra Hard 66.6
- IFEval 93.3
- MultiChallenge 66
- Multi-IF 85.6
- tau2-bench 78.8
- AceBench 88.7
- BFCL V4 66.5
- BrowseComp-ZH 64.7
- SpreadsheetBench 40.1
- MMMU-Pro 68.6
- MathVista 84.8
- MathVerse 85.1
- MATH-Vision (MathVision) 74.3
- VisualPuzzles 64.8
- VisuLogic 32
- VLMs Are Blind 91.4
- ChartQA 87.8
- AI2D 96.9
- DocVQA 95.5
- OCRBench 87.8
- CharXiv 67.1
- SimpleVQA 67.6
- HallusionBench 63.9
- MMStar 75.5
- BLINK 70
- CVBench 87.2
- CountBench 96.5
- Video-MME 81.3
- Video-MMMU 81.1
- MMVU 72.2
- GenEval 90.1
- VBench 84.2
Dropped since the last post: CCBench (MMBench Chinese Culture) , MMMU , MVBench , C-Eval , CMMLU , MATH-500 , CNMO 2024 , GSM8K , CMATH , BIG-Bench Hard , DROP , CLUEWSC , LiveCodeBench
Its technical report adds 9 evaluations 52 results in all
ERNIE 5.0 Technical ReportCapabilities
- AISHELL-1 0.31
- AISHELL-2 2.64
- WenetSpeech
- LibriSpeech
- FLEURS
- VoiceBench
- MMAU 80.4
- Clotho-AQA 73.7
- Seed-TTS-eval
- 5 Feb 2026
- Terminal-Bench 2.0 65.4
- SWE-bench Verified 80.8
- OSWorld-Verified 72.7
- tau2-bench 95.6
- MCP Atlas 59.5
- BrowseComp 84
- Humanity's Last Exam 53
- Vals Finance Agent 60.7
- GDPval-AA 1606
- ARC-AGI-2 68.8
- GPQA Diamond 91.3
- MMMU-Pro 77.3
- MMMLU 91.1
Dropped since the last post: MMMU
Its system card adds 80 evaluations 25 new 12 dropped 93 results in all
System Card: Claude Opus 4.6Capabilities
- SWE-bench Multilingual 77.8
- OpenRCA 34.9
- ARC-AGI 94
- AIME 2025 99.8
- Real-World Finance (Anthropic) 64.1
- Vending-Bench 2 8017.6
- CyberGym 66.6
- OpenAI MRCR v2
- GraphWalks
- LAB-Bench FigQA 58
- CharXiv 68.5
- WebArena 68
- DeepSearchQA 91.3
- BioPipelineBench (Anthropic) 53.1
- BioMysteryBench 61.5
- Structural Biology (Anthropic)
- Organic Chemistry (Anthropic) 53.9
- Phylogenetics (Anthropic) 61.3
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 99.6
- Single-turn benign request evaluations (Anthropic) 0.68
- Higher-difficulty harmful request evaluations (Anthropic) 99.2
- Higher-difficulty benign request evaluations (Anthropic) 0.04
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- SSH stress-testing evaluation (Anthropic) 95
- Agentic coding malicious requests (Anthropic) 99.3
- Malicious use of Claude Code (Anthropic) 83.2
- Malicious computer use (Anthropic) 88.3
Robustness
Alignment
- External pilot use failure modes (Anthropic)
- Reward-hack-prone coding tasks (Anthropic) 0
- Impossible tasks (Anthropic) 50
- Agentic code behavior scores (Anthropic)
- Over-eager behavior on GUI computer use tasks (Anthropic) 11.4
- Anthropic automated behavioral audit
- Petri
- UK AISI alignment and monitorability evaluations
- Refusal to assist with AI safety R&D (Anthropic) 14
- Self-preference bias (Anthropic)
- Internal codebase sabotage propensity (Anthropic) 0
- Participation in junk science (Anthropic) 1.1
- Deference to governments in local languages (Anthropic)
- SHADE-Arena 18
- Agentic misalignment suite (Anthropic)
Bio and chem risk
Cyber risk
AI R&D risk
- SWE-bench Verified (hard subset) 47.2
- Internal model use survey (Anthropic) 0
- Kernels task (Anthropic) 190
- Time series forecasting task (Anthropic)
- Text-based reinforcement learning task (Anthropic) 0.975
- LLM training task (Anthropic) 34
- Quadruped reinforcement learning task (Anthropic)
- Novel compiler task (Anthropic)
- Internal AI research evaluation suite 2 0.612
Dropped since the lab’s previous system card: MMMU , SWE-Bench Pro , BrowseComp-Plus , Multi-agent search (Anthropic internal benchmark) , SpreadsheetBench , Sycophancy on user-provided prompts (Anthropic) , Training distribution hack rate (Anthropic) , Subversion Strategy Eval , LAB-Bench ProtocolQA , LAB-Bench SeqQA , LAB-Bench CloningScenarios , Bioinformatics evaluations (Anthropic)
- 5 Feb 2026
- SWE-Bench Pro 56.8
- Terminal-Bench 2.0 77.3
- OSWorld-Verified 64.7
- GDPval 70.9
- SWE-Lancer 81.4
Dropped since the last post: Capture the Flag (Professional)
Its system card adds 15 evaluations 4 new 10 dropped 15 results in all
GPT-5.3-Codex System CardSafety
Alignment
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: StrongREJECT , Image input safety evaluations , Malware Refusals (Golden Set) , Coding prompt injections , Long-form Biological Risk Questions , SWE-Lancer , PaperBench , MLE-bench , OpenAI PRs , METR Time Horizon
- 2 Feb 2026
- SciReasoner evaluation suite 55.5
- Scientists' First Exam (SFE) 52.7
- SMolInstruct 74.8
- Matbench 72.8
- Mol-Instructions 48.8
- MicroVQA 63.3
- Biology-Instructions 52.5
- XLRS-Bench 52.8
- MSEarth 65.2
- MMMU-Pro 72.8
- MMLU-Pro 86.6
- AIME 2025 93.1
- IMO-AnswerBench 77.3
- RefCOCO / RefCOCO+ / RefCOCOg 91.9
- IFBench 71.2
- OCRBench v2 60.1
- SArena 83.5
- LiveCodeBench v6 74.3
- GAIA 77.4
- tau2-bench 80.9
- ScreenSpot-v2 93.6
Dropped since the last post: MMStar , MM-Vet , MMBench-EN v1.1 , MTVQA-Test , AI2D , OCRBench , WildVision , MME-RealWorld , HallusionBench , MVBench , Video-MME , MLVU , LongVideoBench , MMMU , MathVista , MATH-Vision (MathVision) , MathVerse , DynaMath , We-Math , OlympiadBench , LogicVista , MATH-500 , AIME 2024 , GPQA Diamond , C-Eval , GAOKAO-Bench , IFEval , SGP-Bench , ScreenSpot , OSWorld-G , VSI-Bench , ERQA , SpaCE-10 , OmniSpatial
Its technical report adds 1 evaluation 1 new 67 dropped 22 results in all
Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion ScaleCapabilities
Dropped since the lab’s previous technical report: MMStar , MM-Vet , MMBench-EN v1.1 , MME , MTVQA-Test , MMMB , AI2D , ChartQA , TextVQA , DocVQA , InfographicVQA , OCRBench , SEED-Bench-2-Plus , CharXiv , VCR (Visual Caption Restoration) , WildVision , MME-RealWorld , RealWorldQA , R-Bench (real-world robustness) , BLINK , Mantis-Eval , MMIU , MuirBench , MMT-Bench , MIRB , HallusionBench , CRPE , POPE , MVBench , Video-MME , MMBench-Video , MLVU , LongVideoBench , MMMU , MathVista , MATH-Vision (MathVision) , MathVerse , DynaMath , We-Math , OlympiadBench , LogicVista , MATH-500 , AIME 2024 , GPQA Diamond , C-Eval , GAOKAO-Bench , IFEval , MMLU , CMMLU , TriviaQA , Natural Questions , C3 , RACE , WinoGrande , HellaSwag , BIG-Bench Hard , GSM8K , MATH , HumanEval , SGP-Bench , ScreenSpot , OSWorld-G , Windows Agent Arena , VSI-Bench , ERQA , SpaCE-10 , OmniSpatial
- 2 Feb 2026
- SWE-bench Verified 70.6
- SWE-bench Multilingual 62.8
- SWE-Bench Pro 44.3
- Terminal-Bench 2.0 36.2
- Aider Polyglot 66.2
Dropped since the last post: MMLU-Redux , GPQA Diamond , AIME 2025 , ZebraLogic , MultiPL-E , IFEval , Creative Writing v3 , WritingBench , Berkeley Function Calling Leaderboard (BFCL) , Multi-IF , PolyMath , LiveBench , Arena-Hard v2.0 , LibriSpeech , Common Voice , FLEURS , VoiceBench , MMAU , MuChoMusic , MMStar , HallusionBench , MM-MT-Bench , MMMU , MMMU-Pro , MathVista , MATH-Vision (MathVision) , AI2D , ChartQA , CountBench , Video-MME , LVBench , MLVU , WorldSense (omnimodal) , Daily-Omni , Video-Holmes , Seed-TTS-eval , WenetSpeech
Its technical report adds 23 evaluations 19 new 36 dropped 28 results in all
Qwen3-Coder-Next Technical ReportCapabilities
- Template Following generalization (Qwen) 92.7
- EvalPlus 86.6
- MultiPL-E 88.2
- CRUXEval 95.9
- LiveCodeBench v6 58.9
- OJBench 23
- Codeforces rating 2100
- FullStackBench 60.6
- Spider 83.7
- Bird-SQL 63.6
- MMLU 87.7
- MMLU-Redux 91.2
- MMLU-Pro 80.5
- GPQA Diamond 74.5
- SuperGPQA 57.5
- HMMT 2025 70.2
- HMMT Nov 2025 75.6
- AIME 2024 89
- AIME 2025 83.1
Cyber risk
- AthenaBench
- PrimeVul 48.3
- SecCodeBench
- CWEval 56.3
Dropped since the lab’s previous technical report: ZebraLogic , IFEval , Creative Writing v3 , WritingBench , Berkeley Function Calling Leaderboard (BFCL) , Multi-IF , PolyMath , LiveBench , Arena-Hard v2.0 , WenetSpeech , LibriSpeech , Common Voice , FLEURS , VoiceBench , MMAU , MMSU , MuChoMusic , MMStar , HallusionBench , MM-MT-Bench , MMMU , MMMU-Pro , MathVista , MATH-Vision (MathVision) , AI2D , ChartQA , CountBench , Video-MME , LVBench , MLVU , WorldSense (omnimodal) , Daily-Omni , Video-Holmes , Seed-TTS-eval , MiniMax TTS Multilingual Test Set , CV3-Eval
- 2 Feb 2026
- tau2-bench 88.2
- BrowseComp 69
- BrowseComp-ZH 73.7
- GAIA 84.5
- xbench-DeepSearch 83.7
- xbench-DeepSearch-2510 56.3
- ResearchRubrics 65.3
- AIME 2025 97.3
- HMMT 2025 98.4
- HMMT Nov 2025 94
- IMO-AnswerBench 85.4
- LiveCodeBench v6 86.4
- SWE-bench Verified 74.4
- Terminal-Bench 2.0 51
Dropped since the last post: MMMU , MATH-Vision (MathVision) , SimpleVQA , HallusionBench , ZeroBench , DynaMath , GPQA Diamond , CNMO 2024
Its technical report adds 16 evaluations 30 results in all
Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active ParametersCapabilities
- CF-Div2-Stepfun 86.1
- MMLU-Pro 84.4
- GPQA Diamond 83.5
- Humanity's Last Exam 23.1
- ARC-AGI 54.8
- SWE-bench Multilingual 67.4
- Arena-Hard v2.0 74
- MultiChallenge 55.7
- IFBench 67.4
- LongBench v2 57.5
- OpenAI MRCR 28.8
- FRAMES 76.5
- RepoQA 88.5
- Data Analysis Benchmark (StepFun) 39.6
- Consulting and Recommendations Benchmark (StepFun) 70.5
- AndroidDaily 57
- 30 Jan 2026
- OmniDocBench v1.5 94.6
- OCRBench 94
Dropped since the last post: AIME 2025 , GPQA Diamond , LiveCodeBench v6 , Humanity's Last Exam , SWE-bench Verified , tau2-bench , BrowseComp
Its technical report adds 0 evaluations 39 dropped 2 results in all
GLM-OCR Technical ReportDropped since the lab’s previous technical report: MMBench-EN v1.1 , MMBench-CN v1.1 , MMStar , BLINK , MuirBench , HallusionBench , ZeroBench , GeoBench , MMMU , MMMU-Pro , MathVista , MATH-Vision (MathVision) , MathVerse , DynaMath , LogicVista , AI2D , We-Math , MMLongBench-Doc , ChartQA Pro , ChartMuseum , RefCOCO / RefCOCO+ / RefCOCOg , TreeBench , Ref-L4 , OmniSpatial , CVBench , ERQA , All-Angles Bench , OSWorld , AndroidWorld , WebVoyager , WebQuest , Design2Code , Flame-React-Eval , Video-MME , MMVU , Video-MMMU , LVBench , MotionBench , MVBench
- 27 Jan 2026
- MMLU 87.2
- MMLU-Pro 75.2
- GPQA Diamond 63.3
- AIME 2025 24
Its technical report adds 1 evaluation 5 results in all
Arcee Trinity Large Technical ReportHallucination
- SimpleQA 23.9
- 27 Jan 2026
- SWE-bench Verified 54.2
Dropped since the last post: MATH , AIME 2024 , AIME 2025 , OMEGA , BIG-Bench Hard , ZebraLogic , AGIEval , HumanEval+ , MBPP+ , LiveCodeBench , IFEval , IFBench , MMLU , PopQA , GPQA Diamond , AlpacaEval 2.0 , Tulu 3 Safety Eval
Its technical report adds 0 evaluations 27 dropped 1 results in all
SERA: Soft-Verified Efficient Repository AgentsDropped since the lab’s previous technical report: MATH , AIME 2024 , AIME 2025 , OMEGA , BIG-Bench Hard , ZebraLogic , AGIEval , HumanEval+ , MBPP+ , LiveCodeBench , IFEval , IFBench , MMLU , PopQA , GPQA Diamond , AlpacaEval 2.0 , Tulu 3 Safety Eval , DoAnythingNow (DAN) , HarmBench , TrustLLM JailbreakTrigger , WildJailbreak , WildGuardTest , XSTest , BBQ (Bias Benchmark for QA) , StrongREJECT , ToxiGen , WMDP
- 27 Jan 2026
- Humanity's Last Exam 50.2
- AIME 2025 96.1
- HMMT 2025 95.4
- IMO-AnswerBench 81.8
- GPQA Diamond 87.6
- MMLU-Pro 87.1
- MMMU-Pro 78.5
- CharXiv 77.5
- MATH-Vision (MathVision) 84.2
- MathVista 90.1
- ZeroBench 9
- OCRBench 92.3
- OmniDocBench v1.5 88.8
- InfographicVQA 92.6
- SimpleVQA 71.2
- WorldVQA 46.3
- Video-MMMU 86.6
- MMVU 80.4
- MotionBench 70.4
- Video-MME 87.4
- LongVideoBench 79.8
- LVBench 75.9
- SWE-bench Verified 76.8
- SWE-Bench Pro 50.7
- SWE-bench Multilingual 73
- Terminal-Bench 2.0 50.8
- PaperBench 63.5
- CyberGym 41.3
- SciCode 48.7
- OJBench 57.4
- LiveCodeBench v6 85
- LongBench v2 61
- BrowseComp 74.9
- WideSearch 72.7
- DeepSearchQA 77.1
- FinSearchComp 67.8
- Seal-0 57.4
Dropped since the last post: MMLU-Redux , Longform Writing , HealthBench , BrowseComp-ZH , FRAMES , Multi-SWE-bench , LiveCodeBench , Terminal-Bench
Its technical report adds 11 evaluations 11 new 30 dropped 48 results in all
Kimi K2.5: Visual Agentic Intelligence (arXiv 2602.02276)Capabilities
- HMMT Nov 2025 91.1
- AdvancedIF 75.6
- GDPval-AA 41
- MMMU 84.3
- BabyVision 36.5
- BLINK 78.9
- MMVP 87
- OSWorld-Verified 63.3
- WebArena 58.9
- Swarm Bench 58.3
Hallucination
- SimpleQA Verified 36.9
Dropped since the lab’s previous technical report: MultiPL-E , Multi-SWE-bench , SWE-Lancer , Terminal-Bench , Aider Polyglot , tau2-bench , AceBench , AIME 2024 , MATH-500 , CNMO 2024 , PolyMath , ZebraLogic , AutoLogi , SuperGPQA , MMLU , MMLU-Redux , IFEval , MultiChallenge , SimpleQA , LiveBench , Arena-Hard v2.0 , FACTS Grounding , Vectara Hallucination Leaderboard (HHEM) , FaithJudge , FRAMES , OpenAI MRCR , DROP , Kimi Chinese in-house open-ended benchmark , LMArena (Elo) , Kimi Promptfoo red-team safety evaluation
- 26 Jan 2026
- AIME 2025 80.3
- MMLU-Pro 77
- IFBench 55.8
- Arena-Hard v2.0 62.5
Dropped since the last post: KMMLU , HAE-RAE Bench , KBL , KBank-MMLU (Upstage) , KorMedMCQA , MMLU , IFEval , HumanEval , SWE-bench Verified , MATH-500 , AIME 2024 , Berkeley Function Calling Leaderboard (BFCL)
- 23 Jan 2026
- MMLU-Pro 85.7
- MMLU-Redux 92.8
- C-Eval 93.7
- GPQA Diamond 87.4
- Humanity's Last Exam 49.8
- LiveCodeBench v6 85.9
- HMMT 2025 98
- HMMT Nov 2025 94.7
- IMO-AnswerBench 83.9
- SWE-bench Verified 75.3
- IFBench 70.9
- MultiChallenge 63.3
- Arena-Hard v2.0 90.2
- tau2-bench 82.1
- BFCL V4 67.7
- VITA-Bench 40.9
- DeepPlanning 28.7
- AA-LCR (Artificial Analysis Long Context Reasoning) 68.7
Dropped since the last post: AIME 2025 , SuperGPQA , LMArena (Elo)
- 20 Jan 2026
- GPQA Diamond 37.9
- MMLU-Pro 49.6
- IFEval 88.4
- IFBench 44.9
- Multi-IF 69.3
- GSM8K 85.6
- MATH-500 88
- AIME 2025 31.7
- Berkeley Function Calling Leaderboard (BFCL) 57
- 19 Jan 2026
- AIME 2025 91.6
- GPQA Diamond 75.2
- LiveCodeBench v6 64
- Humanity's Last Exam 14.4
- SWE-bench Verified 59.2
- tau2-bench 79.5
- BrowseComp 42.8
Dropped since the last post: MMBench-EN v1.1 , MMBench-CN v1.1 , MMStar , BLINK , MuirBench , MMMU , MMMU-Pro , Video-MMMU , AI2D , MathVista , We-Math , ChartQA Pro , ChartMuseum , OCRBench , MMLongBench-Doc , RefCOCO / RefCOCO+ / RefCOCOg , OSWorld , AndroidWorld , WebVoyager , WebQuest , Design2Code , Flame-React-Eval , Video-MME , MMVU , LVBench , MotionBench
- 14 Jan 2026
- AIME 2025 100
- HMMT 2025 97.5
- IMO-AnswerBench 86.8
- AMO-Bench 66
- BrowseComp 73.1
- BrowseComp-ZH 77.7
- RWSearch 79.5
- tau2-bench 88.2
- tau2-Noise (LongCat) 67.1
- VITA-Bench 29.3
- VitaBench-Noise (LongCat) 20.5
- Random Complex Tasks (LongCat) 35.8
- Humanity's Last Exam 25.2
- GPQA Diamond 85.2
- LiveCodeBench v6 82.8
- OJBench 42.2
- OIBench 47.7
- SWE-bench Verified 70
Dropped since the last post: OmniBench , WorldSense (omnimodal) , Daily-Omni , UNO-Bench , MMBench , RealWorldQA , MMStar , MathVista , MMMU , MM-Vet , BLINK , MuirBench , Mantis-Eval , ChartQA , DocVQA , OCRBench , OmniDocBench , RefCOCO / RefCOCO+ / RefCOCOg , CountBench , VisualWebBench , ScreenSpot-v2 , AndroidControl , MVBench , NExT-QA , TempCompass , Video-MME , LongVideoBench , MMVU , Video-MMMU , LibriSpeech , AISHELL-1 , AISHELL-2 , FLEURS , Common Voice , WenetSpeech , CoVoST2 , MMAU , Clotho-AQA , OpenAudioBench , VoiceBench , MMLU , MMLU-Pro , C-Eval , CMMLU , IFEval , COLLIE , Meeseeks , MATH-500 , AIME 2024 , Beyond AIME , DROP , ZebraLogic , GraphWalks , LiveCodeBench , HumanEval+ , MBPP+
Its technical report adds 0 evaluations 56 dropped 18 results in all
LongCat-Flash-Thinking-2601 Technical ReportDropped since the lab’s previous technical report: MMBench , RealWorldQA , MMStar , MathVista , MMMU , MM-Vet , BLINK , MuirBench , Mantis-Eval , ChartQA , DocVQA , OCRBench , OmniDocBench , RefCOCO / RefCOCO+ / RefCOCOg , CountBench , VisualWebBench , ScreenSpot-v2 , AndroidControl , MVBench , NExT-QA , TempCompass , Video-MME , LongVideoBench , MMVU , Video-MMMU , LibriSpeech , AISHELL-1 , AISHELL-2 , FLEURS , Common Voice , WenetSpeech , CoVoST2 , MMAU , Clotho-AQA , OpenAudioBench , VoiceBench , MMLU , MMLU-Pro , C-Eval , CMMLU , IFEval , COLLIE , Meeseeks , MATH-500 , AIME 2024 , Beyond AIME , DROP , ZebraLogic , GraphWalks , HumanEval+ , MBPP+ , OmniBench , WorldSense (omnimodal) , Daily-Omni , UNO-Bench , Real-time Audio-Visual Interaction Evaluation
- 13 Jan 2026
- MMMU 78.1
- MATH-Vision (MathVision) 70.8
- MathVista 84
- PhyX 59.5
- MMBench 92
- MMStar 77.5
- ReMI 67.3
- OCRBench 86.8
- AI2D 89.3
- ScreenSpot-v2 92.6
- ScreenSpot-Pro 51.5
- OSWorld-G 59
- BLINK 66.8
- All-Angles Bench 57.2
- HumanEval-V 66
- AIME 2025 87.7
- HMMT 2025 78.2
- LiveCodeBench 75.8
Its technical report adds 52 evaluations 69 results in all
STEP3-VL-10B Technical ReportCapabilities
- MMMU-Pro 64.1
- LogicVista 66.9
- DynaMath 56.4
- ZeroBench 1
- MathVerse 75.7
- We-Math 73
- VisuLogic 29.7
- Humanity's Last Exam 10.7
- SimpleVQA 53.1
- MMVP 68.2
- M3GIA 78.4
- Do You See Me 67.5
- CountBench 88.8
- CountQA 33.7
- PixMo-Count 70.8
- MM-MT-Bench 8.14
- MIA-Bench 92
- MM-IFEval 61.9
- Design2Code 79.5
- OmniAI OCR Benchmark 77
- CC-OCR
- CVBench 83.5
- MMSI-Bench 32.2
- ERQA 48.9
- OmniSpatial 51.6
- MindCube 62.8
- RealWorldQA 74.4
- SpatialViz-Bench 45.5
- STARE 61.8
- CoreCognition 66.7
- V* 82.8
- ViewSpatial-Bench 46.1
- CharXiv 59.5
- CSVQA 63.3
- OmniDocBench 21.5
- MMBench-GUI 81.5
- MMLU-Pro 76
- GPQA Diamond 70.8
- SuperGPQA 50.4
- LiveBench 69.7
- AIME 2024 90.9
- CNMO 2024 78.2
- Beyond AIME 63.2
- IMO-AnswerBench 62.1
- LiveCodeBench v6 75.8
- IFEval 82.2
- IFBench 43.3
- MultiChallenge 62.6
- Arena-Hard v2.0 58.6
- WildBench 86
Health
- HealthBench 44.7
Hallucination
- HallusionBench 64.9
- 13 Jan 2026
- HealthBench 65.1
- HealthBench Hard 44.4
- HealthBench-Hallu 3.5
- ScanBench 73.8
Its technical report adds 0 evaluations 4 results in all
- 8 Jan 2026
- VSI-Bench 68.6
- MMSI-Bench 42.5
- MindCube 89.9
- ViewSpatial-Bench 61.3
- SITE 47.5
- BLINK 68
- 3DSRBench 62.4
- EmbSpatialBench 81
- 6 Jan 2026
- LiveCodeBench v6 67.9
Dropped since the last post: Putnam 2025
- 5 Jan 2026 Falcon H1R 7B Technology Innovation Institute Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling (Falcon blog)
- AIME 2024 88.1
- AIME 2025 83.1
- HMMT 2025 64.9
- AMO-Bench 36.3
- LiveCodeBench v6 68.6
- SciCode 28.3
- Terminal-Bench Hard 4.96
- GPQA Diamond 61.3
- MMLU-Pro 72.1
- Humanity's Last Exam 11.1
- IFBench 53.4
Dropped since the last post: Open Arabic LLM Leaderboard v2 , ArabicMMLU , AraTrust
Its technical report adds 7 evaluations 7 new 15 dropped 18 results in all
Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time ScalingCapabilities
- MATH-500 97.4
- tau2-bench
Safety
- HarmBench
- XSTest 96.4
- SALAD-Bench
Robustness
Dropped since the lab’s previous technical report: Open Arabic LLM Leaderboard v2 , AlGhafa , ArabicMMLU , MadinahQA , AraTrust , ALRAGE , ArabCulture , AraDiCE , IFEval , GSM8K , HellaSwag , ARC-Challenge , WinoGrande , TruthfulQA , MMLU
- 5 Jan 2026LFM2.5-1.2B-Instruct Liquid AI Introducing LFM2.5: The Next Generation of On-Device AI (Liquid AI blog)
- GPQA Diamond 38.9
- MMLU-Pro 44.4
- IFEval 86.2
- IFBench 47.3
- Multi-IF 61
- AIME 2025 14
- Berkeley Function Calling Leaderboard (BFCL) 49.1
Dropped since the last post: RealWorldQA , MM-IFEval , InfographicVQA , OCRBench , BLINK , MMStar , MMMU , MathVista , SEED-Bench , MM-Vet , MME , MMLU
2025
- 31 Dec 2025
- KMMLU 73
- KMMLU-Pro 64
- CLIcK 78.9
- HAE-RAE Bench 73.3
- KoBALT 44.3
- KBank-MMLU (Upstage) 65.5
- KBL 65.5
- KorMedMCQA 84.4
- Ko-AIME 2025 (Upstage) 80
- HRM8K 87.6
- MMLU 88.2
- MMLU-Pro 80.4
- GPQA Diamond 68.1
- Humanity's Last Exam 10.5
- AIME 2024 91.7
- AIME 2025 84.3
- HMMT 2025 73.3
- HMMT Nov 2025 80
- LiveCodeBench v6 74.2
- IFBench 53.7
- IFEval 88
- Arena-Hard v2.0 74.8
- WritingBench 7.51
- tau2-bench 55.8
- AA-LCR (Artificial Analysis Long Context Reasoning) 35
Its technical report adds 3 evaluations 28 results in all
Solar Open Technical ReportCapabilities
- Ko-AIME 2024 (Upstage) 80.3
- Ko-IFEval (Upstage) 87.5
- Ko-Arena-Hard v2 (Upstage) 79.9
- 31 Dec 2025
- MMLU-Pro 83.8
- GPQA Diamond 79.1
- Humanity's Last Exam 13.6
- IMO-AnswerBench 76.3
- AIME 2025 92.8
- HMMT Nov 2025 86.8
- LiveCodeBench Pro 25.9
- LiveCodeBench v6 80.7
- Terminal-Bench 2.0 29
- SWE-bench Verified 49.4
- tau2-bench 70.8
- BrowseComp 31.4
- IFBench 67.3
- IFEval 89.7
- AA-LCR (Artificial Analysis Long Context Reasoning) 53.5
- OpenAI MRCR v2 52.3
- KMMLU-Pro 67.3
- KoBALT 61.8
- CLIcK 83.9
- HRM8K 90.9
- Ko-LongBench (LG) 86.8
- MMMLU 85.7
- WMT24++ 90.5
- WildJailbreak 89.9
- KGC-Safety (LG) 96.1
Dropped since the last post: MMLU-Redux , HMMT 2025 , LiveCodeBench v5 , Multi-IF , Berkeley Function Calling Leaderboard (BFCL) , tau-bench , KMMLU-Redux , HRM8K-KSM , HELMET , RULER , LongBench
Its technical report adds 2 evaluations 2 new 11 dropped 26 results in all
K-EXAONE Technical ReportCapabilities
- CodeUtilityBench (LG) 71.9
- OpenAI MRCR 52.3
Dropped since the lab’s previous technical report: MMLU-Redux , HMMT 2025 , LiveCodeBench v5 , Multi-IF , Berkeley Function Calling Leaderboard (BFCL) , tau-bench , KMMLU-Redux , HRM8K-KSM , HELMET , RULER , LongBench
- 23 Dec 2025
- SWE-bench Verified 74
- Multi-SWE-bench 49.4
- SWE-bench Multilingual 72.5
- Terminal-Bench 2.0 47.9
- SWT-bench 69.3
- SWE-Perf 3.1
- SWE-Review 8.9
- OctoCodingbench 26.1
- VIBE (MiniMax) 88.6
- Toolathlon 43.5
- BrowseComp 62
- AIME 2025 83
- MMLU-Pro 88
- GPQA Diamond 83
- Humanity's Last Exam 22.2
- LiveCodeBench 81
- SciCode 41
- IFBench 70
- AA-LCR (Artificial Analysis Long Context Reasoning) 62
- tau2-bench 87
Dropped since the last post: Terminal-Bench , ArtifactsBench , BrowseComp-ZH , GAIA , xbench-DeepSearch , FinSearchComp , TheAgentCompany , Terminal-Bench Hard , Artificial Analysis Intelligence Index v3
- 22 Dec 2025
- MMLU-Pro 84.3
- GPQA Diamond 85.7
- Humanity's Last Exam 42.8
- AIME 2025 95.7
- HMMT 2025 97.1
- HMMT Nov 2025 93.5
- IMO-AnswerBench 82
- LiveCodeBench v6 84.9
- SWE-bench Verified 73.8
- SWE-bench Multilingual 66.7
- Terminal-Bench Hard 33.3
- Terminal-Bench 2.0 41
- BrowseComp 67.5
- BrowseComp-ZH 66.6
- tau2-bench 87.4
Dropped since the last post: MMBench-EN v1.1 , MMBench-CN v1.1 , MMStar , BLINK , MuirBench , MMMU , MMMU-Pro , Video-MMMU , MathVista , AI2D , DynaMath , We-Math , ZeroBench , MM-BrowseComp , Design2Code , Flame-React-Eval , OSWorld , AndroidWorld , WebVoyager , WebQuest , MMLongBench-Doc , MMLongBench , LVBench , OCRBench , OCRBench v2 , ChartQA Pro , ChartMuseum , CharXiv , OmniSpatial , RefCOCO / RefCOCO+ / RefCOCOg , TreeBench , Ref-L4
- 19 Dec 2025
- PutnamBench 87.9
- FATE-H 80
- FATE-X 33
- IMO 2025 35
- Putnam 2025 11
- CombiBench 48
Dropped since the last post: AIME 2025 , HMMT 2025 , Beyond AIME , AMO-Bench , IMO-AnswerBench , AetherCode , LiveCodeBench v6 , LiveCodeBench Pro , GPQA Diamond , PHYBench , BIOBench (ByteDance Seed) , KOR-Bench , ARC-AGI , Inverse IFEval , MARS-Bench , MultiChallenge , COLLIE , EIFBench , MMLU , MMLU-Pro , SuperGPQA , LPFQA , BrowseComp , BrowseComp-ZH , GAIA , WideSearch , Humanity's Last Exam , MM-BrowseComp , HLE-VL , SWE-bench Verified , Multi-SWE-bench , AInstein-SWE-Bench , Terminal-Bench 2.0 , U-Artifacts , BFCL V4 , tau2-bench , DeepConsult , DeepResearch Bench , DiscoX , FinSearchComp , XpertBench , WorldTravel , OSWorld , Online-Mind2Web , AndroidWorld , MMMU , MMMU-Pro , MathVista , MATH-Vision (MathVision) , DynaMath , LogicVista , ZeroBench , VPCT , VLMsAreBiased , VLMs Are Blind , SimpleVQA , HallusionBench , MMStar , MMBench-EN v1.1 , MuirBench , MMVP , ScreenSpot-Pro , CountBench , BLINK , RefSpatial-Bench , ERQA , CVBench , AI2D , CharXiv , DUDE , MMLongBench , Video-MMMU , MMVU , Video-Holmes , MINERVA , Video SimpleQA , TVBench , TempCompass , TOMATO , EgoTempo , MotionBench , Video-MME , LongVideoBench , LVBench , OVBench , OVO-Bench , StreamingBench , OmniDocBench v1.5 , Scientists' First Exam (SFE) , EMMA , MME-CC , Point-Bench , MMSI-Bench , VCR-Bench , VideoReasonBench , CG-Bench , LiveSports-3K , ViSpeak-Bench , OmniMMI , Education (ByteDance Seed in-house) , Customer Support Q&A (ByteDance Seed in-house) , Information Processing (ByteDance Seed in-house) , Intention Recognition (ByteDance Seed in-house) , Information Extraction (ByteDance Seed in-house) , Complex Workflow (ByteDance Seed in-house)
- 18 Dec 2025
Dropped since the last post: GDPval , Investment Banking Modeling Tasks (OpenAI) , BrowseComp , GPQA Diamond , Humanity's Last Exam , HMMT 2025 , AIME 2025 , ARC-AGI , ARC-AGI-2
Its system card addendum adds 16 evaluations 2 new 17 results in all
Addendum to GPT-5.2 System Card: GPT-5.2-CodexSafety
Robustness
- StrongREJECT 93.3
Alignment
Bio and chem risk
Cyber risk
- CVE-Bench 87
- Cyber Range 72.7
- Atomic Challenges
- CyScenarioBench 0
AI R&D risk
- OpenAI PRs 55
- MLE-bench 10
- PaperBench 43
- OpenAI-Proof Q&A 8
- 18 Dec 2025
Seed1.8 ByteDance Seed Official Release of Seed1.8: A Generalized Agentic Model (blog and model card) - AIME 2025 94.3
- HMMT 2025 89.7
- Beyond AIME 77
- AMO-Bench 60
- IMO-AnswerBench 76.3
- AetherCode 38.2
- LiveCodeBench v6 79.5
- LiveCodeBench Pro 1930
- GPQA Diamond 83.8
- PHYBench 41
- BIOBench (ByteDance Seed) 42.3
- KOR-Bench 76.2
- ARC-AGI 67.9
- Inverse IFEval 80.3
- MARS-Bench 70.1
- MultiChallenge 66.7
- COLLIE 72.6
- EIFBench 48.6
- MMLU 92.3
- MMLU-Pro 84.9
- SuperGPQA 64.8
- LPFQA 49.1
- BrowseComp 67.6
- BrowseComp-ZH 81.3
- GAIA 87.4
- WideSearch 63.8
- Humanity's Last Exam 40.9
- MM-BrowseComp 46.3
- HLE-VL 31.5
- SWE-bench Verified 72.9
- Multi-SWE-bench 42
- AInstein-SWE-Bench 36.7
- Terminal-Bench 2.0 45.2
- U-Artifacts 49.2
- BFCL V4 57.2
- tau2-bench 72
- DeepConsult 41
- DeepResearch Bench 43.5
- DiscoX 74.6
- FinSearchComp 62.8
- XpertBench 51.3
- WorldTravel 47.2
- OSWorld 61.9
- Online-Mind2Web 85.9
- AndroidWorld 70.7
- MMMU 83.4
- MMMU-Pro 73.2
- MathVista 87.7
- MATH-Vision (MathVision) 81.3
- DynaMath 61.5
- LogicVista 78.3
- ZeroBench 11
- VPCT 61
- VLMsAreBiased 62
- VLMs Are Blind 93
- SimpleVQA 65.4
- HallusionBench 63.9
- MMStar 79.9
- MMBench-EN v1.1 91.6
- MuirBench 78.7
- MMVP 86
- ScreenSpot-Pro 73.1
- CountBench 96.3
- BLINK 74.3
- RefSpatial-Bench 56.3
- ERQA 58.8
- CVBench 88
- AI2D 89.1
- CharXiv 71.4
- DUDE 69.4
- MMLongBench 77.7
- Video-MMMU 82.7
- MMVU 73.1
- Video-Holmes 65.5
- MINERVA 62.4
- Video SimpleQA 67.8
- TVBench 71.5
- TempCompass 86.9
- TOMATO 60.8
- EgoTempo 67
- MotionBench 70.6
- Video-MME 87.8
- LongVideoBench 77.4
- LVBench 73
- OVBench 65.1
- OVO-Bench 72.6
- StreamingBench 84.4
- OmniDocBench v1.5 0.106
- Scientists' First Exam (SFE) 51.2
- EMMA 60.9
- MME-CC 43.4
- Point-Bench 76.5
- MMSI-Bench 25.8
- VCR-Bench 59.8
- VideoReasonBench 52.8
- CG-Bench 62.4
- LiveSports-3K 77.5
- ViSpeak-Bench 79
- OmniMMI 53
- Education (ByteDance Seed in-house) 60.8
- Customer Support Q&A (ByteDance Seed in-house) 69
- Information Processing (ByteDance Seed in-house) 53.9
- Intention Recognition (ByteDance Seed in-house) 63.2
- Information Extraction (ByteDance Seed in-house) 43.7
- Complex Workflow (ByteDance Seed in-house) 54.6
Dropped since the last post: Windows Agent Arena , Terminal-Bench , UI-TARS 15 Games Collection
Its model card adds 3 evaluations 2 new 74 dropped 108 results in all
Seed1.8 Model Card: Towards Generalized Real-World AgencyCapabilities
- ZeroVideo 6.9
Safety
Dropped since the lab’s previous model card: IMO 2025 , CMO 2025 , SimpleQA Verified , HealthBench , HealthBench Hard , Encyclo-K , AIME 2026 , HMMT Nov 2025 , MathArena (Apex, IMO, USAMO) , Codeforces rating , SUPERChem , BABE , FrontierScience-Research , FrontierScience-Olympiad , ARC-AGI-2 , ProcBench , MRCR v2 , GraphWalks , LongBench v2 , FRAMES , DeR2 Bench , CL-bench , Global PIQA , MMMLU , LongFact-Objects , LongFact-Concepts , FActScore , PutnamBench , Chinese Complex Instruction Following (ByteDance Seed in-house) , Math Kangaroo 2025 (MathArena) , MathCanvas-Bench , HiPhO , XLRS-Bench , PhyX , VisuLogic , VisFactor , RealWorldQA , BabyVision , MTVQA-Test , WorldVQA , Vibe-Eval , ViVerBench , TreeBench , All-Angles Bench , ChartQA Pro , OCRBench v2 , CharXiv (DQ) , LongDocURL , MMLongBench-Doc , MORSE-500 , ContPhy , VideoEval-Pro , CrossVid , ODV-Bench , SWE-Lancer , SWE-Bench Pro , SWE-bench Multilingual , SciCode , SWE-EVO , Aider Polyglot , ArtifactsBench , CodeSimpleQA , SpreadsheetBench Verified , HLE-Verified , DeepSearchQA , Seal-0 , MCPMark , VITA-Bench , ResearchRubrics , MineDojo , NL2Repo-Bench , GDPVal-Diamond , Compositional Tasks (ByteDance Seed in-house) , Text Classification (ByteDance Seed in-house)
- 17 Dec 2025
- Humanity's Last Exam 43.5
- ARC-AGI-2 33.6
- GPQA Diamond 90.4
- AIME 2025 99.7
- MMMU-Pro 81.2
- ScreenSpot-Pro 69.1
- CharXiv 80.3
- OmniDocBench v1.5 0.121
- Video-MMMU 86.9
- LiveCodeBench Pro 2316
- Terminal-Bench 2.0 47.6
- SWE-bench Verified 78
- tau2-bench 90.2
- Toolathlon 49.4
- MCP Atlas 57.4
- Vending-Bench 2 3635
- FACTS Benchmark Suite 61.9
- SimpleQA Verified 68.7
- MMMLU 91.8
- Global PIQA 92.8
- MRCR v2 67.2
Dropped since the last post: MathArena (Apex, IMO, USAMO)
- 16 Dec 2025
- RoboCasa 66.2
- RoboCasa GR1 Tabletop Tasks 47.6
- SimplerEnv 62.1
- LIBERO
- BEHAVIOR Challenge 2025
Dropped since the last post: MATH-500 , AIME 2024 , AIME 2025 , GPQA Diamond , LiveCodeBench v5 , Berkeley Function Calling Leaderboard (BFCL) , IFEval , Arena-Hard-Auto , Humanity's Last Exam , MMLU-Pro
- 16 Dec 2025
- MMLU-Pro 84.9
- GPQA Diamond 83.7
- Humanity's Last Exam 22.1
- AIME 2025 94.1
- HMMT 2025 84.4
- LiveCodeBench v6 80.6
- Arena-Hard v2.0 54.1
- LongBench v2 60.6
- MRCR v2 45.7
- SWE-bench Verified 73.4
- SWE-bench Multilingual 71.7
- Terminal-Bench Hard 30.5
- Terminal-Bench 2.0 38.5
- BrowseComp 58.3
- tau2-bench 80.3
Its technical report adds 0 evaluations 15 results in all
- 15 Dec 2025
- MMLU-Pro 78.3
- AIME 2025 89.1
- GPQA Diamond 73
- LiveCodeBench v6 68.3
- SciCode 33.3
- Humanity's Last Exam 10.6
- miniF2F 50
- Terminal-Bench Hard 8.5
- SWE-bench Verified 38.8
- tau2-bench 49
- BFCL V4 53.8
- IFBench 71.5
- MultiChallenge 38.5
- Arena-Hard v2.0 67.7
- AA-LCR (Artificial Analysis Long Context Reasoning) 35.9
- RULER 86.3
- MMLU-ProX 59.5
- WMT24++ 86.2
Dropped since the last post: MMMU , MathVista , AI2D , OCRBench v2 , OCRBench , OCR-Reasoning , ChartQA , DocVQA , Video-MME
Its technical report adds 0 evaluations 49 dropped 18 results in all
Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic ReasoningDropped since the lab’s previous technical report: MMBench-EN v1.1 , MMBench-CN v1.1 , MMStar , BLINK , MuirBench , HallusionBench , ZeroBench , CRPE , POPE , MME-RealWorld , RealWorldQA , MMT-Bench , R-Bench (real-world robustness) , WildVision , MMMU , MMMU-Pro , MathVista , MATH-Vision (MathVision) , MathVerse , DynaMath , LogicVista , We-Math , MMLongBench-Doc , OCRBench , OCRBench v2 , ChartQA , RD-TableBench , AI2D , TextVQA , DocVQA , InfographicVQA , OCR-Reasoning , VCR (Visual Caption Restoration) , SEED-Bench-2-Plus , CharXiv , TreeBench , CVBench , ScreenSpot , ScreenSpot-v2 , ScreenSpot-Pro , LongVideoBench , MLVU , Video-MME , MTVQA-Test , MMMB , Multilingual MMBench , MATH-500 , LiveCodeBench , IFEval
- 12 Dec 2025
- MATH 96.2
- AIME 2024 80.6
- AIME 2025 78.1
- OMEGA 53.4
- BIG-Bench Hard 88.6
- ZebraLogic 80.1
- AGIEval 89.2
- HumanEval+ 91.5
- MBPP+ 68.3
- LiveCodeBench 83.3
- IFEval 93.8
- IFBench 68.1
- MMLU 86.4
- PopQA 30.9
- GPQA Diamond 57.5
- AlpacaEval 2.0 69.1
- Tulu 3 Safety Eval 83.6
Its technical report adds 10 evaluations 10 new 6 dropped 27 results in all
Olmo 3Safety
- DoAnythingNow (DAN) 54.7
- HarmBench 89.7
- TrustLLM JailbreakTrigger 86.4
- WildJailbreak
- WildGuardTest 96.9
- XSTest 91.8
- StrongREJECT 90.5
- ToxiGen 100
Bias
Bio and chem risk
- WMDP 32.7
Dropped since the lab’s previous technical report: DROP , GSM8K , TruthfulQA , DeepMind Mathematics , IFEval-OOD , MMLU-Pro
- 11 Dec 2025
- GDPval 74.1
- Investment Banking Modeling Tasks (OpenAI) 71.7
- BrowseComp 77.9
- GPQA Diamond 93.2
- Humanity's Last Exam 50
- HMMT 2025 100
- AIME 2025 100
- ARC-AGI 90.5
- ARC-AGI-2 54.2
Dropped since the last post: SWE-Bench Pro , SWE-bench Verified , CharXiv , FrontierMath , FrontierMath Tier 4 , OpenAI MRCR v2 , ScreenSpot-Pro , tau2-bench
- 11 Dec 2025
- GDPval 70.9
- SWE-Bench Pro 55.6
- SWE-bench Verified 80
- GPQA Diamond 92.4
- CharXiv 88.7
- AIME 2025 100
- FrontierMath 40.3
- FrontierMath Tier 4 14.6
- ARC-AGI 86.2
- ARC-AGI-2 52.9
- OpenAI MRCR v2 77
- ScreenSpot-Pro 86.3
- tau2-bench 90.3
Dropped since the last post: SWE-Lancer , Terminal-Bench 2.0
Its system card addendum adds 29 evaluations 14 new 29 results in all
Update to GPT-5 System Card: GPT-5.2Capabilities
Health
- HealthBench 63.3
- HealthBench Hard 42
- HealthBench Consensus 94.5
Hallucination
Safety
Robustness
- StrongREJECT 97.5
- Agent JSK 97.8
- PlugInject 99.6
Bias
Alignment
Bio and chem risk
Cyber risk
AI R&D risk
- OpenAI PRs 55
- MLE-bench 16
- PaperBench 39
- OpenAI-Proof Q&A 3
- 9 Dec 2025
- Putnam 2025 72.5
- 9 Dec 2025
- SWE-bench Verified 68
- SWE-bench Multilingual 55.7
- Terminal-Bench 2.0 22.5
Dropped since the last post: AIME 2025 , AIME 2024 , GPQA Diamond , LiveCodeBench
- 9 Dec 2025
- SWE-bench Verified 72.2
- SWE-bench Multilingual 61.3
- Terminal-Bench 2.0 32.6
Dropped since the last post: MMMLU , GPQA Diamond , SimpleQA , AMC , LiveCodeBench , LMArena (Elo)
- 8 Dec 2025
- MMBench-EN v1.1 88.8
- MMBench-CN v1.1 88.2
- MMStar 75.9
- BLINK 65.5
- MuirBench 77.1
- MMMU 76
- MMMU-Pro 66
- Video-MMMU 74.7
- MathVista 85.2
- AI2D 88.8
- DynaMath 54.5
- We-Math 69.8
- ZeroBench 25.8
- MM-BrowseComp 7.6
- Design2Code 88.6
- Flame-React-Eval 86.3
- OSWorld 37.2
- AndroidWorld 57
- WebVoyager 81
- WebQuest 69.3
- MMLongBench-Doc 54.9
- MMLongBench 64.1
- LVBench 59.5
- OCRBench 86.5
- OCRBench v2 65.1
- ChartQA Pro 65.5
- ChartMuseum 58.4
- CharXiv 63.2
- OmniSpatial 52
- RefCOCO / RefCOCO+ / RefCOCOg 88.6
- TreeBench 51.4
- Ref-L4 88.9
Dropped since the last post: AIME 2025 , GPQA Diamond , LiveCodeBench v6 , Humanity's Last Exam , BrowseComp , SWE-bench Verified , Terminal-Bench , tau2-bench
Its technical report adds 11 evaluations 1 dropped 39 results in all
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement LearningCapabilities
- HallusionBench 62.3
- MATH-Vision (MathVision) 63.5
- MathVerse 75.4
- LogicVista 62.1
- CVBench 87.6
- ERQA 47.8
- All-Angles Bench 56.2
- Video-MME 81.8
- MMVU 68.4
- MotionBench 63.6
- MVBench 74.9
Dropped since the lab’s previous technical report: GeoBench
- 5 Dec 2025
- IMO-AnswerBench 73.4
- HMMT 2025 85.8
- Humanity's Last Exam 18.8
- ARC-AGI 52.9
- MultiChallenge 62.6
- Terminal-Bench 34.6
- SWE-bench Verified 53
- tau2-bench 72.4
Dropped since the last post: MMLU-Pro , GPQA Diamond , Chinese SimpleQA , DROP , ZebraLogic , MATH-500 , AIME 2024 , LiveCodeBench , C-Eval , CMMLU , Arena-Hard-Auto , CFBench , CELLO , T-Eval
- 3 Dec 2025
- AIME 2024 71.9
- AIME 2025 69.3
- BIG-Bench Hard 86.4
- DROP 83.5
- GPQA Diamond 65.5
- IFEval 77.9
- MATH-500 93.8
- MMLU 87.7
- MMLU-Pro 80.7
- MuSR 69.7
- OpenBookQA 96.6
- SimpleQA 6
- RefusalBench (Nous Research) 74.6
Dropped since the last post: LiveCodeBench v6 , Arena-Hard-Auto , RewardBench , EQ-Bench 3 , Creative Writing v3
- 2 Dec 2025
- AIME 2025 85
- AIME 2024 89.8
- GPQA Diamond 71.2
- LiveCodeBench 64.6
Dropped since the last post: LiveCodeBench v5
Its technical report adds 3 evaluations 3 new 1 dropped 6 results in all
Ministral 3Capabilities
- HMMT 2025 67.5
- PHYBench 26
- LiveCodeBench v6 64.6
Dropped since the lab’s previous technical report: SWE-bench Verified
- 2 Dec 2025
Nova 2 Omni Preview Amazon Amazon Nova 2: Multimodal Reasoning and Generation Models (Technical Report and Model Card) - MMLU-Pro 80.7
- AIME 2025 92.1
- IFBench 68.7
- MultiChallenge 75.5
- tau2-bench 80
- tau2-bench Verified 73.5
- BFCL V4 58.3
- MMMU-Pro 61.4
- OCRBench v2 58.2
- RealKIE-FCC Verified 59.8
- RefCOCO / RefCOCO+ / RefCOCOg 86.3
- Video-MME 77.9
- MAVERIX 66.6
- QVHighlights 76.7
- ScreenSpot 85.4
- Multilingual LibriSpeech 4.7
- Common Voice 5.7
- CoVoST2 40.7
- MMAU 75.3
Dropped since the last post: GPQA Diamond , LongCodeBench , MCP Atlas , SWE-bench Verified , Terminal-Bench , LiveCodeBench v5
Its technical report adds 4 evaluations 1 new 7 dropped 23 results in all
Amazon Nova 2: Multimodal Reasoning and Generation ModelsCapabilities
- ImgEdit-Bench 4.12
Safety
- Aegis 2.0 80.7
- WildGuardTest 84.1
- Jigsaw Toxic Comment Classification 57.8
Dropped since the lab’s previous technical report: GPQA Diamond , LongCodeBench , OpenAI MRCR , MCP Atlas , SWE-bench Verified , Terminal-Bench , LiveCodeBench v5
- 2 Dec 2025 Nova 2 Lite Amazon Amazon Nova 2: Multimodal Reasoning and Generation Models (Technical Report and Model Card)
- MMLU-Pro 80.9
- GPQA Diamond 79.6
- AIME 2025 91
- IFBench 70.8
- MultiChallenge 76.6
- LongCodeBench 84
- tau2-bench 76
- tau2-bench Verified 70.7
- BFCL V4 60.3
- MCP Atlas 24.6
- MMMU-Pro 61.8
- OCRBench v2 56.1
- RealKIE-FCC Verified 62.1
- QVHighlights 77.2
- ScreenSpot 83.3
- SWE-bench Verified 64.5
- Terminal-Bench 32.5
- LiveCodeBench v5 71
Dropped since the last post: Common Voice
Its technical report adds 4 evaluations 1 dropped 22 results in all
Amazon Nova 2: Multimodal Reasoning and Generation ModelsCapabilities
Safety
- Aegis 2.0 85.8
- WildGuardTest 84.7
- Jigsaw Toxic Comment Classification 56.5
Dropped since the lab’s previous technical report: Common Voice
- 2 Dec 2025 Nova 2.0 Pro Preview Amazon Amazon Nova 2: Multimodal Reasoning and Generation Models (Technical Report and Model Card)
- MMLU-Pro 81.6
- GPQA Diamond 81.4
- AIME 2025 92.3
- IFBench 80.2
- MultiChallenge 77.7
- LongCodeBench 84
- tau2-bench 92.7
- tau2-bench Verified 71.5
- BFCL V4 61.6
- MMMU-Pro 63.5
- OCRBench v2 64.5
- RealKIE-FCC Verified 67
- QVHighlights 76.7
- ScreenSpot 88.1
- Common Voice 5.5
- SWE-bench Verified 70
- Terminal-Bench 41.3
- LiveCodeBench v5 74.6
Dropped since the last post: MMLU , MATH-500 , BigCodeBench-Hard , MBXP , IFEval , MMMU , CharXiv , EgoSchema , TallyQA , SimpleQA , Berkeley Function Calling Leaderboard (BFCL)
Its technical report adds 4 evaluations 4 new 12 dropped 22 results in all
Amazon Nova 2: Multimodal Reasoning and Generation ModelsCapabilities
Safety
- Aegis 2.0 81.4
- WildGuardTest 84.9
- Jigsaw Toxic Comment Classification 54.7
Dropped since the lab’s previous technical report: MMLU , MATH-500 , BigCodeBench-Hard , MBXP , IFEval , MMMU , CharXiv , CharXiv (DQ) , EgoSchema , TallyQA , SimpleQA , Berkeley Function Calling Leaderboard (BFCL)
- 2 Dec 2025
- MMMLU 85.5
- GPQA Diamond 43.9
- SimpleQA 23.8
- AMC 52
- LiveCodeBench 34.4
- LMArena (Elo) 1418
Dropped since the last post: AIME 2024 , AIME 2025 , LiveCodeBench v5
- 1 Dec 2025
- SimpleQA 8.9
- MuSR 63.5
- MMLU 85
- MATH-500 92.1
- GPQA Diamond 58.5
- Berkeley Function Calling Leaderboard (BFCL) 59.7
Dropped since the last post: IFEval , AGIEval , MixEval , TriviaQA
- 1 Dec 2025
- MMLU-Pro 85
- GPQA Diamond 82.4
- Humanity's Last Exam 40.8
- LiveCodeBench 83.3
- Codeforces rating 2386
- AIME 2025 93.1
- HMMT 2025 92.5
- HMMT Nov 2025 90.2
- IMO-AnswerBench 78.3
- Terminal-Bench 2.0 46.4
- SWE-bench Verified 73.1
- SWE-bench Multilingual 70.2
- BrowseComp 67.6
- BrowseComp-ZH 65
- tau2-bench 80.3
- MCP-Universe 45.9
- MCPMark 38
- Toolathlon 35.2
Dropped since the last post: Aider Polyglot , SimpleQA , Terminal-Bench
DeepSeek-V3.2-Speciale: Its technical report adds 4 evaluations 4 new 8 dropped 12 results in all
DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (arXiv 2512.02556)Capabilities
- IMO 2025 35
- CMO 2025 102
- IOI 2025 492
- ICPC World Finals 2025 10
Dropped since the lab’s previous technical report: MMLU-Pro , BrowseComp , BrowseComp-ZH , SimpleQA , Aider Polyglot , SWE-bench Verified , SWE-bench Multilingual , Terminal-Bench
Its technical report adds 0 evaluations 4 dropped 18 results in all
DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (arXiv 2512.02556)Dropped since the lab’s previous technical report: IMO 2025 , CMO 2025 , IOI 2025 , ICPC World Finals 2025
- 27 Nov 2025
DeepSeekMath-V2 DeepSeek DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning (model card) - IMO-ProofBench 61.9
- IMO 2025 83.3
- Putnam 2024 98.3
Dropped since the last post: miniF2F , PutnamBench , ProverBench (DeepSeek)
Its technical report adds 3 evaluations 3 new 6 dropped 6 results in all
DeepSeekMath-V2: Towards Self-Verifiable Mathematical ReasoningCapabilities
Dropped since the lab’s previous technical report: miniF2F , ProofNet , PutnamBench , CombiBench , FormalMATH , ProverBench (DeepSeek)
- 26 Nov 2025
- MATH-500 98.1
- AIME 2024 90.8
- AIME 2025 88
- GPQA Diamond 74.4
- Humanity's Last Exam 14.6
- MMLU-Pro 81.9
Dropped since the last post: LiveCodeBench v5 , IFEval
Its technical report adds 1 evaluation 1 new 2 dropped 6 results in all
INTELLECT-3: Technical ReportCapabilities
- LiveCodeBench v6 69.3
Dropped since the lab’s previous technical report: LiveCodeBench v5 , IFEval
- 24 Nov 2025
- WebVoyager 73.5
- Online-Mind2Web 34.1
- DeepShop 26.2
- WebTailBench 38.4
Dropped since the last post: AIME 2024 , AIME 2025 , MATH-500 , GPQA Diamond
Its technical report adds 4 evaluations 4 new 4 dropped 8 results in all
Fara-7B: An Efficient Agentic Model for Computer UseCapabilities
- ScreenSpot-v2 89.3
- ScreenSpot 86.7
Safety
- AgentHarm 94.2
- WebTailBench-Refusals 81.9
Dropped since the lab’s previous technical report: AIME 2024 , AIME 2025 , MATH-500 , GPQA Diamond
- 24 Nov 2025Claude Opus 4.5 Anthropic Introducing Claude Opus 4.5 (launch post); System Card: Claude Opus 4.5, Table 2.3.A
- SWE-bench Verified 80.9
- Terminal-Bench 2.0 59.3
- tau2-bench 93.6
- MCP Atlas 62.3
- OSWorld-Verified 66.3
- ARC-AGI-2 37.6
- GPQA Diamond 87
- MMMU 80.7
- MMMLU 90.8
Dropped since the last post: Terminal-Bench , AIME 2025 , Vals Finance Agent
Its system card adds 68 evaluations 23 new 11 dropped 77 results in all
System Card: Claude Opus 4.5Capabilities
- SWE-Bench Pro 51.6
- SWE-bench Multilingual 76.2
- BrowseComp-Plus 67.6
- Multi-agent search (Anthropic internal benchmark) 74.8
- ARC-AGI 80
- Vending-Bench 2 4967.1
- Vals Finance Agent 55.2
- CyberGym 50.6
- SpreadsheetBench 64.3
- Humanity's Last Exam 30.8
- AIME 2025 92.8
- LAB-Bench FigQA 54.9
- WebArena 65.3
Hallucination
- 100Q-Hard (Anthropic) 45.9
- SimpleQA Verified 44.1
- AA-Omniscience 51.9
- False-premise questions (Anthropic) 1.2
Safety
Robustness
Alignment
- Anthropic automated behavioral audit
- Petri
- Sycophancy on user-provided prompts (Anthropic) 10
- Agentic misalignment suite (Anthropic)
- Self-preference bias (Anthropic)
- Internal codebase sabotage propensity (Anthropic) 0
- Reward-hack-prone coding tasks (Anthropic) 0
- Impossible tasks (Anthropic) 55
- Training distribution hack rate (Anthropic) 1
- SHADE-Arena 7.7
- Subversion Strategy Eval
- UK AISI alignment and monitorability evaluations
Bio and chem risk
- Long-form virology tasks
- Virology Capabilities Test (VCT) 47.7
- DNA Synthesis Screening Evasion
- LAB-Bench ProtocolQA 90.7
- LAB-Bench SeqQA 85.2
- LAB-Bench CloningScenarios 75.8
- Creative biology 52.4
- Short-horizon computational biology tasks
- Bioinformatics evaluations (Anthropic) 73.7
- ASL-4 virology uplift trial (Anthropic) 66.8
Cyber risk
AI R&D risk
- SWE-bench Verified (hard subset) 46.7
- Kernels task (Anthropic) 252.4
- Time series forecasting task (Anthropic)
- Text-based reinforcement learning task (Anthropic) 0.975
- LLM training task (Anthropic) 16.5
- Quadruped reinforcement learning task (Anthropic)
- Novel compiler task (Anthropic)
- Internal AI research evaluation suite 2 0.604
- Internal model use survey (Anthropic) 50
Dropped since the lab’s previous system card: Honesty human feedback win rate (Anthropic) , SimpleQA , MCP prompt injection evaluation (Anthropic) , Computer use prompt injection evaluation (Anthropic) , Tool use prompt injection evaluation (Anthropic) , Cyber-harness network challenges (Incalmo cyber ranges) , Average in-eval reward hacking rate (Anthropic) , Apollo Research strategic deception evaluations , Cooperation with user deception (Anthropic) , Sycophancy prompts (Anthropic) , Research sandbagging (Anthropic)
- 20 Nov 2025
- MATH 96.1
- AIME 2024 76.8
- AIME 2025 72.5
- OMEGA 50.8
- BIG-Bench Hard 89.8
- ZebraLogic 76
- AGIEval 88.2
- HumanEval+ 91.4
- MBPP+ 68
- LiveCodeBench 83.5
- IFEval 89
- IFBench 47.6
- MMLU 85.4
- PopQA 31.9
- GPQA Diamond 58.1
- AlpacaEval 2.0 74.2
- Tulu 3 Safety Eval 68.8
Dropped since the last post: DROP , GSM8K , TruthfulQA
Its technical report adds 10 evaluations 27 results in all
Olmo 3Safety
- DoAnythingNow (DAN) 20.2
- HarmBench 73.5
- TrustLLM JailbreakTrigger 73.3
- WildJailbreak
- WildGuardTest 89.4
- XSTest 93.9
- StrongREJECT 80.8
- ToxiGen 100
Bias
Bio and chem risk
- WMDP 34.8
- 19 Nov 2025
- SWE-bench Verified 77.9
- SWE-Lancer 79.9
- Terminal-Bench 2.0 58.1
Dropped since the last post: GPQA Diamond , AIME 2025 , FrontierMath , MMMU , tau2-bench , BrowseComp Long Context 128k
Its system card adds 20 evaluations 4 new 38 dropped 21 results in all
GPT-5.1-Codex-Max System CardSafety
Robustness
- StrongREJECT 96.7
- Coding prompt injections 100
Alignment
Bio and chem risk
Cyber risk
AI R&D risk
- PaperBench 40
- MLE-bench 17
- OpenAI PRs 53
- OpenAI-Proof Q&A 8
Autonomy
Dropped since the lab’s previous system card: Standard Refusal Evaluation , Sycophancy evaluation (offline) , Instruction Hierarchy Evaluation , Browsing prompt injections , Tool-calling prompt injections , Gray Swan Agent Red Teaming (ART) benchmark , Hallucinations on production-representative prompts , LongFact-Concepts , LongFact-Objects , FActScore , SimpleQA , SimpleQA Hallucination Rate , Coding Deception , Broken Search Tool , CharXiv Missing Image , AbstentionBench , Deception in production traffic , HealthBench , HealthBench Hard , HealthBench Consensus , HealthBench Hard Hallucinations , HealthBench Consensus Urgent , HealthBench Consensus Global Health , MMMLU , BBQ (Bias Benchmark for QA) , Expert Red Teaming for Violent Attack Planning , Virology Capabilities Test (VCT) , Virology Capabilities Test (text-only) , Human Pathogen Capabilities Test , Molecular Biology Capabilities Test , World-Class Bio , Challenging prompts from red teamers with biosafety-relevant PhDs , Filtered, adversarial sample of production prompts , Pairwise Bioweaponization Campaign , Capture the Flag (Collegiate) , Pattern Labs cyberoffensive challenges , SWE-bench Verified , Apollo Research strategic deception evaluations
- 19 Nov 2025
- SWE-bench Verified 42
- GPQA Diamond 77.7
- MMLU-Pro 84.7
- MMMLU 86.2
- Humanity's Last Exam 16.7
- SimpleQA Verified 30.9
- AIME 2025 89.5
- MATH-500 98.6
Dropped since the last post: MMLU , GSM8K , MATH , MGSM , SimpleQA
- 19 Nov 2025
- Research-Eval 63.9
- FRAMES 87.6
- X Browse (xAI) 56.3
- tau2-bench
- BFCL V4
Dropped since the last post: LMArena (Elo) , EQ-Bench 3 , Creative Writing v3 , FActScore
- 18 Nov 2025
- Humanity's Last Exam 45.8
- ARC-AGI-2 31.1
- GPQA Diamond 91.9
- AIME 2025 100
- MathArena (Apex, IMO, USAMO) 23.4
- MMMU-Pro 81
- ScreenSpot-Pro 72.7
- CharXiv 81.4
- OmniDocBench v1.5 0.115
- Video-MMMU 87.6
- LiveCodeBench Pro 2439
- Terminal-Bench 2.0 54.2
- SWE-bench Verified 76.2
- tau2-bench 85.4
- Vending-Bench 2 5478.2
- FACTS Benchmark Suite 70.5
- SimpleQA Verified 72.1
- MMMLU 91.8
- Global PIQA 93.4
- MRCR v2 77
Dropped since the last post: Online-Mind2Web , WebVoyager , AndroidWorld
Its system card adds 17 evaluations 17 results in all
Frontier Safety Framework Report - Gemini 3 ProAlignment
- GDM Stealth 25
- GDM Situational Awareness 27.3
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- 17 Nov 2025
- LMArena (Elo) 1483
- EQ-Bench 3 1586
- Creative Writing v3 1721.9
- FActScore 2.97
Dropped since the last post: GPQA Diamond , AIME 2025 , HMMT 2025 , Humanity's Last Exam , LiveCodeBench , BrowseComp , SimpleQA , Research-Eval , BrowseComp-ZH , xbench-DeepSearch , X Browse (xAI) , LMArena Search Arena
Its model card adds 15 evaluations 3 new 1 dropped 15 results in all
Grok 4.1 Model CardSafety
Robustness
Alignment
Bio and chem risk
Cyber risk
- Cybench 39
Persuasion
Dropped since the lab’s previous model card: Soft Bias (Internal)
- 13 Nov 2025
- SWE-bench Verified 76.3
- GPQA Diamond 88.1
- AIME 2025 94
- FrontierMath 26.7
- MMMU 85.4
- tau2-bench 80.2
- BrowseComp Long Context 128k 90
Dropped since the last post: Code refactoring tasks (OpenAI)
Its system card addendum adds 3 evaluations 3 results in all
- 11 Nov 2025 ERNIE-4.5-VL-28B-A3B-Thinking Baidu Introducing ERNIE-4.5-VL-28B-A3B-Thinking (Hugging Face model card and ERNIE Blog)
- MathVista 82.5
- MathVerse 81
- MATH-Vision (MathVision) 61.9
- MMMU 72.2
- VLMs Are Blind 77.3
- VisuLogic 31.1
- VisualPuzzles 52.5
- ChartQA 87.1
- AI2D 95.3
- DocVQA 93.6
- OCRBench 85.8
- CharXiv 56.9
- SimpleVQA 53.1
- HallusionBench 60.4
- MMStar 71.1
- BLINK 62.8
- CVBench 83.8
- CountBench 89
- TempCompass 75.1
Dropped since the last post: AIME 2025 , Berkeley Function Calling Leaderboard (BFCL) , ZebraLogic , MuSR , BIG-Bench Hard , HumanEval+ , MBPP , IFEval , Multi-IF , Chinese SimpleQA , WritingBench
- 6 Nov 2025
- Humanity's Last Exam 44.9
- AIME 2025 94.5
- HMMT 2025 89.4
- IMO-AnswerBench 78.6
- GPQA Diamond 84.5
- MMLU-Pro 84.6
- MMLU-Redux 94.4
- Longform Writing 73.8
- HealthBench 58
- BrowseComp 60.2
- BrowseComp-ZH 62.3
- Seal-0 56.3
- FinSearchComp 47.4
- FRAMES 87
- SWE-bench Verified 71.3
- SWE-bench Multilingual 61.1
- Multi-SWE-bench 41.9
- SciCode 44.8
- LiveCodeBench 83.1
- OJBench 48.7
- Terminal-Bench 47.1
Dropped since the last post: SWE-Dev
- 31 Oct 2025
- OmniBench 61.4
- WorldSense (omnimodal) 60.9
- Daily-Omni 82.4
- UNO-Bench 49.9
- MMBench 87.5
- RealWorldQA 74.8
- MMStar 70.9
- MathVista 77.9
- MMMU 70.7
- MM-Vet 69
- BLINK 63.1
- MuirBench 77.1
- Mantis-Eval 84.8
- ChartQA 87.6
- DocVQA 91.8
- OCRBench 84.9
- OmniDocBench 22.8
- RefCOCO / RefCOCO+ / RefCOCOg 93.9
- CountBench 92.4
- VisualWebBench 78.7
- ScreenSpot-v2 91.2
- AndroidControl 91.2
- MVBench 75.2
- NExT-QA 86.2
- TempCompass 82.2
- Video-MME 78.2
- LongVideoBench 69.3
- MMVU 67.1
- Video-MMMU 67.5
- LibriSpeech 2.79
- AISHELL-1 0.63
- AISHELL-2 2.78
- FLEURS 3.99
- Common Voice 4.98
- WenetSpeech 6.39
- CoVoST2 37.3
- MMAU 75.9
- Clotho-AQA 72.8
- OpenAudioBench 80.1
- VoiceBench 88.7
- MMLU 90.3
- MMLU-Pro 82.7
- C-Eval 91.7
- CMMLU 89.4
- IFEval 82.4
- COLLIE 45.7
- Meeseeks 39
- MATH-500 97.6
- AIME 2024 72.9
- Beyond AIME 47.4
- GPQA Diamond 74.4
- DROP 83.5
- ZebraLogic 86
- GraphWalks 56
- LiveCodeBench 52.6
- HumanEval+ 90.8
- MBPP+ 80.2
Dropped since the last post: MMLU-Redux , Arena-Hard v2.0 , HMMT 2025 , AIME 2025 , Sudoku-Bench , ARC-AGI , OJBench , SWE-bench Verified , Berkeley Function Calling Leaderboard (BFCL) , tau2-bench , VITA-Bench , miniF2F , LongCat safety evaluation (Meituan)
Its technical report adds 2 evaluations 1 new 13 dropped 58 results in all
LongCat-Flash-Omni Technical ReportCapabilities
Dropped since the lab’s previous technical report: MMLU-Redux , Arena-Hard v2.0 , HMMT 2025 , AIME 2025 , Sudoku-Bench , ARC-AGI , OJBench , SWE-bench Verified , Berkeley Function Calling Leaderboard (BFCL) , tau2-bench , VITA-Bench , miniF2F , LongCat safety evaluation (Meituan)
- 29 Oct 2025
- SWE-Bench Pro 40.1
Dropped since the last post: Conversational SWE Task Benchmark (Windsurf) , End-To-End SWE Task Benchmark (Windsurf)
- 29 Oct 2025
- 28 Oct 2025
- MMLU 59.4
- MMLU-Pro 34
- BIG-Bench Hard 60.4
- AGIEval 49.2
- GPQA Diamond 29.9
- IFEval 77.4
- GSM8K 76.3
- GSM-Symbolic 72.3
- MATH 45.3
- DeepMind Mathematics 34
- HumanEval 74
- HumanEval+ 69
- MBPP 65
- MBPP+ 57
- CRUXEval 33.1
- BigCodeBench 30.2
- Berkeley Function Calling Leaderboard (BFCL) 54.8
- MultiPL-E 32.2
- MMMLU 45
- INCLUDE 42.1
- MGSM 37.8
- SALAD-Bench 93.4
- AttaQ 85.3
- 28 Oct 2025
- MMMU 68
- MathVista 76.9
- AI2D 87.1
- OCRBench v2 62
- OCRBench 85.6
- OCR-Reasoning 36.4
- ChartQA 89.7
- DocVQA 94.4
- Video-MME 65.9
Dropped since the last post: AIME 2025 , MATH-500 , GPQA Diamond , LiveCodeBench , Berkeley Function Calling Leaderboard (BFCL) , IFEval , Humanity's Last Exam , RULER
Its technical report adds 45 evaluations 41 new 3 dropped 54 results in all
NVIDIA Nemotron Nano V2 VLCapabilities
- MMBench-EN v1.1 82.6
- MMBench-CN v1.1 76.3
- MMStar 71.7
- BLINK 56.7
- MuirBench 44.2
- HallusionBench 73.1
- ZeroBench 18.3
- CRPE 64.3
- POPE 87
- MME-RealWorld 62.2
- RealWorldQA 72.2
- MMT-Bench 63.8
- R-Bench (real-world robustness) 69.7
- WildVision 20.2
- MMMU-Pro 28
- MATH-Vision (MathVision) 53.6
- MathVerse 58.2
- DynaMath 32.3
- LogicVista 58.4
- We-Math 39.3
- MMLongBench-Doc 32.1
- RD-TableBench 57.4
- TextVQA 76.1
- InfographicVQA 80.4
- VCR (Visual Caption Restoration) 71.8
- SEED-Bench-2-Plus 70
- CharXiv 41.3
- TreeBench 42.5
- CVBench 78.3
- ScreenSpot 40.1
- ScreenSpot-v2 42.8
- ScreenSpot-Pro 5.5
- LongVideoBench 57
- MLVU 73.6
- MTVQA-Test 24.3
- MMMB
- Multilingual MMBench
- MATH-500 96.9
- AIME 2025 71.3
- GPQA Diamond 64.1
- LiveCodeBench 69.4
- IFEval 78.2
- SciCode 17.6
- MMLU-Pro 77.1
- RULER 72.1
Dropped since the lab’s previous technical report: IFBench , AIME 2024 , Berkeley Function Calling Leaderboard (BFCL)
- 27 Oct 2025
- SWE-bench Verified 69.4
- Multi-SWE-bench 36.2
- SWE-bench Multilingual 56.5
- Terminal-Bench 46.3
- ArtifactsBench 66.8
- BrowseComp 44
- BrowseComp-ZH 48.5
- GAIA 75.7
- xbench-DeepSearch 72
- Humanity's Last Exam 31.8
- tau2-bench 87
- FinSearchComp 65.5
- TheAgentCompany 36
- AIME 2025 78
- MMLU-Pro 82
- GPQA Diamond 78
- LiveCodeBench 83
- SciCode 36
- IFBench 72
- AA-LCR (Artificial Analysis Long Context Reasoning) 61
- Terminal-Bench Hard 24
- Artificial Analysis Intelligence Index v3 61
Dropped since the last post: AIME 2024 , MATH-500 , FullStackBench , ZebraLogic , OpenAI MRCR , LongBench v2 , tau-bench , SimpleQA , MultiChallenge
Its technical report adds 5 evaluations 14 dropped 11 results in all
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World IntelligenceCapabilities
- VIBE-Pro (MiniMax) 42.4
- WideSearch 62.3
- Toolathlon 18.8
- GDPval-AA 34
AI R&D risk
- MLE-bench 40
Dropped since the lab’s previous technical report: SWE-Bench Pro , NL2Repo-Bench , Terminal-Bench 2.0 , HyperTask , RISE , MM-ClawBench , MEWC v2 , Finance Modeling Pro , AIME 2026 , SciCode , IFBench , Humanity's Last Exam , MMLU-Pro , Artificial Analysis Intelligence Index v4.0
- 15 Oct 2025
- SWE-bench Verified 73.3
- Terminal-Bench 41
- tau2-bench 76.6
- OSWorld-Verified 50.7
- AIME 2025 96.3
- GPQA Diamond 73
- MMMLU 83
- MMMU 73.2
Its system card adds 25 evaluations 25 results in all
System Card: Claude Haiku 4.5Safety
Robustness
Alignment
Bio and chem risk
Cyber risk
- Cybench 46.9
AI R&D risk
- 10 Oct 2025
- 10 Oct 2025
- AIME 2025 93.4
- Codeforces rating 2088
- ARC-AGI 55.9
- Arena-Hard v2.0 81.6
- HMMT 2025 86.7
- LiveCodeBench v6 78.3
- HealthBench 57.9
- Creative Writing v3 85.4
- ICPC World Finals 2025 5
Its technical report adds 19 evaluations 27 results in all
Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking ModelCapabilities
- Omni-MATH 82.6
- CNMO 2024 88.5
- FinanceReasoning 87.4
- UGMathBench 76.5
- Aider-Edit 78.6
- BIG-Bench Extra Hard 59.6
- ZebraLogic 95.2
- Humanity's Last Exam 16
- GPQA Diamond 78.6
- MMLU-Pro 80.5
- C-Eval 91.5
- PHYBench 42.6
- AGIEval 88.1
- TriviaQA 78.6
- CMMLU 89.1
- IFEval 85.2
- MultiChallenge 50.9
- Berkeley Function Calling Leaderboard (BFCL) 68.8
- IMO 2025
- 7 Oct 2025
- Online-Mind2Web 69
- WebVoyager 88.9
- AndroidWorld 69.7
Dropped since the last post: SWE-bench Verified , Artificial Analysis Intelligence Index v3 , Humanity's Last Exam , GPQA Diamond , AIME 2025 , LiveCodeBench v5 , SimpleQA , SimpleQA Verified , FACTS Grounding , MMMU , Vibe-Eval , MRCR v2 , Global MMLU Lite
Its model card adds 0 evaluations 17 dropped 3 results in all
Gemini 2.5 Computer Use Model CardDropped since the lab’s previous model card: Humanity's Last Exam , GPQA Diamond , AIME 2025 , LiveCodeBench v5 , SWE-bench Verified , SimpleQA , SimpleQA Verified , FACTS Grounding , MMMU , Vibe-Eval , MRCR v2 , Global MMLU Lite , Text to Text Safety , Multilingual Safety , Image to Text Safety , Tone , Instruction Following (safety evaluation)
- 7 Oct 2025
- MMLU 64.8
- MMLU-Pro 37.4
- GPQA Diamond 29.3
- IFEval 77.6
- IFBench 25.9
- Multi-IF 58.2
- GSM8K 84.4
- GSM-Plus 64.8
- MATH-500 74.2
- MATH Level 5 62.4
- MGSM 72.4
- MMMLU 55.3
- LiveCodeBench v6 21
- LiveCodeBench v5 21.4
- HumanEval+ 69.5
- Creative Writing v3 44.2
Its technical report adds 0 evaluations 12 results in all
- 2 Oct 2025
- MMLU 78.4
- MMLU-Pro 55.5
- BIG-Bench Hard 81.6
- AGIEval 70.6
- GPQA Diamond 40.6
- AlpacaEval 2.0 42.5
- IFEval 87.5
- Arena-Hard-Auto 46.5
- GSM8K 87.3
- GSM-Symbolic 87.4
- MATH 74
- DeepMind Mathematics 59.3
- HumanEval 88
- HumanEval+ 83
- MBPP 84
- MBPP+ 71
- CRUXEval 50.3
- BigCodeBench 46.2
- Berkeley Function Calling Leaderboard (BFCL) 64.7
- MultiPL-E 57.4
- MMMLU 69.7
- INCLUDE 64
- MGSM 38.7
- SALAD-Bench 97.3
- AttaQ 86.6
Dropped since the last post: PopQA , TruthfulQA , DROP , AIME 2024 , MATH-500
- 30 Sep 2025GLM-4.6 Z.ai GLM-4.6 launch blog (Z.ai) and Hugging Face card, chart 'LLM Performance Evaluation: Agentic, Reasoning and Coding'
- AIME 2025 93.9
- GPQA Diamond 81
- LiveCodeBench v6 82.8
- Humanity's Last Exam 30.4
- BrowseComp 45.1
- SWE-bench Verified 68
- Terminal-Bench 40.5
- tau2-bench 75.9
Dropped since the last post: MMBench-EN v1.1 , MMBench-CN v1.1 , MMStar , BLINK , MuirBench , HallusionBench , ZeroBench , GeoBench , MMMU , MMMU-Pro , MathVista , MATH-Vision (MathVision) , MathVerse , DynaMath , LogicVista , AI2D , We-Math , MMLongBench-Doc , OCRBench , ChartQA Pro , ChartMuseum , RefCOCO / RefCOCO+ / RefCOCOg , TreeBench , Ref-L4 , OmniSpatial , CVBench , ERQA , All-Angles Bench , OSWorld , AndroidWorld , WebVoyager , WebQuest , Design2Code , Flame-React-Eval , Video-MME , MMVU , Video-MMMU , LVBench , MotionBench , MVBench
- 29 Sep 2025
- MMLU-Pro 85
- GPQA Diamond 79.9
- Humanity's Last Exam 19.8
- LiveCodeBench 74.1
- AIME 2025 89.3
- HMMT 2025 83.6
- Codeforces rating 2121
- Aider Polyglot 74.5
- BrowseComp 40.1
- BrowseComp-ZH 47.9
- SimpleQA 97.1
- SWE-bench Verified 67.8
- SWE-bench Multilingual 57.9
- Terminal-Bench 37.7
Its technical report adds 0 evaluations 23 dropped 14 results in all
DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse AttentionDropped since the lab’s previous technical report: MMLU , MMLU-Redux , DROP , IFEval , FRAMES , AlpacaEval 2.0 , Arena-Hard-Auto , AIME 2024 , MATH-500 , CNMO 2024 , CLUEWSC , C-Eval , Chinese SimpleQA , AMC 12 2024 , SimpleSafetyTests , BBQ (Bias Benchmark for QA) , Anthropic Red Team , XSTest , Do-Not-Answer , HarmBench , DeepSeek in-house safety benchmark , DeepSeek multilingual safety evaluation , DeepSeek jailbreak robustness evaluation
- 29 Sep 2025
- SWE-bench Verified 82
- Terminal-Bench 50
- tau2-bench 84.7
- OSWorld-Verified 61.4
- AIME 2025 100
- GPQA Diamond 83.4
- MMMLU 89.1
- MMMU 77.8
- Vals Finance Agent 55.3
Dropped since the last post: tau-bench
Its system card adds 56 evaluations 22 new 8 dropped 56 results in all
System Card: Claude Sonnet 4.5Hallucination
Safety
Robustness
Alignment
- Average in-eval reward hacking rate (Anthropic) 15.4
- Reward-hack-prone coding tasks (Anthropic) 1
- Impossible tasks (Anthropic) 53
- Training distribution hack rate (Anthropic) 2
- Anthropic automated behavioral audit
- Petri
- UK AISI alignment and monitorability evaluations
- Apollo Research strategic deception evaluations 0
- Cooperation with user deception (Anthropic)
- Agentic misalignment suite (Anthropic) 0
- Sycophancy prompts (Anthropic)
- Self-preference bias (Anthropic)
- Subversion Strategy Eval
- Research sandbagging (Anthropic)
- SHADE-Arena 1.5
Bio and chem risk
Cyber risk
AI R&D risk
- SWE-bench Verified (hard subset) 45.3
- Kernels task (Anthropic) 108.6
- Time series forecasting task (Anthropic) 5.3
- Text-based reinforcement learning task (Anthropic) 0.85
- LLM training task (Anthropic) 5.5
- Quadruped reinforcement learning task (Anthropic) 1.3
- Novel compiler task (Anthropic)
- Internal AI research evaluation suite 2 0.514
- Internal model use survey (Anthropic) 0
Dropped since the lab’s previous system card: Ambiguous context evaluations (Anthropic) , Child safety evaluations (Anthropic) , StrongREJECT , Malicious computer use (Anthropic) , SycophancyEval (Sharma et al.) , Bioweapons acquisition uplift trial (Anthropic) , Bioweapons knowledge questions , METR data deduplication
- 25 Sep 2025 Gemini 2.5 Flash-Lite Google DeepMind Gemini 2.5 Flash-Lite Model Card, Preview (09-2025) columns (Google DeepMind)
- Humanity's Last Exam 7.3
- GPQA Diamond 71.7
- AIME 2025 48.2
- LiveCodeBench v5 58.4
- SWE-bench Verified 38.9
- SimpleQA Verified 9.6
- FACTS Grounding 87.5
- MMMU 72
- Vibe-Eval 59.8
- MRCR v2 25.6
- Global MMLU Lite 84.9
Dropped since the last post: IFEval , BIG-Bench Hard , HellaSwag , PIQA , ARC-Challenge , WinoGrande
- 25 Sep 2025 Gemini 2.5 Flash Google DeepMind Continuing to bring you our latest models, with an improved Gemini 2.5 Flash and Flash-Lite release (Google Developers Blog)
- SWE-bench Verified 54
- Artificial Analysis Intelligence Index v3
- Humanity's Last Exam 13.2
- GPQA Diamond 80.8
- AIME 2025 75.6
- LiveCodeBench v5 71.7
- SimpleQA 25.3
- SimpleQA Verified 27.8
- FACTS Grounding 87.5
- MMMU 80.3
- Vibe-Eval 62
- MRCR v2 52.4
- Global MMLU Lite 87.9
Dropped since the last post: IMO 2025 , LiveCodeBench v6
Its model card adds 5 evaluations 3 dropped 17 results in all
Gemini 2.5 Flash & 2.5 Flash Image Model CardSafety
Dropped since the lab’s previous model card: AIME 2024 , Aider Polyglot , MRCR (Michelangelo, original)
- 24 Sep 2025 Code World Model (CWM) Meta Code World Model (CWM) README and model card (GitHub, facebookresearch/cwm)
- LiveCodeBench v5 68.6
- LiveCodeBench v6 63.5
- MATH-500 96.6
- AIME 2024 76
- AIME 2025 68.2
- SWE-bench Verified 65.8
Dropped since the last post: MVPBench , TempCompass
Its technical report adds 12 evaluations 6 new 59 dropped 18 results in all
CWM: An Open-Weights LLM for Research on Code Generation with World ModelsCapabilities
- Aider Polyglot 35.1
- Terminal-Bench 26.3
- CRUXEval 94.3
- HaltEval-prelim (Meta) 94
- BigO(Bench)
- MMLU 77.7
- MMLU-Pro 60.2
- GPQA Diamond 40.6
- GSM8K 93.3
- HumanEval+ 75
- MBPP 73.4
- RULER
Dropped since the lab’s previous technical report: IFEval , HumanEval , MultiPL-E , MATH , ARC-Challenge , Berkeley Function Calling Leaderboard (BFCL) , Nexus , API-Bank , Gorilla APIBench , ZeroSCROLLS/QuALITY , Qasper , SQuALITY , InfiniteBench/En.MC , InfiniteBench/En.QA , Needle In A Haystack , MGSM , Multilingual MMLU (Meta) , LSAT , SAT Reading , SAT Math , GMAT Quantitative , GMAT Verbal , GRE Physics , AP Art History , AP Biology , AP Calculus , AP Chemistry , AP English Language and Composition , AP English Literature and Composition , AP Environmental Science , AP Macroeconomics , AP Microeconomics , AP Physics , AP Psychology , AP Statistics , AP US Government , AP US History , AP World History , GRE Quantitative , GRE Verbal , SQuAD v2 , QuAC , RACE , CommonSenseQA , PIQA , SIQA , OpenBookQA , WinoGrande , DROP , WorldSense , AGIEval , BIG-Bench Hard , QuALITY , Adversarial SQuAD , Dynabench SQuAD , GSM-Plus , PAWS , Llama verbatim memorization probe (Meta) , CyberSecEval Prompt Injection
Its technical report adds 14 evaluations 14 new 18 dropped 14 results in all
Code World Model Preparedness ReportAlignment
- MASK 44.8
Bio and chem risk
Cyber risk
Dropped since the lab’s previous technical report: SWE-bench Verified , Aider Polyglot , Terminal-Bench , CRUXEval , HaltEval-prelim (Meta) , BigO(Bench) , LiveCodeBench v5 , LiveCodeBench v6 , MATH-500 , AIME 2024 , AIME 2025 , MMLU , MMLU-Pro , GPQA Diamond , GSM8K , HumanEval+ , MBPP , RULER
- 24 Sep 2025
- SWE-bench Verified 69.6
- tau2-bench 74.8
- AIME 2025 100
- HMMT 2025 100
- SuperGPQA
- LiveCodeBench v6
- GPQA Diamond
- LMArena (Elo)
Dropped since the last post: MMMU , MMMU-Pro , MathVista , MATH-Vision (MathVision) , MathVerse , ZeroBench , VisuLogic , MMBench-EN v1.1 , RealWorldQA , MMStar , SimpleVQA , HallusionBench , MM-MT-Bench , MIA-Bench , MMLongBench-Doc , DocVQA , InfographicVQA , AI2D , OCRBench , OCRBench v2 , CC-OCR , CharXiv , RefCOCO / RefCOCO+ / RefCOCOg , CountBench , ODinW-13 , ARKitScenes , Hypersim , SUNRGBD , Objectron , BLINK , MuirBench , ERQA , EmbSpatialBench , RefSpatial-Bench , RoboSpatial-Home , Video-MME , MLVU , LVBench , Charades-STA , Video-MMMU , ScreenSpot , ScreenSpot-Pro , OSWorld-G , OSWorld , Design2Code , MMLU , MMLU-Pro , MMLU-Redux , SimpleQA , ZebraLogic , LiveBench , Humanity's Last Exam , SIFo , IFEval , Creative Writing v3 , WritingBench , Berkeley Function Calling Leaderboard (BFCL) , Multi-IF , MMLU-ProX , INCLUDE
- 23 Sep 2025
- MMMU 80.6
- MMMU-Pro 69.3
- MathVista 85.8
- MATH-Vision (MathVision) 74.6
- MathVerse 85
- ZeroBench 27.7
- VisuLogic 34.4
- MMBench-EN v1.1 90.6
- RealWorldQA 81.3
- MMStar 78.7
- SimpleVQA 61.3
- HallusionBench 66.7
- MM-MT-Bench 8.5
- MIA-Bench 92.7
- MMLongBench-Doc 56.2
- DocVQA 96.5
- InfographicVQA 89.5
- AI2D 89.2
- OCRBench 87.5
- OCRBench v2 66.8
- CC-OCR 81.5
- CharXiv 66.1
- RefCOCO / RefCOCO+ / RefCOCOg 92.4
- CountBench 93.7
- ODinW-13 43.2
- ARKitScenes 53.7
- Hypersim 11
- SUNRGBD 34.9
- Objectron 71.2
- BLINK 67.1
- MuirBench 80.1
- ERQA 52.5
- EmbSpatialBench 84.3
- RefSpatial-Bench 69.9
- RoboSpatial-Home 73.9
- Video-MME 79
- MLVU 83.8
- LVBench 63.6
- Charades-STA 63.5
- Video-MMMU 80
- ScreenSpot 95.4
- ScreenSpot-Pro 61.8
- OSWorld-G 68.3
- OSWorld 38.1
- Design2Code 93.4
- MMLU 90.6
- MMLU-Pro 83.8
- MMLU-Redux 93.7
- SuperGPQA 64.3
- SimpleQA 44.4
- AIME 2025 89.7
- HMMT 2025 77.4
- LiveCodeBench v6 70.1
- ZebraLogic 97.3
- LiveBench 79.6
- Humanity's Last Exam 13.6
- SIFo 77.3
- IFEval 88.2
- Creative Writing v3 85.7
- WritingBench 86.7
- Berkeley Function Calling Leaderboard (BFCL) 71.9
- Multi-IF 79.1
- MMLU-ProX 80.6
- INCLUDE 80
Dropped since the last post: GPQA Diamond , Codeforces rating , OJBench , Arena-Hard v2.0 , tau-bench , tau2-bench , PolyMath
Its technical report adds 25 evaluations 22 new 12 dropped 82 results in all
Qwen3-VL Technical ReportCapabilities
- MathVision-Wildphoto 63.8
- We-Math 74.8
- DynaMath 82.8
- Math-VR 66.8
- VLMs Are Blind 79.5
- LogicVista 72.2
- VisualPuzzles 57.2
- MMBench-CN v1.1 88.6
- ChartQA 90.3
- OmniDocBench
- CharXiv (DQ) 90.5
- VSI-Bench 60
- MVBench 75.2
- MMVU 71.1
- V* 85.9
- HR-Bench
- ChartMimic 78.4
- AndroidWorld 62
- Windows Agent Arena 32.1
- GPQA Diamond 77.1
- Codeforces rating 1964
- OJBench 27.5
- Arena-Hard v2.0 74.8
- tau2-bench
- PolyMath 57.8
Dropped since the lab’s previous technical report: C-Eval , Arena-Hard-Auto , AlignBench , MATH-500 , AIME 2024 , ZebraLogic , AutoLogi , LiveCodeBench v5 , MMMLU , MT-AIME2024 , MLogiQA , RULER
- 22 Sep 2025
- MMLU-Redux 86.6
- GPQA Diamond 69.6
- AIME 2025 65
- ZebraLogic 76
- MultiPL-E 81.4
- IFEval 81
- Creative Writing v3 80.6
- WritingBench 82.6
- Berkeley Function Calling Leaderboard (BFCL) 64.4
- Multi-IF 64
- PolyMath 37.9
- LiveBench 71.8
- Arena-Hard v2.0 55.1
- LibriSpeech 1.85
- Common Voice 5.18
- FLEURS 5.33
- VoiceBench 85.5
- MMAU 77.5
- MuChoMusic 52
- MMStar 68.5
- HallusionBench 59.7
- MM-MT-Bench 7.4
- MMMU 69.1
- MMMU-Pro 57
- MathVista 75.9
- MATH-Vision (MathVision) 56.3
- AI2D 85.2
- ChartQA 86.8
- CountBench 90
- Video-MME 70.5
- LVBench 50.2
- MLVU 75.2
- WorldSense (omnimodal) 54
- Daily-Omni 75.8
- Video-Holmes 57.3
- Seed-TTS-eval 1.39
- WenetSpeech 5.29
Dropped since the last post: MMLU-Pro , SuperGPQA , HMMT 2025 , LiveCodeBench v6 , Codeforces rating , OJBench , tau-bench , tau2-bench , MMLU-ProX , INCLUDE
Its technical report adds 3 evaluations 3 new 23 dropped 40 results in all
Qwen3-Omni Technical ReportCapabilities
Dropped since the lab’s previous technical report: MMLU-Pro , MATH , GSM8K , HumanEval , MBPP , LiveCodeBench , CoVoST2 , Qwen in-house voice-chat benchmark , MMBench-EN v1.1 , MM-Vet , MME , MuirBench , CRPE , RealWorldQA , MME-RealWorld , TextVQA , DocVQA , OCRBench v2 , RefCOCO / RefCOCO+ / RefCOCOg , ODinW-13 , MVBench , EgoSchema , OmniBench
- 22 Sep 2025
- MMLU-Pro 85
- GPQA Diamond 80.7
- Humanity's Last Exam 21.7
- LiveCodeBench 74.9
- Codeforces rating 2046
- Aider Polyglot 76.1
- BrowseComp 38.5
- BrowseComp-ZH 45
- SimpleQA 96.8
- SWE-bench Verified 68.4
- SWE-bench Multilingual 57.8
- Terminal-Bench 36.7
Dropped since the last post: MMLU-Redux , AIME 2024 , AIME 2025 , HMMT 2025
- 21 Sep 2025
- MMLU-Pro 82.6
- MMLU-Redux 89.3
- IFEval 86.9
- Arena-Hard v2.0 69.9
- MATH-500 99.2
- HMMT 2025 83.7
- AIME 2024 93.3
- AIME 2025 90.6
- Beyond AIME 69.5
- GPQA Diamond 81.5
- ZebraLogic 95.5
- Sudoku-Bench 56
- ARC-AGI 50.3
- LiveCodeBench 79.4
- OJBench 40.7
- SWE-bench Verified 59.4
- Berkeley Function Calling Leaderboard (BFCL) 74.4
- tau2-bench 74
- VITA-Bench 29.5
- miniF2F 81.6
- LongCat safety evaluation (Meituan) 95.7
Dropped since the last post: MMLU , C-Eval , CMMLU , COLLIE , Meeseeks , DROP , GraphWalks , HumanEval+ , MBPP+ , Terminal-Bench , AceBench
Its technical report adds 1 evaluation 11 dropped 21 results in all
Introducing LongCat-Flash-Thinking: A Technical ReportCapabilities
- LiveCodeBench v6 79.4
Dropped since the lab’s previous technical report: MMLU , C-Eval , CMMLU , COLLIE , Meeseeks , DROP , GraphWalks , HumanEval+ , MBPP+ , Terminal-Bench , AceBench
- 19 Sep 2025
- GPQA Diamond 85.7
- AIME 2025 92
- HMMT 2025 93.3
- Humanity's Last Exam 20
- LiveCodeBench 80
- BrowseComp 44.9
- SimpleQA 95
- Research-Eval 66
- BrowseComp-ZH 51.2
- xbench-DeepSearch 74
- X Browse (xAI) 58
- LMArena Search Arena 1163
Dropped since the last post: USAMO 2025 , ARC-AGI-2 , Vending-Bench
Its model card adds 13 evaluations 13 results in all
Grok 4 Fast Model CardSafety
Robustness
Bias
- Soft Bias (Internal) 0.79
Alignment
Bio and chem risk
Cyber risk
- Cybench 30
Persuasion
- MakeMeSay 12
- 18 Sep 2025
- MMAU-Pro 56.8
- MMAU 74.9
- Big Bench Audio 72.9
Dropped since the last post: GPQA Diamond , SuperGPQA , DROP , MMLU-Pro , IFEval , MATH-500 , AIME 2024 , AIME 2025 , LiveCodeBench v5 , LiveCodeBench v6
Its technical report adds 7 evaluations 7 new 11 dropped 10 results in all
MiMo-Audio: Audio Language Models are Few-Shot LearnersCapabilities
- MMAR 63.6
- MMSU 62.9
- MultiChallenge Audio (Xiaomi) 15.2
- Seed-TTS-eval
- InstructTTSEval
- LibriSpeech 3.76
- AISHELL-1 1.78
Dropped since the lab’s previous technical report: GPQA Diamond , SuperGPQA , DROP , MMLU-Pro , IFEval , MATH-500 , AIME 2024 , AIME 2025 , LiveCodeBench v5 , LiveCodeBench v6 , AlignBench
- 17 Sep 2025
- AIME 2024 86.1
- AIME 2025 77.3
- GPQA Diamond 70.1
- LiveCodeBench v5 70.9
- 17 Sep 2025 Magistral Medium 1.2 Mistral AI Magistral Small 1.2 (Magistral-Small-2509) model card (Hugging Face)
- AIME 2024 91.8
- AIME 2025 83.5
- GPQA Diamond 76.3
- LiveCodeBench v5 75
- 15 Sep 2025
Dropped since the last post: Big Bench Audio , MultiChallenge Audio (OpenAI) , ComplexFuncBench Audio (OpenAI)
Its system card addendum adds 4 evaluations 1 new 4 results in all
- 11 Sep 2025
- MMLU-Pro 82.7
- MMLU-Redux 92.5
- GPQA Diamond 77.2
- SuperGPQA 60.8
- AIME 2025 87.8
- HMMT 2025 73.9
- LiveBench 76.6
- LiveCodeBench v6 68.7
- Codeforces rating 2071
- OJBench 29.7
- IFEval 88.9
- Arena-Hard v2.0 62.3
- WritingBench 84.6
- Berkeley Function Calling Leaderboard (BFCL) 72
- tau-bench 59.3
- tau2-bench 57.4
- Multi-IF 77.8
- MMLU-ProX 78.7
- INCLUDE 78.9
- PolyMath 56.3
Dropped since the last post: OmniBench , LibriSpeech , Common Voice , FLEURS , CoVoST2 , MMAU , VoiceBench , Seed-TTS-eval , MMMU , MMMU-Pro , MathVista , MATH-Vision (MathVision) , MMBench-EN v1.1 , MM-Vet , MMStar , MME , MuirBench , CRPE , RealWorldQA , MME-RealWorld , MM-MT-Bench , AI2D , TextVQA , DocVQA , ChartQA , OCRBench v2 , RefCOCO / RefCOCO+ / RefCOCOg , ODinW-13 , Video-MME , MVBench , EgoSchema , MATH , GSM8K , HumanEval , MBPP , MultiPL-E , LiveCodeBench , WenetSpeech
- 9 Sep 2025
- AIME 2025 78
- Berkeley Function Calling Leaderboard (BFCL) 65
- ZebraLogic 89.8
- MuSR 86.7
- BIG-Bench Hard 87.8
- HumanEval+ 90.8
- MBPP 80.2
- IFEval 84.3
- Multi-IF 63.3
- Chinese SimpleQA 49.1
- WritingBench 8.65
Dropped since the last post: MMLU , MMLU-Pro , C-Eval , CMMLU , LiveBench , SimpleQA , SysBench , MATH-500 , GSM8K , CMATH , OlympicArena , AIME 2024 , DROP , LiveCodeBench , MBPP+ , FullStackBench
- 8 Sep 2025π0.5 Physical Intelligence openpi LIBERO example results for π0.5, with the π0.5 open-weights release (GitHub)
- LIBERO 96.8
Its technical report adds 0 evaluations 1 results in all
- 5 Sep 2025
- SWE-bench Verified 69.2
- SWE-bench Multilingual 55.9
- Multi-SWE-bench 33.5
- Terminal-Bench 44.5
- SWE-Dev 66.6
Dropped since the last post: LiveCodeBench v6 , OJBench , MultiPL-E , Aider Polyglot , tau2-bench , AceBench , AIME 2024 , AIME 2025 , MATH-500 , HMMT 2025 , CNMO 2024 , PolyMath , ZebraLogic , AutoLogi , GPQA Diamond , SuperGPQA , Humanity's Last Exam , MMLU , MMLU-Redux , MMLU-Pro , IFEval , MultiChallenge , SimpleQA , LiveBench
- 2 Sep 2025
- HellaSwag 64
- WinoGrande 73.3
- XNLI 45.3
- XCOPA 69.8
- PIQA 81.9
Its technical report adds 29 evaluations 30 results in all
Apertus: Democratizing Open and Compliant LLMs for Global Language EnvironmentsCapabilities
- MMLU 69.6
- Global MMLU 62.7
- TruthfulQA 61.2
- HumanEval 73
- MBPP 47
- GSM8K 77.6
- MGSM 64.3
- MATH 30.8
- MathQA 33.9
- BIG-Bench Hard 64.2
- DROP 50.8
- ACPBench 62.9
- IFEval 75.2
- Multi-IF 74.7
- INCLUDE 58.2
- BLEnD 66.3
- CulturalBench 74.2
- SwitzerlandQA 67.2
- AGIEval 40.5
- ARC-Challenge 85
- GPQA Main 30.6
- GSM8K-Platinum 74.6
- MLogiQA 40.5
- RULER
Safety
- HarmBench 31.9
- RealToxicityPrompts 0.2
- LinguaSafe
Bias
- BBQ (Bias Benchmark for QA) 67.4
- ToxiGen 70.3
- 2 Sep 2025
- OSWorld 47.5
- Windows Agent Arena 50.6
- Terminal-Bench 45.3
- SWE-bench Verified 68.7
- AndroidWorld 73.3
- Online-Mind2Web 88.2
- BrowseComp-ZH 50.5
- BrowseComp 29.6
- UI-TARS 15 Games Collection 59.8
Dropped since the last post: IMO 2025 , miniF2F , PutnamBench , Past IMO (Seed-Prover formal set) , CombiBench , miniCTX-v2
- 1 Sep 2025
- FLORES-200 88.6
- WMT24++ 85.8
- Mandarin-Minority translation (Tencent) 60.8
Dropped since the last post: AIME 2024 , AIME 2025 , MATH , GPQA Diamond , OlympiadBench , LiveCodeBench , FullStackBench , ArtifactsBench , BIG-Bench Hard , DROP , ZebraLogic , IFEval , SysBench , LengthCtrl (Tencent) , InsCtrl (Tencent) , ComplexNLU (Tencent) , Word-Task (Tencent) , Berkeley Function Calling Leaderboard (BFCL) , tau-bench , ComplexFuncBench , C3-Bench
Its technical report adds 2 evaluations 2 new 26 dropped 5 results in all
Hunyuan-MT Technical ReportCapabilities
Dropped since the lab’s previous technical report: AIME 2024 , AIME 2025 , MATH , GPQA Diamond , OlympiadBench , LiveCodeBench , FullStackBench , ArtifactsBench , BIG-Bench Hard , DROP , ZebraLogic , IFEval , SysBench , LengthCtrl (Tencent) , InsCtrl (Tencent) , ComplexNLU (Tencent) , Word-Task (Tencent) , Berkeley Function Calling Leaderboard (BFCL) , tau-bench , ComplexFuncBench , C3-Bench , McEval , PenguinScrolls , LongBench v2 , FRAMES , RULER
- 29 Aug 2025
- MMLU 89.7
- MMLU-Pro 82.7
- Arena-Hard v2.0 86.5
- C-Eval 90.4
- CMMLU 84.3
- IFEval 89.7
- COLLIE 57.1
- Meeseeks 43
- MATH-500 96.4
- AIME 2024 70.4
- AIME 2025 61.3
- Beyond AIME 43
- GPQA Diamond 73.2
- DROP 79.1
- ZebraLogic 89.3
- GraphWalks 51
- LiveCodeBench 48
- HumanEval+ 88.4
- MBPP+ 79.6
- SWE-bench Verified 60.4
- Terminal-Bench 39.5
- tau2-bench 67.7
- AceBench 76.1
- VITA-Bench 24.3
- LongCat safety evaluation (Meituan) 87.7
Its technical report adds 1 evaluation 25 results in all
LongCat-Flash Technical ReportCapabilities
- 28 Aug 2025gpt-realtime OpenAI Introducing gpt-realtime and Realtime API updates for production voice agents (OpenAI)
Dropped since the last post: AIME 2025 , FrontierMath , HMMT 2025 , GPQA Diamond , Humanity's Last Exam , SWE-bench Verified , Aider Polyglot , MultiChallenge , BrowseComp , COLLIE , tau2-bench , MMMU , MMMU-Pro , Video-MMMU , CharXiv , ERQA , HealthBench , HealthBench Hard , HealthBench Hard Hallucinations , LongFact , FActScore
- 28 Aug 2025
- SWE-bench Verified 70.8
Dropped since the last post: AIME 2025 , AIME 2024 , GPQA Diamond , LiveCodeBench , MMLU-Pro , LOFT , SimpleQA , MMMU , EgoSchema
Its model card adds 10 evaluations 10 results in all
Grok Code Fast 1 Model CardSafety
Robustness
Alignment
- MASK 71.9
Bio and chem risk
- BioLP-bench 19.9
- Virology Capabilities Test (VCT) 28.7
- WMDP
Cyber risk
- Cybench 22.5
- 28 Aug 2025
- WMT24++ 83.8
Dropped since the last post: Berkeley Function Calling Leaderboard (BFCL) , tau-bench , m-tau-bench (Cohere) , DeepResearch Bench
Its technical report adds 2 evaluations 2 new 43 dropped 3 results in all
Command A Translate: Raising the Bar of Machine Translation with Difficulty FilteringCapabilities
- WMT25 -6.3
Dropped since the lab’s previous technical report: MMLU , MMLU-Pro , GPQA Diamond , IFEval , InFoBench , ChatRAG Bench , StrategyQA , Bamboogle , DROP , HotpotQA , Berkeley Function Calling Leaderboard (BFCL) , tau-bench , NTREX-128 , FLORES-200 , m-ArenaHard , m-tau-bench (Cohere) , Language Confusion Benchmark , AL-QASIDA , MBPP+ , LiveCodeBench v5 , BigCodeBench , LBPP (Less Basic Python Problems) , HumanEvalPack , HumanEval , HumanEval-COBOL (Cohere) , RepoQA , SWE-bench Verified , Aider Polyglot , Spider , Bird-SQL , Multi-dialect SQL evaluation (Cohere) , MATH , AIME 2024 , Default safety evaluation (Cohere) , XSTest , Safety mode controllability (Cohere) , Enterprise generative evaluation (Cohere) , Enterprise RAG evaluation (Cohere) , RULER , LongBench v2 , Cohere enterprise human evaluation , MGSM , INCLUDE
- 26 Aug 2025
- MATH-500 96.3
- AIME 2024 81.9
- AIME 2025 78.1
- GPQA Diamond 70.5
- BIG-Bench Hard 86.3
- LiveCodeBench v6 61.3
- MMLU 87.2
- MMLU-Pro 80.5
- SimpleQA 25.8
- IFEval 81.5
- Arena-Hard-Auto 94.4
- RefusalBench (Nous Research) 57.1
- RewardBench 73
- DROP 83.5
- MuSR 66.1
- OpenBookQA 94.2
- EQ-Bench 3 85.4
- Creative Writing v3 79.8
Its technical report adds 0 evaluations 18 results in all
- 25 Aug 2025
- MMStar 77.9
- MM-Vet 81.2
- MMBench-EN v1.1 87.4
- MTVQA-Test 39.3
- AI2D 87.3
- OCRBench 90.7
- WildVision 82.8
- MME-RealWorld 65.1
- HallusionBench 57.3
- MVBench 76.5
- Video-MME 72.9
- MLVU 78.2
- LongVideoBench 67.1
- MMMU 77.7
- MathVista 82.7
- MATH-Vision (MathVision) 63.9
- MathVerse 68.5
- DynaMath 46.5
- We-Math 62.3
- OlympiadBench 68.7
- LogicVista 66.7
- MATH-500 98
- AIME 2024 84.7
- AIME 2025 75.6
- GPQA Diamond 73.2
- MMLU-Pro 81.3
- C-Eval 90.9
- GAOKAO-Bench 94.5
- IFEval 83.7
- SGP-Bench 70.7
- ScreenSpot 89.8
- ScreenSpot-v2 92.9
- OSWorld-G 53.2
- VSI-Bench 69.5
- ERQA 46.8
- SpaCE-10 55
- OmniSpatial 51.9
Dropped since the last post: Scientists' First Exam (SFE) , PHYSICS , SMolInstruct , ChemBench , Matbench , MicroVQA , ProteinLMBench , MSEarth , XLRS-Bench
Its technical report adds 35 evaluations 35 new 9 dropped 72 results in all
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and EfficiencyCapabilities
- MME 2525.9
- MMMB
- ChartQA 88
- TextVQA 84.5
- DocVQA 94.9
- InfographicVQA 82
- SEED-Bench-2-Plus 71.9
- CharXiv 48.3
- VCR (Visual Caption Restoration) 97
- RealWorldQA 75.2
- BLINK 61.4
- Mantis-Eval 77
- MMIU 61.3
- MuirBench 47.5
- MMT-Bench 72.7
- MIRB 73
- RefCOCO / RefCOCO+ / RefCOCOg 92.4
- MMBench-Video 1.74
- MMLU 89.1
- CMMLU 90.2
- TriviaQA 74.8
- Natural Questions 50.4
- C3 97.8
- RACE 94.2
- WinoGrande 91.7
- HellaSwag 97
- BIG-Bench Hard 86.5
- GSM8K 91.6
- MATH 94.5
- HumanEval 98.2
- SArena
- Windows Agent Arena 18
Robustness
Dropped since the lab’s previous technical report: SMolInstruct , ChemBench , Matbench , ProteinLMBench , Scientists' First Exam (SFE) , PHYSICS , MicroVQA , MSEarth , XLRS-Bench
- 21 Aug 2025 Command A Reasoning Cohere Command A Reasoning: Enterprise-grade control for AI agents (Cohere blog)
- Berkeley Function Calling Leaderboard (BFCL) 70.3
- tau-bench 59.8
- m-tau-bench (Cohere) 46.5
- DeepResearch Bench
Dropped since the last post: ChartQA , InfographicVQA , AI2D , MathVista , DocVQA , TextVQA , OCRBench , MMMU , RealWorldQA
- 21 Aug 2025
- MMLU-Redux 93.7
- MMLU-Pro 84.8
- GPQA Diamond 80.1
- Humanity's Last Exam 29.8
- BrowseComp 30
- BrowseComp-ZH 49.2
- SimpleQA 93.4
- LiveCodeBench 74.8
- Codeforces rating 2091
- Aider Polyglot 76.3
- SWE-bench Verified 66
- SWE-bench Multilingual 54.5
- Terminal-Bench 31.3
- AIME 2024 93.1
- AIME 2025 88.4
- HMMT 2025 84.2
Dropped since the last post: FRAMES , CNMO 2024 , Berkeley Function Calling Leaderboard (BFCL) , tau-bench
- 20 Aug 2025
- MMLU-Pro 82.7
- MMLU 87.4
- GPQA Diamond 71.4
- SuperGPQA 55.7
- SimpleQA 9.7
- AIME 2024 91.7
- AIME 2025 84.7
- Beyond AIME 65
- ARC-AGI-2 1.45
- KOR-Bench 70.6
- Humanity's Last Exam 10.1
- LiveCodeBench v6 67.4
- IFEval 85.8
- tau-bench 58.2
- SWE-bench Verified 56
- Multi-SWE-bench 17
- MMMLU 78.4
- RULER 94.6
- AIR-Bench 2024 75.6
Dropped since the last post: HumanEval , MBPP , BigCodeBench , LiveCodeBench , Aider-Edit , CanItEdit , MBXP , NaturalCodeBench
- 18 Aug 2025
- AIME 2025 72.1
- MATH-500 97.8
- GPQA Diamond 64
- LiveCodeBench 71.1
- Berkeley Function Calling Leaderboard (BFCL) 66.9
- IFEval 90.3
- Humanity's Last Exam 6.5
- RULER 78.9
Dropped since the last post: MMMU , AI2D , ChartQA , InfographicVQA , OCRBench , OCRBench v2 , DocVQA , Video-MME
Its technical report adds 2 evaluations 7 results in all
- 14 Aug 2025 Gemma 3 270M Google DeepMind Introducing Gemma 3 270M (Google Developers Blog) and Gemma 3 model card
- IFEval 51.2
- BIG-Bench Hard 26.7
- HellaSwag 37.7
- PIQA 66.2
- ARC-Challenge 28.2
- WinoGrande 52.3
Dropped since the last post: MGSM , WMT24++ , INCLUDE , MMLU-ProX , MMMLU , Global MMLU , ECLeKTic , GPQA Diamond , LiveCodeBench v5 , AIME 2025 , MMLU , MBPP , HumanEval , LiveCodeBench , HiddenMath , Global MMLU Lite , MMLU-Pro , BoolQ , SIQA , TriviaQA , Natural Questions , ARC-Easy , DROP
- 12 Aug 2025
- RealWorldQA 65.2
- MM-IFEval 37.7
- InfographicVQA 58.7
- OCRBench 74.2
- BLINK 44.4
- MMStar 49.5
- MMMU 38.4
- MathVista 51.1
- SEED-Bench 72
- MM-Vet 48.1
- MME 1753
- MMLU 51
Dropped since the last post: GPQA Diamond , IFEval , IFBench , GSM8K , MGSM , MMMLU
Its technical report adds 2 evaluations 2 new 12 dropped 12 results in all
LFM2 Technical ReportDropped since the lab’s previous technical report: MMLU , MMLU-Pro , GPQA Diamond , IFEval , IFBench , Multi-IF , GSM8K , GSM-Plus , MATH-500 , MATH Level 5 , MMMLU , MGSM
- 11 Aug 2025
- MMBench-EN v1.1 88.2
- MMBench-CN v1.1 88.3
- MMStar 75.3
- BLINK 65.3
- MuirBench 75.3
- HallusionBench 65.4
- ZeroBench 23.4
- GeoBench 79.7
- MMMU 75.4
- MMMU-Pro 65.2
- MathVista 84.6
- MATH-Vision (MathVision) 65.6
- MathVerse 72.1
- DynaMath 53.9
- LogicVista 62.4
- AI2D 88.1
- We-Math 68.8
- MMLongBench-Doc 44.7
- OCRBench 86.5
- ChartQA Pro 64
- ChartMuseum 55.3
- RefCOCO / RefCOCO+ / RefCOCOg 91.3
- TreeBench 50.1
- Ref-L4 89.5
- OmniSpatial 51
- CVBench 87.3
- ERQA 50
- All-Angles Bench 56.9
- OSWorld 35.8
- AndroidWorld 57
- WebVoyager 84.4
- WebQuest 68.8
- Design2Code 82.2
- Flame-React-Eval 82.5
- Video-MME 80.7
- MMVU 68.7
- Video-MMMU 72.4
- LVBench 53.8
- MotionBench 62.4
- MVBench 73
Dropped since the last post: tau-bench , Berkeley Function Calling Leaderboard (BFCL) , BrowseComp , MMLU-Pro , AIME 2024 , MATH-500 , SciCode , GPQA Diamond , Humanity's Last Exam , LiveCodeBench , Artificial Analysis Intelligence Index v2 , SWE-bench Verified , Terminal-Bench
Its technical report adds 0 evaluations 34 dropped 40 results in all
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement LearningDropped since the lab’s previous technical report: tau-bench , Berkeley Function Calling Leaderboard (BFCL) , BrowseComp , MMLU-Pro , AIME 2024 , MATH-500 , SciCode , GPQA Diamond , Humanity's Last Exam , LiveCodeBench v5 , Artificial Analysis Intelligence Index v2 , SWE-bench Verified , Terminal-Bench , MMLU , SimpleQA , IFEval , SysBench , MultiChallenge , SafetyBench , Novel logical reasoning problems expert evaluation (Z.ai) , Real-scenario prompts human evaluation (Z.ai) , CC-Bench , Challenging translation human evaluation (Z.ai) , BIG-Bench Hard , HellaSwag , PIQA , TriviaQA , EvalPlus , GSM8K , MATH , CLUEWSC , C-Eval , C3 , Chinese SimpleQA
- 10 Aug 2025
- HealthBench 60.1
- HealthBench Hard 34.7
- HealthBench Consensus 91.5
- AIME 2024 83.4
- AIME 2025 72.9
- Arena-Hard v2.0 45.8
- CFBench 77.6
- WritingBench 8.56
Its technical report adds 2 evaluations 10 results in all
Baichuan-M2: Scaling Medical Capability with Large Verifier SystemCapabilities
- IFEval 86
- AlignBench 8.77
- 7 Aug 2025
- AIME 2025 85.2
- FrontierMath 9.6
- GPQA Diamond 71.2
- Humanity's Last Exam 8.7
- HMMT 2025 75.6
- MMMU 75.6
- MMMU-Pro 62.6
- CharXiv 62.7
- Video-MMMU 66.8
- ERQA 50.1
- SWE-Lancer 49000
- SWE-bench Verified 54.7
- Aider Polyglot 48.4
- MultiChallenge 54.9
- Internal API instruction following (OpenAI) 56.1
- COLLIE 96.9
- tau2-bench 46.3
- OpenAI MRCR 43.2
- GraphWalks 64
- BrowseComp Long Context 128k 80.4
- Video-MME 65.7
- LongFact 1
- FActScore 7.3
Its system card adds 8 evaluations 22 dropped 9 results in all
GPT-5 System CardHallucination
- LongFact-Concepts 0.9
- LongFact-Objects 3.1
- SimpleQA 11
- SimpleQA Hallucination Rate 31
Safety
Robustness
Dropped since the lab’s previous system card: HealthBench , HealthBench Hard , HealthBench Consensus , HealthBench Hard Hallucinations , HealthBench Consensus Urgent , HealthBench Consensus Global Health , Long-form Biological Risk Questions , Multimodal Troubleshooting Virology , ProtocolQA Open-Ended , Tacit Knowledge and Troubleshooting , TroubleshootingBench , Challenging prompts from red teamers with biosafety-relevant PhDs , Filtered, adversarial sample of production prompts , Capture the Flag (Collegiate) , Capture the Flag (Professional) , Cyber Range , SWE-bench Verified , OpenAI PRs , MLE-bench , SWE-Lancer , PaperBench , OpenAI-Proof Q&A
- 7 Aug 2025
- AIME 2025 91.1
- FrontierMath 22.1
- GPQA Diamond 82.3
- Humanity's Last Exam 16.7
- HMMT 2025 87.8
- MMMU 81.6
- MMMU-Pro 74.1
- CharXiv 75.5
- Video-MMMU 82.5
- ERQA 62.9
- SWE-Lancer 75000
- SWE-bench Verified 71
- Aider Polyglot 71.6
- MultiChallenge 62.3
- Internal API instruction following (OpenAI) 65.8
- COLLIE 98.5
- tau2-bench 70.8
- OpenAI MRCR 84.3
- GraphWalks 73.4
- BrowseComp Long Context 128k 89.4
- Video-MME 78.5
- LongFact 0.7
- FActScore 3.5
Dropped since the last post: Codeforces rating , HealthBench , HealthBench Hard , AIME 2024 , MMLU , tau-bench
Its system card adds 28 evaluations 14 new 19 dropped 31 results in all
GPT-5 System CardHealth
Hallucination
- LongFact-Concepts 0.9
- LongFact-Objects 1.8
- SimpleQA 22
- SimpleQA Hallucination Rate 26
Safety
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
- OpenAI PRs 42
- MLE-bench 3
- PaperBench 23
- OpenAI-Proof Q&A 1
Dropped since the lab’s previous system card: Challenging Refusal Evaluation , Human Sourced Jailbreaks , PersonQA , Pairwise safety red teaming , Person Identification evaluation , Ungrounded Inference evaluation , BBQ (Bias Benchmark for QA) , First-Person Fairness evaluation , Instruction Hierarchy Evaluation , Image generation refusals , METR Time Horizon , Apollo Research in-context scheming evaluations , Apollo Research sabotage evaluations , Pattern Labs cyberoffensive challenges , Capture the Flag (High School) , OpenAI Research Engineer Interview: Multiple Choice , OpenAI Research Engineer Interview: Coding , SWE-Lancer Diamond (SWE Manager) , MMMLU
- 7 Aug 2025
- AIME 2025 100
- FrontierMath 32.1
- HMMT 2025 100
- GPQA Diamond 89.4
- Humanity's Last Exam 42
- SWE-bench Verified 74.9
- Aider Polyglot 88
- MultiChallenge 69.6
- BrowseComp 54.9
- COLLIE 99
- tau2-bench 80.1
- MMMU 84.2
- MMMU-Pro 78.4
- Video-MMMU 84.6
- CharXiv 81.1
- ERQA 65.7
- HealthBench 67.2
- HealthBench Hard 46.2
- HealthBench Hard Hallucinations 1.6
- LongFact 0.8
- FActScore 1
Dropped since the last post: Codeforces rating , AIME 2024 , MMLU , tau-bench
Its system card adds 50 evaluations 31 new 12 dropped 55 results in all
GPT-5 System CardCapabilities
Health
Hallucination
Safety
- Standard Refusal Evaluation
- OpenAI Production Benchmarks and safe-completion evals
- Sycophancy evaluation (offline) 0.04
- Image input safety evaluations
- Expert Red Teaming for Violent Attack Planning 65.1
- Challenging prompts from red teamers with biosafety-relevant PhDs 92.1
- Filtered, adversarial sample of production prompts 95.7
- Pairwise Bioweaponization Campaign 60
Robustness
Alignment
Bio and chem risk
- Long-form Biological Risk Questions
- Multimodal Troubleshooting Virology 42
- ProtocolQA Open-Ended 30
- Tacit Knowledge and Troubleshooting 66
- TroubleshootingBench 22
- Virology Capabilities Test (VCT) 41.4
- Virology Capabilities Test (text-only) 46
- Human Pathogen Capabilities Test 53.7
- Molecular Biology Capabilities Test 48
- World-Class Bio 47.6
Cyber risk
AI R&D risk
- OpenAI PRs 45
- MLE-bench 8
- SWE-Lancer 55
- PaperBench 24
- OpenAI-Proof Q&A 2
Autonomy
Dropped since the lab’s previous system card: PersonQA , Multilingual performance on translated expert-domain tasks , First-Person Fairness evaluation , Irrelevant instructions, text-based web browser , Irrelevant instructions, visual browser , In-context data exfiltration, visual browser , Active data exfiltration, visual browser , User Confirmations During Computer Use , Agentic harmful task refusals , Proactive refusals of high-risk tasks , Capture the Flag (High School) , OpenAI Research Engineer Interview: Multiple Choice
Its system card addendum adds 5 evaluations 5 results in all
- 5 Aug 2025
- Codeforces rating 2516
- Humanity's Last Exam 17.3
- HealthBench 42.5
- HealthBench Hard 10.8
- AIME 2024 96
- AIME 2025 98.7
- GPQA Diamond 71.5
- MMLU 85.3
- tau-bench 54.8
Dropped since the last post: MMMU , MathVista , SWE-Lancer , SWE-bench Verified , Aider Polyglot , MultiChallenge , BrowseComp , CharXiv
Its model card adds 21 evaluations 30 results in all
gpt-oss-120b & gpt-oss-20b Model CardCapabilities
- SWE-bench Verified 60.7
- Aider Polyglot 34.2
- MMMLU 75.7
Health
Hallucination
- SimpleQA 6.7
- SimpleQA Hallucination Rate 91.4
- PersonQA 15.5
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
- OpenAI PRs 37
- PaperBench 7
- 5 Aug 2025
- SWE-bench Verified 74.5
- Terminal-Bench 43.3
- GPQA Diamond 80.9
- tau-bench 69.2
- MMMLU 89.5
- MMMU 77.1
- AIME 2025 78
Its system card addendum adds 29 evaluations 1 new 29 results in all
System Card Addendum: Claude Opus 4.1Safety
Bias
Alignment
Bio and chem risk
Cyber risk
- Cybench 51.4
AI R&D risk
- 5 Aug 2025
- Codeforces rating 2622
- Humanity's Last Exam 19
- HealthBench 57.6
- HealthBench Hard 30
- AIME 2024 96.6
- AIME 2025 97.9
- GPQA Diamond 80.1
- MMLU 90
- tau-bench 67.8
Dropped since the last post: FrontierMath , DSBench , SpreadsheetBench , Investment Banking Modeling Tasks (OpenAI) , BrowseComp , WebArena
Its model card adds 21 evaluations 30 results in all
gpt-oss-120b & gpt-oss-20b Model CardCapabilities
- SWE-bench Verified 62.4
- Aider Polyglot 44.4
- MMMLU 81.3
Health
Hallucination
- SimpleQA 16.8
- SimpleQA Hallucination Rate 78.2
- PersonQA 29.8
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
- OpenAI PRs 37
- PaperBench 11
- 1 Aug 2025
- Humanity's Last Exam 34.8
- IMO 2025 60.7
- AIME 2025 99.2
- LiveCodeBench v6 87.6
Dropped since the last post: GPQA Diamond , LiveCodeBench v5 , Aider Polyglot , SWE-bench Verified , SimpleQA , FACTS Grounding , MMMU , Vibe-Eval , MRCR v2 , Global MMLU Lite
- 31 Jul 2025
- 31 Jul 2025
- ChartQA 90.9
- InfographicVQA 82.9
- AI2D 94
- MathVista 73.5
- DocVQA 95.9
- TextVQA 84.8
- OCRBench 86.9
- MMMU
- RealWorldQA
Dropped since the last post: Cohere enterprise human evaluation , MMLU , MATH , IFEval , tau-bench , Berkeley Function Calling Leaderboard (BFCL) , MBPP+ , RepoQA
- 31 Jul 2025
- IMO 2025 4
- miniF2F 99.6
- PutnamBench 50.4
- Past IMO (Seed-Prover formal set) 78.1
- CombiBench 30
- miniCTX-v2 81.8
Dropped since the last post: AIME 2024 , AIME 2025 , Beyond AIME , GPQA Diamond , LiveCodeBench , MMMU , MMMU-Pro , MATH-Vision (MathVision) , RealWorldQA , ChartQA , InfographicVQA
- 31 Jul 2025 Seed Diffusion Preview ByteDance Seed Seed Diffusion Preview blog and tech report (arXiv 2508.02193)
- HumanEval 82.8
- MBPP 79.4
- BigCodeBench 53.2
- LiveCodeBench 33.7
- Aider-Edit 44.4
- CanItEdit 54.3
- MBXP 72.6
- NaturalCodeBench 42.2
Dropped since the last post: MME , MMBench , MMMU , MM-Vet , MathVista , GenEval , GEdit-Bench , IntelligentBench
Its technical report adds 0 evaluations 9 dropped 8 results in all
Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed InferenceDropped since the lab’s previous technical report: HumanEval+ , MBPP+ , MHPP , BigCodeBench-Hard , FullStackBench , CRUXEval , CodeEditorBench , SWE-bench Verified , Multi-SWE-bench mini
- 31 Jul 2025
- MMMU 74.2
- MATH-Vision (MathVision) 64.8
- SimpleVQA 62.2
- HallusionBench 64.2
- ZeroBench 23
- DynaMath 50.1
- AIME 2025 82.9
- GPQA Diamond 73
- LiveCodeBench v6 67.1
- HMMT 2025 70
- CNMO 2024 83.7
- 29 Jul 2025
- 28 Jul 2025
- tau-bench 70
- Berkeley Function Calling Leaderboard (BFCL) 77.8
- BrowseComp 26.4
- MMLU-Pro 84.6
- AIME 2024 91
- MATH-500 98.2
- SciCode 41.7
- GPQA Diamond 79.1
- Humanity's Last Exam 14.4
- LiveCodeBench 72.9
- Artificial Analysis Intelligence Index v2 67.7
- SWE-bench Verified 64.2
- Terminal-Bench 37.5
Dropped since the last post: MMLU , GSM8K , MATH , BIG-Bench Hard , HellaSwag , HumanEval , IFEval , AlignBench , LongBench , Needle In A Haystack
GLM-4.5-Air: Its technical report adds 8 evaluations 7 new 7 dropped 20 results in all
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation ModelsCapabilities
- LiveCodeBench v5 70.7
- MMLU 87.4
- IFEval 86.3
- SysBench 77.4
- MultiChallenge 42.5
- Novel logical reasoning problems expert evaluation (Z.ai) 53.4
Hallucination
- SimpleQA 14.5
Safety
- SafetyBench 87.8
Dropped since the lab’s previous technical report: GSM8K , MATH , BIG-Bench Hard , HumanEval , AlignBench , Math23K , GLM-4 All Tools information seeking (Z.ai)
Its technical report adds 22 evaluations 14 new 34 results in all
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation ModelsCapabilities
- LiveCodeBench v5 72.9
- MMLU 90
- IFEval 86.1
- SysBench 81
- MultiChallenge 52.8
- Novel logical reasoning problems expert evaluation (Z.ai) 62
- Real-scenario prompts human evaluation (Z.ai)
- CC-Bench
- Challenging translation human evaluation (Z.ai) 1.71
- BIG-Bench Hard 86.2
- HellaSwag 87.1
- PIQA 85.3
- TriviaQA 80
- EvalPlus 78.1
- GSM8K 79.4
- MATH 61
- CLUEWSC 83.5
- C-Eval 86.9
- C3 83.1
Hallucination
- SimpleQA 26.4
- Chinese SimpleQA 70.1
Safety
- SafetyBench 89.9
- 25 Jul 2025 Llama-3.3-Nemotron-Super-49B-v1.5 NVIDIA Llama-3_3-Nemotron-Super-49B-v1_5 model card (Hugging Face)
- MATH-500 97.4
- AIME 2024 87.5
- AIME 2025 82.7
- GPQA Diamond 72
- LiveCodeBench v5 73.6
- Berkeley Function Calling Leaderboard (BFCL) 71.8
- IFEval 88.6
- Arena-Hard-Auto 92
- Humanity's Last Exam 7.64
- MMLU-Pro 79.5
Dropped since the last post: RoboCasa , RoboCasa GR1 Tabletop Tasks
- 25 Jul 2025
- MMLU-Pro 84.4
- MMLU-Redux 93.8
- GPQA Diamond 81.1
- SuperGPQA 64.9
- AIME 2025 92.3
- HMMT 2025 83.9
- LiveBench 78.4
- Humanity's Last Exam 18.2
- LiveCodeBench v6 74.1
- Codeforces rating 2134
- OJBench 32.5
- IFEval 87.8
- Arena-Hard v2.0 79.7
- Creative Writing v3 86.1
- WritingBench 88.3
- Berkeley Function Calling Leaderboard (BFCL) 71.9
- tau-bench 56.9
- tau2-bench 58.5
- Multi-IF 80.6
- MMLU-ProX 81
- INCLUDE 81
- PolyMath 60.1
Dropped since the last post: Terminal-Bench , SWE-bench Verified , SWE-bench-Live , SWE-bench Multilingual , Multi-SWE-bench mini , Multi-SWE-bench flash , Aider Polyglot , Spider 2.0 , WebArena , Mind2Web
- 24 Jul 2025
- AIME 2024 70.5
- AIME 2025 62
- GPQA Diamond 65.8
- LiveCodeBench v5 59.2
Dropped since the last post: SWE-bench Verified
- 24 Jul 2025
- AIME 2024 72
- AIME 2025 61
- GPQA Diamond 71.5
- LiveCodeBench v5 59.4
Dropped since the last post: FLEURS , Common Voice , Audio Understanding Benchmark (Mistral)
- 24 Jul 2025
- MMLU-Pro 83.5
- MMMU 77.7
- GPQA Diamond 77.3
- MMStar 74.9
- MathVista 81.5
- AIME 2025 86
- MATH-Vision (MathVision) 62.5
- IFEval 86.7
- Scientists' First Exam (SFE) 44.3
- PHYSICS 44
- SMolInstruct 51
- ChemBench 83.4
- Matbench 75
- MicroVQA 63.9
- ProteinLMBench 63.1
- MSEarth 65.7
- XLRS-Bench 55
Its technical report adds 0 evaluations 17 results in all
- 22 Jul 2025
- Terminal-Bench 37.5
- SWE-bench Verified 69.6
- SWE-bench-Live 26.3
- SWE-bench Multilingual 54.7
- Multi-SWE-bench mini 25.8
- Multi-SWE-bench flash 27
- Aider Polyglot 61.8
- Spider 2.0 31.1
- WebArena 49.9
- Mind2Web 55.8
- Berkeley Function Calling Leaderboard (BFCL) 68.7
- tau-bench 68.8
Dropped since the last post: MMLU-Redux , GPQA Diamond , C-Eval , LiveBench , IFEval , Arena-Hard-Auto , AlignBench , Creative Writing v3 , WritingBench , MATH-500 , AIME 2024 , AIME 2025 , ZebraLogic , AutoLogi , LiveCodeBench v5 , Codeforces rating , Multi-IF , INCLUDE , MMMLU , MT-AIME2024 , PolyMath , MLogiQA
- 17 Jul 2025
- Humanity's Last Exam 41.6
- FrontierMath 27.4
- DSBench 89.9
- SpreadsheetBench 45.5
- Investment Banking Modeling Tasks (OpenAI) 41
- BrowseComp 68.9
- WebArena 65.4
Dropped since the last post: AIME 2024 , GPQA Diamond , Codeforces rating
Its system card adds 32 evaluations 11 new 14 dropped 32 results in all
ChatGPT Agent System CardHallucination
- SimpleQA 91.4
- SimpleQA Hallucination Rate 7.9
- PersonQA 94.3
Safety
- Standard Refusal Evaluation
- OpenAI Production Benchmarks and safe-completion evals
- Image input safety evaluations
- User Confirmations During Computer Use 91
- Agentic harmful task refusals
- Proactive refusals of high-risk tasks 89
- Challenging prompts from red teamers with biosafety-relevant PhDs 87.9
- Filtered, adversarial sample of production prompts 96.9
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Challenging Refusal Evaluation , Human Sourced Jailbreaks , Pairwise safety red teaming , Person Identification evaluation , Ungrounded Inference evaluation , Image generation refusals , METR Time Horizon , Apollo Research in-context scheming evaluations , Apollo Research sabotage evaluations , Pattern Labs cyberoffensive challenges , OpenAI Research Engineer Interview: Coding , SWE-Lancer , SWE-Lancer Diamond (SWE Manager) , MMMLU
- 15 Jul 2025
Dropped since the last post: SWE-bench Verified
Its technical report adds 9 evaluations 9 new 14 dropped 12 results in all
VoxtralCapabilities
Dropped since the lab’s previous technical report: AIME 2024 , AIME 2025 , MATH-500 , GPQA Diamond , LiveCodeBench v5 , LiveCodeBench v6 , Aider Polyglot , Humanity's Last Exam , Multilingual AIME 2024 (Mistral) , Mistral internal function calling benchmark , IFEval (Mistral internal version) , MMMU , MathVista , MMMU-Pro
- 15 Jul 2025
- MMLU-Redux 92.3
- MMLU-Pro 81.8
- GPQA Diamond 75.4
- AIME 2025 85.3
- HMMT 2025 72.9
- LiveCodeBench v5 72.6
- LiveCodeBench v6 66.7
- IFEval 84.8
- Multi-IF 73.5
- Berkeley Function Calling Leaderboard (BFCL) 65.2
- tau-bench 57.1
- KMMLU-Pro 67.7
- KMMLU-Redux 72.7
- HRM8K-KSM 87.6
- MMMLU 85.6
- HELMET 58.3
- RULER 88.2
- LongBench 48.1
- Ko-LongBench (LG) 76.9
Dropped since the last post: MATH-500 , AIME 2024 , CSAT Math 2025
Its technical report adds 1 evaluation 1 new 4 dropped 20 results in all
EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning ModesCapabilities
- WMT24++ 90.7
Dropped since the lab’s previous technical report: MATH-500 , AIME 2024 , CSAT Math 2025 , MMLU
- 11 Jul 2025
- LiveCodeBench v6 53.7
- OJBench 27.1
- MultiPL-E 85.7
- SWE-bench Verified 65.8
- SWE-bench Multilingual 47.3
- Terminal-Bench 30
- Aider Polyglot 60
- tau2-bench 64.3
- AceBench 76.5
- AIME 2024 69.6
- AIME 2025 49.5
- MATH-500 97.4
- HMMT 2025 38.8
- CNMO 2024 74.3
- PolyMath 65.1
- ZebraLogic 89
- AutoLogi 89.5
- GPQA Diamond 75.1
- SuperGPQA 57.2
- Humanity's Last Exam 4.7
- MMLU 89.5
- MMLU-Redux 92.7
- MMLU-Pro 81.1
- IFEval 89.8
- MultiChallenge 54.1
- SimpleQA 31
- LiveBench 76.4
Dropped since the last post: Codeforces rating , LiveCodeBench , MathVista , MMMU , MATH-Vision (MathVision) , CLUEWSC , C-Eval , HumanEval Multi
Its technical report adds 14 evaluations 14 new 8 dropped 41 results in all
Kimi K2: Open Agentic Intelligence (arXiv 2507.20534)Capabilities
- Multi-SWE-bench 18.3
- SWE-Lancer 39.1
- Arena-Hard v2.0 54.5
- LongBench v2 49.1
- FRAMES 77.1
- OpenAI MRCR 55
- DROP 93.5
- Kimi Chinese in-house open-ended benchmark
- LMArena (Elo)
Hallucination
AI R&D risk
- PaperBench 27.8
Dropped since the lab’s previous technical report: Codeforces rating , LiveCodeBench , MathVista , MMMU , MATH-Vision (MathVision) , CLUEWSC , C-Eval , HumanEval Multi
- 10 Jul 2025 Devstral Small 1.1 Mistral AI Upgrading agentic coding capabilities with the new Devstral models (Mistral news)
- SWE-bench Verified 53.6
Dropped since the last post: WildBench , Arena-Hard v2.0 , IFEval (Mistral internal version) , Infinite Generations (Mistral internal) , MMLU , MMLU-Pro , MATH , GPQA Main , GPQA Diamond , MBPP+ , HumanEval+ , SimpleQA , MMMU , MathVista , ChartQA , DocVQA , AI2D
Its technical report adds 0 evaluations 1 results in all
- 10 Jul 2025 Devstral Medium Mistral AI Upgrading agentic coding capabilities with the new Devstral models (Mistral news)
- SWE-bench Verified 61.6
Dropped since the last post: AIME 2024 , AIME 2025 , GPQA Diamond , LiveCodeBench v5 , Aider Polyglot
- 10 Jul 2025LFM2-1.2B Liquid AI Liquid Foundation Models v2: Our Second Series of Generative AI Models (Liquid AI blog)
Its technical report adds 5 evaluations 12 results in all
LFM2 Technical ReportCapabilities
- MMLU-Pro 19.7
- Multi-IF 45.3
- GSM-Plus 36.1
- MATH-500 42.4
- MATH Level 5 18.6
- 10 Jul 2025
- KMMLU 77
- HAE-RAE Bench 86.2
- KBL 75.5
- KBank-MMLU (Upstage) 71
- KorMedMCQA 87.4
- MMLU 88.2
- MMLU-Pro 76.5
- IFEval 87.8
- Arena-Hard v2.0 46.2
- HumanEval 98.8
- SWE-bench Verified 40.8
- MATH-500 95.4
- AIME 2024 76.5
- AIME 2025 69.1
- Berkeley Function Calling Leaderboard (BFCL) 63.9
- 9 Jul 2025
- AIME 2024 52.3
- AIME 2025 33.6
- MATH-500 92.5
- GPQA Diamond 45.1
Dropped since the last post: Omni-MATH , LiveCodeBench , FlenQA , IFEval , Arena-Hard-Auto , HumanEval+ , MMLU-Pro , Kitab , PhiBench
Its technical report adds 0 evaluations 18 dropped 4 results in all
Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long GenerationDropped since the lab’s previous technical report: HMMT 2025 , Omni-MATH , LiveCodeBench , Codeforces rating , AIME 1983-2024 , BA-Calendar , 3SAT-Search , TSP-Opt , SpatialEval Maze-Nav , SpatialEval Spatial-Map , FlenQA , IFEval , Arena-Hard-Auto , HumanEval+ , MMLU-Pro , Kitab , ToxiGen (discriminative) , PhiBench
- 9 Jul 2025
- Humanity's Last Exam 38.6
- GPQA Diamond 87.5
- LiveCodeBench 79.3
- USAMO 2025 37.5
- HMMT 2025 93.9
- AIME 2025 98.8
- ARC-AGI-2 15.9
- Vending-Bench 4694.1
Its model card adds 13 evaluations 13 results in all
Grok 4 Model CardSafety
Robustness
Bias
- Soft Bias (Internal) 0.36
Alignment
Bio and chem risk
Cyber risk
- Cybench 43
Persuasion
- MakeMeSay 12
- 9 Jul 2025
- Humanity's Last Exam 44.4
- GPQA Diamond 88.4
- LiveCodeBench 79.4
- USAMO 2025 61.9
- HMMT 2025 96.7
- AIME 2025 100
Dropped since the last post: AIME 2024 , MMMU , MMLU-Pro , LOFT , SimpleQA , EgoSchema , LMArena (Elo)
- 1 Jul 2025
- MMBench-EN v1.1 85.8
- MMBench-CN v1.1 84.7
- MMStar 72.9
- BLINK 65.1
- MuirBench 74.7
- MMMU 68
- MMMU-Pro 57.1
- Video-MMMU 61
- AI2D 87.9
- MathVista 80.7
- We-Math 63.8
- ChartQA Pro 59.5
- ChartMuseum 48.8
- OCRBench 84.2
- MMLongBench-Doc 42.4
- RefCOCO / RefCOCO+ / RefCOCOg 87.4
- OSWorld 14.9
- AndroidWorld 41.7
- WebVoyager 69
- WebQuest 63.4
- Design2Code 64.7
- Flame-React-Eval 72.5
- Video-MME 73.6
- MMVU 59.4
- LVBench 45.1
- MotionBench 59
Dropped since the last post: AIME 2024 , AIME 2025 , Omni-MATH , LiveCodeBench v5 , SysBench , IFEval , Arena-Hard-Auto , Berkeley Function Calling Leaderboard (BFCL) , SuperGPQA , GPQA Diamond
Its technical report adds 14 evaluations 14 new 16 dropped 40 results in all
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement LearningCapabilities
- HallusionBench 63.2
- ZeroBench 19.2
- GeoBench 76
- MATH-Vision (MathVision) 54.4
- MathVerse 68.4
- DynaMath 42.5
- LogicVista 60.4
- TreeBench 37.5
- Ref-L4 86.8
- OmniSpatial 47.7
- CVBench 85
- ERQA 45.8
- All-Angles Bench 52.7
- MVBench 68.4
Dropped since the lab’s previous technical report: GSM8K , MATH , BIG-Bench Hard , MMLU , GPQA Diamond , HumanEval , BoolQ , CommonSenseQA , HellaSwag , PIQA , DROP , C-Eval , CMMLU , GAOKAO-Bench , C3 , Berkeley Function Calling Leaderboard (BFCL)
- 30 Jun 2025 ERNIE-4.5-300B-A47B Baidu Announcing the Open Source Release of the ERNIE 4.5 Model Family (ERNIE Blog)
- MMLU 86.5
- MMLU-Pro 78.4
- C-Eval 90.6
- CMMLU 90.2
- LiveBench 68.9
- Chinese SimpleQA 77.1
- SimpleQA 45.9
- IFEval 88
- Multi-IF 76.6
- SysBench 69.8
- MATH-500 96.4
- GSM8K 96.6
- CMATH 96.7
- OlympicArena 75.3
- AIME 2024 54.8
- AIME 2025 35.1
- BIG-Bench Hard 94.3
- MuSR 69.9
- DROP 91.1
- ZebraLogic 58.1
- LiveCodeBench 38.8
- HumanEval+ 92.1
- MBPP+ 81
- FullStackBench 67.1
Its technical report adds 1 evaluation 24 results in all
ERNIE 4.5 Technical ReportCapabilities
- LiveCodeBench v6 38.8
- 27 Jun 2025
- AIME 2024 87.3
- AIME 2025 76.8
- MATH 94.3
- GPQA Diamond 71.2
- OlympiadBench 82.7
- LiveCodeBench 63.9
- FullStackBench 67.8
- ArtifactsBench 43
- BIG-Bench Hard 89.1
- DROP 91.1
- ZebraLogic 84.7
- IFEval 84.7
- SysBench 76.1
- LengthCtrl (Tencent) 55.4
- InsCtrl (Tencent) 71.9
- ComplexNLU (Tencent) 61.2
- Word-Task (Tencent) 62.9
- Berkeley Function Calling Leaderboard (BFCL) 78.3
- tau-bench 54.7
- ComplexFuncBench 61.2
- C3-Bench 63.5
Its technical report adds 5 evaluations 26 results in all
Hunyuan-A13B Technical ReportCapabilities
- McEval 59.6
- PenguinScrolls 87.7
- LongBench v2 55
- FRAMES 81.1
- RULER 76.7
- 26 Jun 2025 Gemma 3n E4B Google DeepMind Introducing Gemma 3n: The developer guide (Google Developers Blog) and Gemma 3n model card
- MGSM 60.7
- WMT24++ 50.1
- INCLUDE 57.2
- MMLU-ProX 19.9
- MMMLU 35.6
- Global MMLU 60.3
- ECLeKTic 1.9
- GPQA Diamond 23.7
- LiveCodeBench v5 25.7
- AIME 2025 11.6
- MMLU 64.9
- MBPP 63.6
- HumanEval 75
- LiveCodeBench 13.2
- HiddenMath 37.7
- Global MMLU Lite 64.5
- MMLU-Pro 50.6
- HellaSwag 78.6
- BoolQ 81.6
- PIQA 81
- SIQA 50
- TriviaQA 70.2
- Natural Questions 20.9
- ARC-Challenge 61.6
- ARC-Easy 81.6
- WinoGrande 71.7
- BIG-Bench Hard 52.9
- DROP 60.8
Dropped since the last post: Humanity's Last Exam , Aider Polyglot , SWE-bench Verified , SimpleQA , FACTS Grounding , MMMU , Vibe-Eval , MRCR v2
- 26 Jun 2025Mercury Inception Labs Introducing Mercury, our General Chat Diffusion Large Language Model (Inception blog)
- MMLU-Pro 69
- GPQA Diamond 51
- Humanity's Last Exam 3.4
- SciCode 18
- HumanEval 85
- MATH-500 83
- AIME 2024 30
Dropped since the last post: MBPP , EvalPlus , MultiPL-E , LiveCodeBench , BigCodeBench
- 25 Jun 2025
- AIME 2024 90
- AIME 2025 86
- Beyond AIME 56.3
- GPQA Diamond 81.5
- LiveCodeBench 64
- MMMU 74.8
- MMMU-Pro 66.4
- MATH-Vision (MathVision) 70.3
- RealWorldQA 77.8
- ChartQA 89.1
- InfographicVQA 88
Dropped since the last post: OlympiadBench , MathVista , V* , VLMs Are Blind , ZeroBench , VisuLogic , SimpleVQA , MMStar , MMBench , MMVP , HallusionBench , TextVQA , AI2D , DocVQA , CharXiv , BLINK , VisualWebBench , RefCOCO / RefCOCO+ / RefCOCOg , CountBench , All-Angles Bench , MotionBench , MVBench , TOMATO , TVBench , TempCompass , LongVideoBench , LVBench , MLVU , Video-MME , OVBench , OVO-Bench , StreamingBench , Video-MMMU , MMVU , Charades-STA , ScreenSpot-v2 , ScreenSpot-Pro , OSWorld , Windows Agent Arena , WebVoyager , Online-Mind2Web , AndroidWorld , OCRBench , LVIS-MG , DREAM-1K , TemporalBench , StreamBench (StreamChat) , TACoS
- 21 Jun 2025
- MMBench-EN v1.1 84.4
- RealWorldQA 70
- OCRBench 86.9
- MMStar 70.4
- MM-Vet 78.1
- MMMU 64
- MMMU-Pro 46.3
- MATH-Vision (MathVision) 56.9
- MathVista 80.1
- Video-MMMU 65.2
- MMVU 57.5
- Video-MME 71.9
- ScreenSpot-Pro 52.8
- ScreenSpot-v2 91.4
- OSWorld-G 52.5
- MMLongBench-Doc 42.1
- MMLU 82
- MMLU-Pro 68.5
- MATH 91.8
- GPQA Diamond 42.3
- V* 83.2
Dropped since the last post: SWE-bench Verified
Its technical report adds 0 evaluations 11 dropped 17 results in all
Kimi-VL Technical ReportDropped since the lab’s previous technical report: LibriSpeech , FLEURS , AISHELL-1 , AISHELL-2 , WenetSpeech , Kimi-ASR Internal Testset , MMAU , Clotho-AQA , OpenAudioBench , VoiceBench , Kimi-Audio speech conversation evaluation
- 20 Jun 2025
- WildBench 65.3
- Arena-Hard v2.0 43.1
- IFEval (Mistral internal version) 84.8
- Infinite Generations (Mistral internal) 1.29
- MMLU 80.5
- MMLU-Pro 69.1
- MATH 69.4
- GPQA Main 44.2
- GPQA Diamond 46.1
- MBPP+ 78.3
- HumanEval+ 92.9
- SimpleQA 12.1
- MMMU 62.5
- MathVista 67.1
- ChartQA 87.4
- DocVQA 94.9
- AI2D 92.9
Dropped since the last post: AIME 2024 , AIME 2025 , LiveCodeBench v5
- 17 Jun 2025
- Humanity's Last Exam 11
- GPQA Diamond 82.8
- AIME 2025 72
- LiveCodeBench v5 63.9
- Aider Polyglot 61.9
- SWE-bench Verified 60.4
- SimpleQA 26.9
- FACTS Grounding 85.3
- MMMU 79.7
- Vibe-Eval 65.4
- MRCR v2 74
- Global MMLU Lite 88.4
Dropped since the last post: LiveCodeBench , Video-MME , Video-MMMU
Its technical report adds 45 evaluations 45 new 2 dropped 56 results in all
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic CapabilitiesCapabilities
- LiveCodeBench 59.3
- ECLeKTic 36.8
- HiddenMath-Hard 75.5
- LOFT 82.1
- ZeroBench 2
- BetterChartQA 67.3
- FLEURS 9.95
- CoVoST2 36.1
- ActivityNet-QA 65.1
- EgoTempo 36.7
- Perception Test MCVQA 75.1
- QVHighlights 52.4
- Video-MMMU 79.2
- 1H-VideoQA 67.5
- LVBench 62.7
- Video-MME 81.5
- VATEX 65.2
- VATEX ZH 43.9
- YouCook2 177.6
- MINERVA 60.7
- Neptune 84.3
- LMArena (Elo)
Safety
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous technical report: Where2Place , SUNRGBD
- 17 Jun 2025
- Humanity's Last Exam 6.9
- GPQA Diamond 66.7
- AIME 2025 63.1
- LiveCodeBench v5 34.3
- Aider Polyglot 27.1
- SWE-bench Verified 44.9
- SimpleQA 13
- FACTS Grounding 86.8
- MMMU 72.9
- Vibe-Eval 57.5
- MRCR v2 30.6
- Global MMLU Lite 84.5
Dropped since the last post: BIG-Bench Hard , BIG-Bench Extra Hard , IFEval , HellaSwag , BoolQ , PIQA , SIQA , TriviaQA , Natural Questions , ARC-Challenge , ARC-Easy , WinoGrande , DROP , MMLU-Pro , LiveCodeBench , Bird-SQL , MATH , HiddenMath , MBPP , HumanEval , Natural2Code , GSM8K , MMLU , AGIEval , ECLeKTic , WMT24++ , MGSM , Flores , XQuAD , IndicGenBench , DocVQA , InfographicVQA , TextVQA , AI2D , ChartQA , VQAv2 , MathVista , COCO Captions , RealWorldQA , ReMI , BLINK , OK-VQA , TallyQA , SpatialSense VQA , CountBench
- 17 Jun 2025
- Humanity's Last Exam 21.6
- GPQA Diamond 86.4
- AIME 2025 88
- LiveCodeBench 69
- Aider Polyglot 82.2
- SWE-bench Verified 67.2
- SimpleQA 54
- FACTS Grounding 87.8
- MMMU 82
- Vibe-Eval 67.2
- Video-MME 86.9
- Video-MMMU 83.6
- MRCR v2 58
- Global MMLU Lite 89.2
Dropped since the last post: WebDev Arena
Its technical report adds 46 evaluations 4 new 60 results in all
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic CapabilitiesCapabilities
- ECLeKTic 46.8
- HiddenMath-Hard 80.5
- LOFT 87
- ZeroBench 4.5
- BetterChartQA 72.4
- FLEURS 6.66
- CoVoST2 38.5
- ActivityNet-QA 66.7
- EgoTempo 44.3
- Perception Test MCVQA 78.4
- QVHighlights 75
- 1H-VideoQA 81
- LVBench 78.7
- VATEX 71.3
- VATEX ZH 59.7
- YouCook2 188.3
- MINERVA 67.6
- Neptune 87.3
- LMArena (Elo)
- WebDev Arena
Safety
Robustness
Alignment
- GDM Stealth 40
- GDM Situational Awareness 18.2
Bio and chem risk
Cyber risk
AI R&D risk
- 16 Jun 2025Kimi-Dev-72B Moonshot AI Introducing Kimi-Dev: A Strong and Open-source Coding LLM for Issue Resolution (model card)
- SWE-bench Verified 60.4
Dropped since the last post: LibriSpeech , FLEURS , AISHELL-1 , AISHELL-2 , WenetSpeech , MMAU , Clotho-AQA , OpenAudioBench , VoiceBench
Its technical report adds 0 evaluations 17 dropped 1 results in all
Kimi-Dev: Agentless Training as Skill Prior for SWE-AgentsDropped since the lab’s previous technical report: MATH-Vision (MathVision) , MathVista , MMMU , MMMU-Pro , Video-MMMU , MMBench-EN v1.1 , RealWorldQA , OCRBench , MMStar , MM-Vet , MMVU , Video-MME , ScreenSpot-Pro , ScreenSpot-v2 , OSWorld-G , MMLongBench-Doc , V*
- 16 Jun 2025
- AIME 2024 86
- AIME 2025 76.9
- MATH-500 96.8
- LiveCodeBench 65
- FullStackBench 68.3
- GPQA Diamond 70
- Humanity's Last Exam 8.4
- ZebraLogic 86.8
- MMLU-Pro 81.1
- SWE-bench Verified 56
- OpenAI MRCR 73.4
- LongBench v2 61.5
- tau-bench 62.8
- SimpleQA 18.5
- MultiChallenge 44.7
Dropped since the last post: MMLU , Chinese SimpleQA , IFEval , Arena-Hard-Auto , DROP , GSM8K , MATH , MBPP , HumanEval , RULER , MTOB
Its technical report adds 1 evaluation 1 new 21 dropped 15 results in all
MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning AttentionCapabilities
Dropped since the lab’s previous technical report: MMLU , Chinese SimpleQA , IFEval , Arena-Hard-Auto , DROP , GSM8K , MATH , EvalPlus , HumanEval , RULER , MTOB , Needle In A Haystack , MR-NIAH (Multi-Round Needles-In-A-Haystack) , General Assistant (MiniMax in-house) , Hard Capability (MiniMax in-house) , Creative Writing (MiniMax in-house) , Knowledge Q&A (MiniMax in-house) , Instruction Following (MiniMax in-house) , Coding (MiniMax in-house) , Safety (MiniMax in-house) , Long Context (MiniMax in-house)
- 11 Jun 2025 GR00T N1.5 NVIDIA GR00T N1.5: An Improved Open Foundation Model for Generalist Humanoid Robots (NVIDIA GEAR)
- RoboCasa 47.5
- RoboCasa GR1 Tabletop Tasks 47.4
Dropped since the last post: GPQA Diamond , AIME 2025 , Berkeley Function Calling Leaderboard (BFCL) , LiveCodeBench v5 , IFEval , MATH-500 , JudgeBench
Its model card adds 2 evaluations 3 results in all
- 11 Jun 2025 V-JEPA 2 Meta Introducing V-JEPA 2 world model and new benchmarks for physical reasoning (Meta AI blog) and GitHub README
- MVPBench 44.5
- TempCompass 76.9
Dropped since the last post: MMMU , MMMU-Pro , MathVista , ChartQA , DocVQA , LiveCodeBench v5 , MMLU-Pro , GPQA Diamond , MGSM , MTOB , MMLU , MATH , MBPP , TydiQA
- 10 Jun 2025
- AIME 2024 70.7
- AIME 2025 62.8
- GPQA Diamond 68.2
- LiveCodeBench v5 55.8
Dropped since the last post: SWE-bench Verified
Its technical report adds 6 evaluations 6 new 20 dropped 10 results in all
MagistralCapabilities
- MATH-500 95.9
- LiveCodeBench v6 47.4
- Humanity's Last Exam 6.4
- MMMU 66
- MathVista 66.8
- MMMU-Pro 51.8
Dropped since the lab’s previous technical report: MMLU , HellaSwag , ARC-Challenge , WinoGrande , MBPP , GSM8K , MT-Bench , PIQA , ARC-Easy , Natural Questions , TriviaQA , HumanEval , MATH , BIG-Bench Hard , AGIEval , Translated ARC-Challenge, HellaSwag and MMLU (Mistral) , Passkey Retrieval , BBQ (Bias Benchmark for QA) , BOLD , LMArena (Elo)
- 10 Jun 2025
- AIME 2024 93
- GPQA Diamond 84
- Codeforces rating 2748
Dropped since the last post: SWE-bench Verified
- 10 Jun 2025
- AIME 2024 90
- AIME 2025 83.3
- GPQA Diamond 70.8
- LiveCodeBench v5 59.4
- Aider Polyglot 47.1
Dropped since the last post: HumanEval , LiveCodeBench v6 , MultiPL-E , Arena-Hard-Auto , IFEval , MATH-500 , MMLU-Pro , RULER , MMMU , DocVQA , AI2D , ChartQA
Its technical report adds 9 evaluations 14 results in all
MagistralCapabilities
- 9 Jun 2025Apple Foundation Model Server (2025) Apple Updates to Apple's On-Device and Server Foundation Language Models (Apple Machine Learning Research)
Its technical report adds 4 evaluations 5 results in all
- 3 Jun 2025
- MMMU 48.2
- AI2D 85
- ChartQA 86.3
- InfographicVQA 77.4
- OCRBench 83.9
- OCRBench v2 60.1
- DocVQA 91.2
- Video-MME 54.7
- 28 May 2025
- LiveCodeBench v5 60.5
- LiveCodeBench v6 46.8
- HumanEval+ 88.4
- MBPP+ 81.5
- BigCodeBench-Hard 28.5
Dropped since the last post: GPQA Diamond , HumanEval , MATH-500 , AIME 2024 , AIME 2025 , MMLU-Pro , Berkeley Function Calling Leaderboard (BFCL)
- 28 May 2025
- MMLU-Redux 93.4
- MMLU-Pro 85
- GPQA Diamond 81
- SimpleQA 27.8
- FRAMES 83
- Humanity's Last Exam 17.7
- LiveCodeBench 73.3
- Codeforces rating 1930
- SWE-bench Verified 57.6
- Aider Polyglot 71.6
- AIME 2024 91.4
- AIME 2025 87.5
- HMMT 2025 79.4
- CNMO 2024 86.9
- Berkeley Function Calling Leaderboard (BFCL) 37
- tau-bench 58.7
Dropped since the last post: MATH-500
- 23 May 2025Sarvam-M Sarvam AI Sarvam-M: Explorations in Post Training and Inferencing Optimizations for a Hybrid Indic LLM (Sarvam AI blog)
- MMLU 87
- ARC-Challenge 95
- TriviaQA 91
- GPQA Diamond 48
- HumanEval 88
- MBPP 75
- GSM8K 94
- MATH 81
- IFEval 87
- MT-Bench 8.14
- IndicGenBench 49
- MILU 75
- 22 May 2025
- SWE-bench Verified 80.2
- Terminal-Bench 41.3
- GPQA Diamond 83.8
- tau-bench 70.3
- MMMLU 86.5
- MMMU 74.4
- AIME 2025 85
Its system card adds 42 evaluations 6 dropped 42 results in all
System Card: Claude Opus 4 & Claude Sonnet 4Safety
Robustness
Bias
Alignment
Bio and chem risk
- Bioweapons acquisition uplift trial (Anthropic) 1.7
- Long-form virology tasks
- Virology Capabilities Test (VCT)
- Bioweapons knowledge questions 51.5
- DNA Synthesis Screening Evasion
- LAB-Bench FigQA
- LAB-Bench ProtocolQA
- LAB-Bench SeqQA
- LAB-Bench CloningScenarios
- Creative biology 37
- Short-horizon computational biology tasks
Cyber risk
AI R&D risk
- SWE-bench Verified (hard subset) 36.7
- METR data deduplication 27.6
- Kernels task (Anthropic) 41.7
- Time series forecasting task (Anthropic) 5.8
- Text-based reinforcement learning task (Anthropic) 0.675
- LLM training task (Anthropic) 1.88
- Quadruped reinforcement learning task (Anthropic) 0.35
- Novel compiler task (Anthropic)
- Internal AI research evaluation suite 2 0.365
Dropped since the lab’s previous system card: Agentic misalignment suite (Anthropic) , Reasoning behavior classifiers (Anthropic) , Many-shot jailbreaking evaluation (Anthropic) , SHADE-Arena , Internal model use survey (Anthropic) , Cyber-harness network challenges (Incalmo cyber ranges)
- 22 May 2025
- SWE-bench Verified 79.4
- Terminal-Bench 50
- GPQA Diamond 83.3
- tau-bench 70.5
- MMMLU 88.8
- MMMU 76.5
- AIME 2025 90
Dropped since the last post: IFEval , MATH-500 , AIME 2024
Its system card adds 48 evaluations 24 new 4 dropped 48 results in all
System Card: Claude Opus 4 & Claude Sonnet 4Safety
- Single-turn harmful request evaluations (Anthropic) 98.4
- Single-turn benign request evaluations (Anthropic) 0.07
- Ambiguous context evaluations (Anthropic)
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Malicious computer use (Anthropic)
- Malicious use of Claude Code (Anthropic) 88
Robustness
Bias
Alignment
- Agentic misalignment suite (Anthropic) 84
- SycophancyEval (Sharma et al.)
- Reasoning behavior classifiers (Anthropic)
- SHADE-Arena 0
- Anthropic automated behavioral audit
- Average in-eval reward hacking rate (Anthropic) 15.2
- Reward-hack-prone coding tasks (Anthropic) 9
- Impossible tasks (Anthropic) 47
- Training distribution hack rate (Anthropic)
Bio and chem risk
- Bioweapons acquisition uplift trial (Anthropic) 2.53
- Long-form virology tasks
- Virology Capabilities Test (VCT)
- Bioweapons knowledge questions 51.5
- DNA Synthesis Screening Evasion
- LAB-Bench FigQA
- LAB-Bench ProtocolQA
- LAB-Bench SeqQA
- LAB-Bench CloningScenarios
- Creative biology 45
- Short-horizon computational biology tasks
Cyber risk
AI R&D risk
- SWE-bench Verified (hard subset) 39.5
- METR data deduplication 32.6
- Kernels task (Anthropic) 72.7
- Time series forecasting task (Anthropic) 6.15
- Text-based reinforcement learning task (Anthropic) 0.625
- LLM training task (Anthropic) 2.99
- Quadruped reinforcement learning task (Anthropic) 1.25
- Novel compiler task (Anthropic)
- Internal AI research evaluation suite 2 0.355
- Internal model use survey (Anthropic) 0
Dropped since the lab’s previous system card: Appropriate harmlessness (Anthropic) , Chain-of-thought faithfulness with clues (Anthropic) , Alignment faking evaluation, helpful-only setting (Anthropic) , RE-Bench subset (Anthropic)
- 21 May 2025 Falcon-Arabic-7B Technology Innovation Institute Falcon-Arabic: A Breakthrough in Arabic Language Models (Falcon blog)
- Open Arabic LLM Leaderboard v2 68
- ArabicMMLU 68.2
- AraTrust 82.5
Its technical report adds 12 evaluations 15 results in all
Adapting Falcon3-7B Language Model for Arabic: Methods, Challenges, and OutcomesCapabilities
- AlGhafa 72.4
- MadinahQA 73.6
- ALRAGE 72.3
- ArabCulture 65.2
- AraDiCE
- IFEval 57.6
- GSM8K 67.7
- HellaSwag 71.4
- ARC-Challenge 53.5
- WinoGrande 68.4
- MMLU 63.3
Hallucination
- TruthfulQA 31.5
- 21 May 2025
- SWE-bench Verified 46.8
Dropped since the last post: MMLU , MMLU-Pro , MATH , GPQA Main , GPQA Diamond , MBPP , HumanEval , SimpleQA , MMMU , MMMU-Pro , MathVista , ChartQA , DocVQA , AI2D , MM-MT-Bench , LongBench v2 , RULER
Its technical report adds 0 evaluations 10 dropped 1 results in all
Devstral: Fine-tuning Language Models for Coding Agent ApplicationsDropped since the lab’s previous technical report: AIME 2024 , AIME 2025 , MATH-500 , GPQA Diamond , LiveCodeBench v5 , LiveCodeBench v6 , Humanity's Last Exam , MMMU , MathVista , MMMU-Pro
- 20 May 2025 Falcon-H1-34B Technology Innovation Institute Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance (Falcon blog)
- BIG-Bench Hard 70.7
- ARC-Challenge 61
- TruthfulQA 65.3
- HellaSwag 81.9
- MMLU 84
- GSM8K 83.6
- MATH-500 83.8
- AIME 2024 23.8
- AIME 2025 16.7
- GPQA Main 41.5
- GPQA Diamond 49.7
- MMLU-Pro 58.7
- HumanEval 87.2
- HumanEval+ 81.7
- MBPP 83.9
- MBPP+ 71.4
- CRUXEval 73.1
- IFEval 89.4
- MT-Bench 9.2
- LiveBench 46.3
Its technical report adds 6 evaluations 26 results in all
Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and PerformanceCapabilities
- AMC 69.4
- MMLU-stem 83.6
- AlpacaEval 2.0 48.3
- Okapi MMLU 77.8
- MGSM 76.3
- HELMET
- 20 May 2025
- MME 2388
- MMBench 85
- MMMU 55.3
- MM-Vet 67.2
- MathVista 73.1
- GenEval 0.88
- GEdit-Bench 6.52
- IntelligentBench 55.3
Dropped since the last post: HumanEval , MBPP , MHPP , BigCodeBench , BigCodeBench-Hard , LiveCodeBench
Its technical report adds 4 evaluations 12 results in all
Emerging Properties in Unified Multimodal PretrainingCapabilities
- MMVP 69.3
- WISE 0.7
- RISEBench 11.9
- KRIS-Bench 60.2
- 16 May 2025
- SWE-bench Verified 72.1
Dropped since the last post: AIME 2024 , AIME 2025 , Codeforces rating , GPQA Diamond , Humanity's Last Exam , MMMU , MathVista , CharXiv , SWE-Lancer , Aider Polyglot , MultiChallenge , BrowseComp , tau-bench
Its system card addendum adds 6 evaluations 4 new 6 results in all
- 15 May 2025
- 12 May 2025
- MMMU 77.9
- MMMU-Pro 67.6
- MATH-Vision (MathVision) 68.7
- OlympiadBench 65
- MathVista 85.6
- V* 89
- VLMs Are Blind 92.1
- ZeroBench 2
- VisuLogic 35
- RealWorldQA 78.4
- SimpleVQA 63.4
- MMStar 77.8
- MMBench 89.9
- MMVP 69.3
- HallusionBench 60.3
- TextVQA 81.8
- AI2D 87.3
- ChartQA 89.1
- InfographicVQA 91.2
- DocVQA 96.9
- CharXiv 60.2
- BLINK 72.1
- VisualWebBench 87.3
- RefCOCO / RefCOCO+ / RefCOCOg 91.3
- CountBench 93.7
- All-Angles Bench 58.6
- MotionBench 68.4
- MVBench 74.4
- TOMATO 44.7
- TVBench 63.6
- TempCompass 83.7
- LongVideoBench 74
- LVBench 64.6
- MLVU 82.1
- Video-MME 77.9
- OVBench 60
- OVO-Bench 72.3
- StreamingBench 68
- Video-MMMU 81.4
- MMVU 70.1
- Charades-STA 64
- ScreenSpot-v2 95.2
- ScreenSpot-Pro 60.9
- OSWorld 36.7
- Windows Agent Arena 39.6
- WebVoyager 87.2
- Online-Mind2Web 76.4
- AndroidWorld 62.1
- OCRBench 861
- LVIS-MG 72.5
- DREAM-1K 43.9
- TemporalBench 79.8
- StreamBench (StreamChat) 72.8
- TACoS 49.6
Its technical report adds 3 evaluations 3 new 14 dropped 57 results in all
Seed1.5-VL Technical ReportCapabilities
Dropped since the lab’s previous technical report: AIME 2025 , AIME 2024 , Beyond AIME , GPQA Diamond , SuperGPQA , MMLU-Pro , Codeforces rating , LiveCodeBench v5 , Aider Polyglot , SWE-bench Verified , ARC-AGI , SimpleQA , COLLIE , IFEval
- 11 May 2025
- AIME 2024 78.8
- AIME 2025 64.9
- LiveCodeBench v5 67.8
- GPQA Diamond 66.8
- IFEval 81.5
Its technical report adds 0 evaluations 5 results in all
- 8 May 2025
- HumanEval 84.8
- MBPP 85.2
- MHPP 36.2
- BigCodeBench 53.3
- BigCodeBench-Hard 26.4
- LiveCodeBench 24.7
Its technical report adds 11 evaluations 11 new 12 dropped 17 results in all
Seed-Coder: Let the Code Model Curate Data for ItselfCapabilities
- HumanEval+ 78.7
- MBPP+ 71.2
- MBXP 75.3
- NaturalCodeBench 49.6
- FullStackBench 55.8
- CRUXEval
- Aider-Edit 57.1
- CanItEdit 50.5
- CodeEditorBench
- SWE-bench Verified 19.2
- Multi-SWE-bench mini 4
Dropped since the lab’s previous technical report: MME , MMBench , MMMU , MM-Vet , MathVista , MMVP , GenEval , WISE , GEdit-Bench , IntelligentBench , RISEBench , KRIS-Bench
- 7 May 2025
- HumanEval 92.1
- LiveCodeBench v6 30.3
- MultiPL-E 81.4
- Arena-Hard-Auto 97.1
- IFEval 89.4
- MATH-500 91
- GPQA Diamond 57.1
- MMLU-Pro 77.2
- RULER 96
- MMMU 66.1
- DocVQA 95.3
- AI2D 93.7
- ChartQA 82.6
- 6 May 2025 Gemini 2.5 Pro Google DeepMind Gemini 2.5 Pro Preview: even better coding performance (Google Developers Blog)
- WebDev Arena 1420
- Video-MME 84.8
Dropped since the last post: Humanity's Last Exam , GPQA Diamond , AIME 2025 , AIME 2024 , LiveCodeBench v5 , Aider Polyglot , SimpleQA , MMMU , Vibe-Eval , MRCR (Michelangelo, original) , Global MMLU Lite
- 30 Apr 2025
- AIME 2024 75.3
- AIME 2025 62.9
- Omni-MATH 76.6
- GPQA Diamond 65.8
- LiveCodeBench 53.8
- FlenQA 97.7
- IFEval 83.4
- Arena-Hard-Auto 73.3
- HumanEval+ 92.9
- MMLU-Pro 74.3
- Kitab
- PhiBench 70.6
Dropped since the last post: MATH-500
Its technical report adds 9 evaluations 21 results in all
Phi-4-reasoning Technical ReportCapabilities
- HMMT 2025 43.8
- Codeforces rating 1736
- AIME 1983-2024 83.1
- BA-Calendar 67.7
- 3SAT-Search 78
- TSP-Opt 37.5
- SpatialEval Maze-Nav 55.1
- SpatialEval Spatial-Map 73.7
Bias
- 30 Apr 2025
- MATH-500 94.6
- GPQA Diamond 52
Dropped since the last post: AIME 2024 , AIME 2025 , Omni-MATH , LiveCodeBench , FlenQA , IFEval , Arena-Hard-Auto , HumanEval+ , MMLU-Pro , Kitab , PhiBench
Its technical report adds 1 evaluation 1 new 15 dropped 3 results in all
Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in MathCapabilities
- AIME 2024 57.5
Dropped since the lab’s previous technical report: ARC-Challenge , ARC-Easy , OpenBookQA , BoolQ , HellaSwag , PIQA , WinoGrande , CommonSenseQA , TruthfulQA , TriviaQA , MMLU , HumanEval+ , GSM8K , IFEval , MT-Bench
- 30 Apr 2025 Amazon Nova Premier Amazon Amazon Nova Premier: Our most capable model for complex tasks and teacher for model distillation (AWS News Blog)
- MMLU 87.4
- GPQA Diamond 57.1
- AIME 2025 16
- MATH-500 82
- BigCodeBench-Hard 28.1
- MBXP 78.4
- IFEval 91.5
- MMMU 68
- OCRBench v2 56.9
- CharXiv 48.8
- EgoSchema 73.8
- TallyQA 61.5
- SimpleQA 86.3
- Berkeley Function Calling Leaderboard (BFCL) 63.7
- ScreenSpot 87.8
- SWE-bench Verified 42.4
Dropped since the last post: GroundUI
Its technical report adds 1 evaluation 17 results in all
Amazon Nova Premier: Technical Report and Model CardCapabilities
- CharXiv (DQ) 84.6
- 30 Apr 2025
- miniF2F 88.9
- PutnamBench 7.4
- ProverBench (DeepSeek) 40
Dropped since the last post: POPE , MME , MMBench , SEED-Bench , GQA , MMMU , MM-Vet , GenEval , DPG-Bench
Its technical report adds 3 evaluations 3 new 9 dropped 6 results in all
DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal DecompositionCapabilities
- ProofNet 37.1
- CombiBench 10
- FormalMATH 28.3
Dropped since the lab’s previous technical report: POPE , MME , MMBench , SEED-Bench , GQA , MMMU , MM-Vet , GenEval , DPG-Bench
- 30 Apr 2025
- AIME 2024 81.3
- AIME 2025 78
- Omni-MATH 81.9
- GPQA Diamond 68.9
- LiveCodeBench 53.1
- FlenQA 97.9
- IFEval 84.9
- Arena-Hard-Auto 79
- HumanEval+ 92.3
- MMLU-Pro 76
- Kitab
- PhiBench 74.2
Dropped since the last post: ARC-Challenge , ARC-Easy , OpenBookQA , BoolQ , HellaSwag , PIQA , WinoGrande , CommonSenseQA , TruthfulQA , TriviaQA , MMLU , GSM8K , MATH-500 , MT-Bench
Its technical report adds 9 evaluations 9 new 1 dropped 21 results in all
Phi-4-reasoning Technical ReportCapabilities
- HMMT 2025 53.6
- Codeforces rating 1723
- AIME 1983-2024 89.4
- BA-Calendar 65.6
- 3SAT-Search 70.9
- TSP-Opt 42.6
- SpatialEval Maze-Nav 53.4
- SpatialEval Spatial-Map 73.3
Bias
Dropped since the lab’s previous technical report: MATH-500
- 29 Apr 2025
- AlpacaEval 2.0 9.1
- BIG-Bench Hard 35
- DROP 34.6
- GSM8K 68.3
- IFEval 70.1
- MATH 20.7
- MMLU 40
- Tulu 3 Safety Eval 87.6
- PopQA 12.9
- TruthfulQA 48.7
Its technical report adds 0 evaluations 10 results in all
- 29 Apr 2025
- GPQA Diamond 54.4
- SuperGPQA 40.5
- DROP 78.7
- MMLU-Pro 58.6
- IFEval 61
- MATH-500 95.8
- AIME 2024 68.2
- AIME 2025 55.4
- LiveCodeBench v5 57.8
- LiveCodeBench v6 49.3
Its technical report adds 1 evaluation 11 results in all
- 29 Apr 2025
- MMLU-Redux 92.7
- GPQA Diamond 71.1
- C-Eval 89.6
- LiveBench 77.1
- IFEval 83.4
- Arena-Hard-Auto 95.6
- AlignBench 8.94
- Creative Writing v3 84.6
- WritingBench 8.03
- MATH-500 98
- AIME 2024 85.7
- AIME 2025 81.5
- ZebraLogic 80.3
- AutoLogi 89
- Berkeley Function Calling Leaderboard (BFCL) 70.8
- LiveCodeBench v5 70.7
- Codeforces rating 2056
- Multi-IF 71.9
- INCLUDE 78.7
- MMMLU 84.3
- MT-AIME2024 80.8
- PolyMath 54.7
- MLogiQA 77.1
- Aider Polyglot
Dropped since the last post: MMMU , MMMU-Pro , MathVista , MATH-Vision (MathVision) , HallusionBench , MMBench-EN v1.1 , AI2D , ChartQA , DocVQA , MMStar , MM-Vet , OCRBench , OCRBench v2 , CC-OCR , Video-MME , MVBench , MMBench-Video , LVBench , EgoSchema , Perception Test MCVQA , MLVU , TempCompass , ScreenSpot , ScreenSpot-Pro , Android in the Zoo (AitZ) , AndroidControl , AndroidWorld , MobileMiniWoB++ , OSWorld
Qwen3-1.7B: Its technical report adds 1 evaluation 1 new 58 dropped 23 results in all
Qwen3 Technical Report (arXiv 2505.09388)Capabilities
Dropped since the lab’s previous technical report: MMMU , MMMU-Pro , MathVista , MATH-Vision (MathVision) , MathVerse , MEGA-Bench , MMBench , MMBench-EN v1.1 , MMStar , MME , MuirBench , BLINK , CRPE , HallusionBench , MTVQA-Test , RealWorldQA , MME-RealWorld , MM-Vet , MM-MT-Bench , MMLU-Pro , MATH , GSM8K , HumanEval , MultiPL-E , CC-OCR , OmniDocBench , AI2D , TextVQA , DocVQA , InfographicVQA , ChartQA , CharXiv , CharXiv (DQ) , SEED-Bench-2-Plus , OCRBench , VCR (Visual Caption Restoration) , OCRBench v2 , RefCOCO / RefCOCO+ / RefCOCOg , ODinW-13 , CountBench , Video-MME , Video-MMMU , MMVU , MVBench , MMBench-Video , LongVideoBench , LVBench , EgoSchema , Perception Test MCVQA , MLVU , TempCompass , Charades-STA , ScreenSpot , ScreenSpot-Pro , AndroidControl , AndroidWorld , MobileMiniWoB++ , OSWorld
Qwen3-14B: Its technical report adds 2 evaluations 1 new 25 results in all
Its technical report adds 1 evaluation 1 dropped 24 results in all
Qwen3 Technical Report (arXiv 2505.09388)Capabilities
- RULER 95
Dropped since the lab’s previous technical report: Belebele
Qwen3-30B-A3B: Its technical report adds 1 evaluation 24 results in all
Qwen3 Technical Report (arXiv 2505.09388)Capabilities
- RULER 91.6
Qwen3-32B: Its technical report adds 6 evaluations 5 new 29 results in all
Qwen3 Technical Report (arXiv 2505.09388)Capabilities
- RULER 93.7
- Belebele
- LengthCtrl (Qwen) 73.5
- ThinkFollow (Qwen) 98.9
- ToolUse (Qwen) 85.5
Hallucination
- CounterFactQA (Qwen) 68.1
Qwen3-4B: Its technical report adds 2 evaluations 4 dropped 25 results in all
Qwen3 Technical Report (arXiv 2505.09388)Dropped since the lab’s previous technical report: CounterFactQA (Qwen) , LengthCtrl (Qwen) , ThinkFollow (Qwen) , ToolUse (Qwen)
Qwen3-8B: Its technical report adds 1 evaluation 1 dropped 24 results in all
Qwen3 Technical Report (arXiv 2505.09388)Capabilities
- RULER 89.1
Dropped since the lab’s previous technical report: Belebele
- 25 Apr 2025Kimi-Audio-7B-Instruct Moonshot AI Kimi-Audio GitHub README (evaluation section) and technical report
- LibriSpeech 1.85
- FLEURS 2.69
- AISHELL-1 0.6
- AISHELL-2 2.56
- WenetSpeech 5.83
- MMAU 65.2
- Clotho-AQA 71.2
- OpenAudioBench 69.1
- VoiceBench 76.9
Dropped since the last post: MATH-Vision (MathVision) , MathVista , MMMU
Its technical report adds 2 evaluations 2 new 3 dropped 11 results in all
Kimi-Audio Technical ReportCapabilities
Dropped since the lab’s previous technical report: MATH-Vision (MathVision) , MathVista , MMMU
- 17 Apr 2025
- Humanity's Last Exam 12.1
- GPQA Diamond 78.3
- AIME 2025 78
- AIME 2024 88
- LiveCodeBench v5 63.5
- Aider Polyglot 51.1
- SimpleQA 29.7
- MMMU 76.7
- Vibe-Eval 62
- MRCR (Michelangelo, original) 84.6
- Global MMLU Lite 88.4
Dropped since the last post: SWE-bench Verified
Its model card adds 5 evaluations 15 dropped 16 results in all
Gemini 2.5 Flash Preview Model CardSafety
Dropped since the lab’s previous model card: SWE-bench Verified , SecureBio VMQA, single choice , LAB-Bench ProtocolQA , LAB-Bench CloningScenarios , LAB-Bench SeqQA , WMDP Biology , WMDP Chemistry , CBRN open-ended questions , InterCode-CTF , In-house CTF suite (Phuong et al. 2024) , Hack the Box challenge set (Google DeepMind) , Cyber key skills benchmark v1 , RE-Bench , GDM Stealth , GDM Situational Awareness
- 16 Apr 2025
- AIME 2024 93.4
- AIME 2025 92.7
- Codeforces rating 2719
- GPQA Diamond 81.4
- Humanity's Last Exam 17.7
- MMMU 81.6
- MathVista 84.3
- SWE-Lancer 65792
- SWE-bench Verified 68.1
- Aider Polyglot 68.9
- MultiChallenge 43
- BrowseComp 28.3
- CharXiv 72
- tau-bench 57.4
Dropped since the last post: MMLU , MMMLU , Internal API instruction following (OpenAI) , COLLIE , IFEval , Multi-IF , OpenAI MRCR , GraphWalks , ComplexFuncBench
Its system card adds 33 evaluations 15 new 19 dropped 35 results in all
OpenAI o3 and o4-mini System CardCapabilities
- MMMLU 85.2
Hallucination
- SimpleQA 20
- SimpleQA Hallucination Rate 79
- PersonQA 36
Safety
Robustness
Bias
Bio and chem risk
Cyber risk
AI R&D risk
Autonomy
Dropped since the lab’s previous system card: XSTest , Production Jailbreaks , Jailbreak Augmented Examples , Discrim-Eval , Red Teaming Network pairwise safety comparison , Gray Swan Jailbreak Arena , Expert comparisons on biothreat information , Biotool and Wet Lab Actions , BioLP-bench , Tacit Knowledge Brainstorm (Open-Ended) , Structured expert probing campaign: radiological and nuclear , Contextual Nuclear Knowledge , Radiological and Nuclear Expert Knowledge , ChangeMyView , Political Persuasion Parallel Generation , MakeMePay , MakeMeSay , Agentic tasks (autonomous replication and adaptation) , MLE-bench
- 16 Apr 2025
- Arena-Hard-Auto 57.6
- AlpacaEval 2.0 62.7
- MMLU 65.5
- PopQA 26.2
- TruthfulQA 66.9
- BIG-Bench Hard 69.1
- DROP 59.4
- GSM8K 80.9
- HumanEval 89.7
- HumanEval+ 86.1
- IFEval 74.8
- AttaQ 88.5
- AIME 2024 8.12
- MATH-500 69
- 16 Apr 2025
- OSWorld 42.5
- Windows Agent Arena 42.1
- WebVoyager 84.8
- Online-Mind2Web 75.8
- AndroidWorld 64.2
- ScreenSpot-v2 94.2
- ScreenSpot-Pro 61.6
Dropped since the last post: AIME 2025 , AIME 2024 , Beyond AIME , GPQA Diamond , SuperGPQA , MMLU-Pro , Codeforces rating , LiveCodeBench v5 , Aider Polyglot , SWE-bench Verified , ARC-AGI , SimpleQA , COLLIE , IFEval
- 16 Apr 2025
- AIME 2024 91.6
- AIME 2025 88.9
- Codeforces rating 2706
- GPQA Diamond 83.3
- Humanity's Last Exam 24.9
- MMMU 82.9
- MathVista 86.8
- CharXiv 78.6
- SWE-Lancer 65250
- SWE-bench Verified 69.1
- Aider Polyglot 81.3
- MultiChallenge 56.5
- BrowseComp 49.7
- tau-bench 61.2
Dropped since the last post: IFEval , OpenAI MRCR , GraphWalks , Video-MME
Its system card adds 33 evaluations 11 new 12 dropped 35 results in all
OpenAI o3 and o4-mini System CardCapabilities
- MMMLU 88.8
Hallucination
- SimpleQA 49
- SimpleQA Hallucination Rate 51
- PersonQA 59
Safety
Robustness
Bias
Bio and chem risk
Cyber risk
AI R&D risk
Autonomy
Dropped since the lab’s previous system card: WildChat (toxic conversations) , XSTest , Challenging Red Teaming Evaluation 1 (created for o3-mini) , Challenging Red Teaming Evaluation 2 (created for deep research) , BioLP-bench , WMDP Biology , Contextual Nuclear Knowledge , Radiological and Nuclear Expert Knowledge , MakeMePay , MakeMeSay , Agentic tasks (autonomous replication and adaptation) , MLE-bench
- 15 Apr 2025
- ARC-Challenge 49.9
- ARC-Easy 74.8
- OpenBookQA 41.6
- BoolQ 80.2
- HellaSwag 68.4
- PIQA 77.1
- WinoGrande 71.9
- CommonSenseQA 71.6
- TruthfulQA 45.3
- TriviaQA 33.6
- MMLU 53.2
- HumanEval+ 38.4
- GSM8K 58.4
- MATH-500 43.4
- IFEval 53.5
- MT-Bench 5.85
Dropped since the last post: MMMU , MMBench , MMMU-Pro , ScienceQA , MathVista , Inter-GPS (Geometry3K) , AI2D , ChartQA , DocVQA , InfographicVQA , TextVQA , OCRBench , POPE , BLINK , Video-MME
Its technical report adds 0 evaluations 27 dropped 16 results in all
BitNet b1.58 2B4T Technical ReportDropped since the lab’s previous technical report: MMMU , MMMU-Pro , ScienceQA , MathVista , Inter-GPS (Geometry3K) , MMBench , POPE , AI2D , ChartQA , TextVQA , DocVQA , InfographicVQA , OCRBench , BLINK , Video-MME , Common Voice , FLEURS , Open ASR Leaderboard , CoVoST2 , Golden3 speech summarization (Microsoft) , AIR-Bench (Audio Instruction Benchmark) , MMAU , Harmful content defect rate (Azure AI Evaluation SDK, Microsoft) , XSTest , Microsoft internal multimodal RAI benchmark , RTVLM , VLGuard
- 14 Apr 2025
- AIME 2024 29.4
- GPQA Diamond 50.3
- MMLU 80.1
- MMMLU 66.9
- SWE-Lancer 9000
- Aider Polyglot 9.8
- Internal API instruction following (OpenAI) 31.6
- MultiChallenge 15
- COLLIE 42.5
- IFEval 74.5
- Multi-IF 57.2
- OpenAI MRCR 36.6
- GraphWalks 25
- MMMU 55.4
- MathVista 56.2
- CharXiv 40.5
- ComplexFuncBench 5.7
- tau-bench 18.3
Dropped since the last post: SWE-bench Verified
- 14 Apr 2025
- AIME 2024 49.6
- GPQA Diamond 65
- MMLU 87.5
- MMMLU 78.5
- SWE-bench Verified 23.6
- SWE-Lancer 31000
- Aider Polyglot 34.7
- Internal API instruction following (OpenAI) 45.1
- MultiChallenge 35.8
- COLLIE 54.6
- IFEval 84.1
- Multi-IF 67
- OpenAI MRCR 47.2
- GraphWalks 61.7
- MMMU 72.7
- MathVista 73.1
- CharXiv 56.8
- ComplexFuncBench 49.3
- tau-bench 45.9
Dropped since the last post: FrontierMath , Codeforces rating , LiveBench , MATH-500 , MGSM , SimpleQA
- 14 Apr 2025
- AIME 2024 80.8
- AIME 2025 63.6
- Omni-MATH 68.4
- LiveCodeBench v5 59.1
- SysBench 81.2
- IFEval 84.5
- Arena-Hard-Auto 90.6
- Berkeley Function Calling Leaderboard (BFCL) 70.5
- SuperGPQA 52.6
- GPQA Diamond 66.1
Dropped since the last post: GSM8K , MATH , BIG-Bench Hard , MMLU , C-Eval , CMMLU , MBPP , AGIEval
- 14 Apr 2025
- SWE-bench Verified 54.6
- Aider Polyglot 52.9
- MultiChallenge 38.3
- IFEval 87.4
- OpenAI MRCR 57.2
- GraphWalks 61.7
- Video-MME 72
- MMMU 74.8
- MathVista 72.2
- CharXiv 56.7
Dropped since the last post: SimpleQA , SimpleQA Hallucination Rate , GPQA Diamond , AIME 2024 , MMMLU , SWE-Lancer
- 10 Apr 2025Kimi-VL-A3B-Thinking Moonshot AI Kimi-VL-A3B-Thinking model card (Hugging Face) and Kimi-VL Technical Report
- MATH-Vision (MathVision) 36.8
- MathVista 71.3
- MMMU 61.7
Dropped since the last post: MMLU , MMLU-Pro , BIG-Bench Hard , TriviaQA , HumanEval , MBPP , GSM8K , MATH , CMATH , C-Eval , CMMLU
Its technical report adds 0 evaluations 3 results in all
- 10 Apr 2025
- AIME 2025 74
- AIME 2024 86.7
- Beyond AIME 48
- GPQA Diamond 77.3
- SuperGPQA 62.1
- MMLU-Pro 87
- Codeforces rating 55
- LiveCodeBench v5 64.9
- Aider Polyglot 54.2
- SWE-bench Verified 47
- ARC-AGI 39.9
- SimpleQA 12.9
- COLLIE 73.1
- IFEval 87.4
Its technical report adds 0 evaluations 14 results in all
- 8 Apr 2025
- 7 Apr 2025
- GPQA Diamond 76
- AIME 2025 72.5
- Berkeley Function Calling Leaderboard (BFCL) 74.1
- LiveCodeBench v5 66.3
- IFEval 89.5
- MATH-500 97
- JudgeBench 79.1
Dropped since the last post: RoboCasa , RoboCasa GR1 Tabletop Tasks
Its technical report adds 2 evaluations 9 results in all
Llama-Nemotron: Efficient Reasoning ModelsCapabilities
- AIME 2024 80.8
- Arena-Hard-Auto 87
- 5 Apr 2025
- MMMU 69.4
- MMMU-Pro 52.2
- MathVista 70.7
- ChartQA 88.8
- DocVQA 94.4
- LiveCodeBench v5 32.8
- MMLU-Pro 74.3
- GPQA Diamond 57.2
- MGSM 90.6
- MTOB 42.2
- MMLU 79.6
- MATH 50.3
- MBPP 67.8
- TydiQA 31.5
Dropped since the last post: LiveCodeBench
Its model card adds 1 evaluation 14 results in all
Llama 4 model card (MODEL_CARD.md, meta-llama/llama-models)Capabilities
- LiveCodeBench 32.8
- 5 Apr 2025 Llama 4 Maverick Meta Llama 4 model card (Hugging Face meta-llama/Llama-4-Maverick-17B-128E-Instruct)
- MMMU 73.4
- MMMU-Pro 59.6
- MathVista 73.7
- ChartQA 90
- DocVQA 94.4
- LiveCodeBench 43.4
- MMLU-Pro 80.5
- GPQA Diamond 69.8
- MGSM 92.3
- MTOB 54
- MMLU 85.5
- MATH 61.2
- MBPP 77.6
- TydiQA 31.7
Dropped since the last post: IFEval , HumanEval , Berkeley Function Calling Leaderboard (BFCL)
Its model card adds 0 evaluations 3 dropped 14 results in all
Llama 4 model card (MODEL_CARD.md, meta-llama/llama-models)Dropped since the lab’s previous model card: IFEval , HumanEval , Berkeley Function Calling Leaderboard (BFCL)
- 5 Apr 2025 Llama 4 Behemoth Meta The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation
- LiveCodeBench 49.4
- MATH-500 95
- MMLU-Pro 82.2
- GPQA Diamond 73.7
- MMMLU 85.8
- MMMU 76.1
- 31 Mar 2025
- ScreenSpot 90.9
- GroundUI 80.5
- 27 Mar 2025
- OmniBench 56.1
- LibriSpeech 2.6
- Common Voice 6.9
- FLEURS 3.55
- CoVoST2 34.7
- MMAU 65.6
- VoiceBench 74.1
- Seed-TTS-eval 2.32
- MMMU 59.2
- MMMU-Pro 36.6
- MathVista 67.9
- MATH-Vision (MathVision) 25
- MMBench-EN v1.1 81.8
- MM-Vet 66.8
- MMStar 64
- MME 2340
- MuirBench 59.2
- CRPE 76.5
- RealWorldQA 70.3
- MME-RealWorld 61.6
- MM-MT-Bench 6
- AI2D 83.2
- TextVQA 84.4
- DocVQA 95.2
- ChartQA 85.3
- OCRBench v2 57.8
- RefCOCO / RefCOCO+ / RefCOCOg 87.7
- ODinW-13 42.4
- Video-MME 72.4
- MVBench 70.3
- EgoSchema 68.6
- MMLU-Pro 47
- MMLU-Redux 71
- LiveBench 29.6
- GPQA Diamond 30.8
- MATH 71.5
- GSM8K 88.7
- HumanEval 78.7
- MBPP 73.2
- MultiPL-E 65.8
- LiveCodeBench 24.6
- WenetSpeech 6.8
Dropped since the last post: AIME 2024 , IFEval , Berkeley Function Calling Leaderboard (BFCL)
Its technical report adds 1 evaluation 1 new 7 dropped 43 results in all
Qwen2.5-Omni Technical ReportCapabilities
Dropped since the lab’s previous technical report: RULER , LV-Eval , LongBench-Chat , Passkey Retrieval , IFEval , Arena-Hard-Auto , MT-Bench
- 25 Mar 2025
- Humanity's Last Exam 18.8
- GPQA Diamond 84
- AIME 2025 86.7
- AIME 2024 92
- LiveCodeBench v5 70.4
- Aider Polyglot 74
- SWE-bench Verified 63.8
- SimpleQA 52.9
- MMMU 81.7
- Vibe-Eval 69.4
- MRCR (Michelangelo, original) 94.5
- Global MMLU Lite 89.8
Dropped since the last post: MMLU-Pro , Bird-SQL , FACTS Grounding , MATH , HiddenMath , CoVoST2 , EgoSchema
Its model card adds 19 evaluations 14 new 7 dropped 31 results in all
Gemini 2.5 Pro Preview Model Card (updated April 28, 2025)Safety
Alignment
- GDM Stealth 20
- GDM Situational Awareness 18.2
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous model card: MMLU-Pro , Bird-SQL , FACTS Grounding , MATH , HiddenMath , CoVoST2 , EgoSchema
- 24 Mar 2025
- MMLU-Pro 81.2
- GPQA Diamond 68.4
- AIME 2024 59.4
- LiveCodeBench 49.2
- MATH-500
Dropped since the last post: MMLU , MMLU-Redux , DROP , IFEval , SimpleQA , FRAMES , AlpacaEval 2.0 , Arena-Hard-Auto , Codeforces rating , SWE-bench Verified , Aider Polyglot , CNMO 2024 , CLUEWSC , C-Eval , Chinese SimpleQA
- 21 Mar 2025
- MMLU-Pro 87.2
- GPQA Diamond 69.3
- Chinese SimpleQA 67.9
- DROP 93.1
- ZebraLogic 79.6
- MATH-500 96.2
- AIME 2024 78.2
- LiveCodeBench 64.9
- C-Eval 91.8
- CMMLU 90
- Arena-Hard-Auto 91.9
- CFBench 81
- CELLO 76.4
- T-Eval 68.8
Dropped since the last post: MMLU , SimpleQA , BIG-Bench Hard , MATH , HumanEval , LiveBench , IFEval
- 18 Mar 2025 GR00T N1 NVIDIA GR00T N1: An Open Foundation Model for Generalist Humanoid Robots (NVIDIA white paper)
- RoboCasa 32.1
- RoboCasa GR1 Tabletop Tasks 50
Dropped since the last post: Arena-Hard-Auto , MATH-500 , AIME 2025 , GPQA Diamond , IFEval , Berkeley Function Calling Leaderboard (BFCL) , MBPP , MT-Bench
- 18 Mar 2025
- MATH-500 95.7
- AIME 2024 72.1
- AIME 2025 65.8
- CSAT Math 2025 94.5
- GPQA Diamond 66.1
- LiveCodeBench v5 59.5
Its technical report adds 2 evaluations 8 results in all
- 18 Mar 2025
- Arena-Hard-Auto 88.3
- MATH-500 96.6
- AIME 2025 58.4
- GPQA Diamond 66.7
- IFEval 89.2
- Berkeley Function Calling Leaderboard (BFCL) 73.7
- MBPP 91.3
- MT-Bench 9.17
Its technical report adds 3 evaluations 9 results in all
Llama-Nemotron: Efficient Reasoning ModelsCapabilities
- AIME 2024 67.5
- LiveCodeBench v5 45.5
- JudgeBench 69.7
- 17 Mar 2025
- MMLU 80.6
- MMLU-Pro 66.8
- MATH 69.3
- GPQA Main 44.4
- GPQA Diamond 46
- MBPP 74.7
- HumanEval 88.4
- SimpleQA 10.4
- MMMU 64
- MMMU-Pro 49.3
- MathVista 68.9
- ChartQA 86.2
- DocVQA 94.1
- AI2D 93.7
- MM-MT-Bench 7.3
- LongBench v2 37.2
- RULER 81.2
Dropped since the last post: MT-Bench , WildBench , Arena-Hard-Auto , IFEval , TriviaQA , ARC-Challenge , GSM8K , AGIEval
- 16 Mar 2025
- 13 Mar 2025
- Cohere enterprise human evaluation 51.7
- MMLU
- MATH
- IFEval
- tau-bench
- Berkeley Function Calling Leaderboard (BFCL)
- MBPP+
- RepoQA
Its technical report adds 35 evaluations 43 results in all
Command A: An Enterprise-Ready Large Language ModelCapabilities
- MMLU-Pro 69.6
- GPQA Diamond 50.8
- InFoBench 94.9
- ChatRAG Bench 72.9
- StrategyQA 76.7
- Bamboogle 76
- DROP 91.1
- HotpotQA 92.1
- NTREX-128 68.8
- FLORES-200 81.2
- m-ArenaHard
- m-tau-bench (Cohere)
- Language Confusion Benchmark 93
- AL-QASIDA
- LiveCodeBench v5 26.9
- BigCodeBench 45.4
- LBPP (Less Basic Python Problems) 51.5
- HumanEvalPack 76.2
- HumanEval 85.4
- HumanEval-COBOL (Cohere) 25.3
- SWE-bench Verified 26.8
- Aider Polyglot 14.7
- Spider 80.2
- Bird-SQL 59.5
- Multi-dialect SQL evaluation (Cohere) 55.3
- AIME 2024 23.3
- Enterprise generative evaluation (Cohere) 94.2
- Enterprise RAG evaluation (Cohere) 4.73
- RULER 95
- LongBench v2 43.4
- MGSM 90.1
- INCLUDE 74.3
- 13 Mar 2025
- AlpacaEval 2.0 42.8
- BIG-Bench Hard 70.6
- DROP 78
- GSM8K 87.6
- IFEval 85.6
- MATH 49.7
- MMLU 77.3
- Tulu 3 Safety Eval 85.9
- PopQA 37.5
- TruthfulQA 73.2
Dropped since the last post: HumanEval , HumanEval+
Its technical report adds 5 evaluations 5 new 2 dropped 15 results in all
2 OLMo 2 FuriousCapabilities
- AGIEval 68.3
- DeepMind Mathematics 34.7
- GPQA Diamond 35.9
- IFEval-OOD 33.1
- MMLU-Pro 52.7
Dropped since the lab’s previous technical report: HumanEval , HumanEval+
- 12 Mar 2025
- GPQA Diamond 42.4
- SimpleQA 10
- FACTS Grounding 74.9
- BIG-Bench Hard 87.6
- BIG-Bench Extra Hard 19.3
- IFEval 90.4
- HellaSwag 85.6
- BoolQ 82.4
- PIQA 83.3
- SIQA 54.9
- TriviaQA 85.5
- Natural Questions 36.1
- ARC-Challenge 70.6
- ARC-Easy 89
- WinoGrande 78.8
- DROP 77.2
- MMLU-Pro 67.5
- LiveCodeBench 29.7
- Bird-SQL 54.4
- MATH 89
- HiddenMath 60.3
- MBPP 74.4
- HumanEval 87.8
- Natural2Code 84.5
- GSM8K 95.9
- MMLU 78.6
- AGIEval 66.2
- Global MMLU Lite 75.7
- ECLeKTic 24.4
- WMT24++ 55.7
- MGSM 74.3
- Flores 48.8
- XQuAD 76.8
- IndicGenBench 63.4
- MMMU 64.9
- DocVQA 86.6
- InfographicVQA 70.6
- TextVQA 68.6
- AI2D 84.5
- ChartQA 78
- VQAv2 72.9
- MathVista 67.6
- COCO Captions 116
- RealWorldQA 53.9
- ReMI 44.8
- BLINK 39.6
- OK-VQA 60.2
- TallyQA 54.3
- SpatialSense VQA 59.4
- CountBench 68
Dropped since the last post: LiveCodeBench v5 , MRCR (Michelangelo, original) , CoVoST2 , EgoSchema
Gemma 3 12B: Its technical report adds 4 evaluations 4 new 8 dropped 54 results in all
Gemma 3 Technical Report (arXiv 2503.19786v1)Capabilities
- RULER 57.1
- MRCR (Michelangelo, original) 49.8
- Perception Test MCVQA 54.9
- ActivityNet-QA 50.4
Dropped since the lab’s previous technical report: LMArena (Elo) , RealToxicityPrompts , CrowS-Pairs , BBQ (Bias Benchmark for QA) , Winogender , TruthfulQA , WinoBias , ToxiGen
Gemma 3 1B: Its technical report adds 0 evaluations 21 dropped 33 results in all
Gemma 3 Technical Report (arXiv 2503.19786v1)Dropped since the lab’s previous technical report: MMMU , RULER , MRCR (Michelangelo, original) , DocVQA , InfographicVQA , TextVQA , AI2D , ChartQA , VQAv2 , MathVista , Perception Test MCVQA , ActivityNet-QA , AGIEval , COCO Captions , RealWorldQA , ReMI , BLINK , OK-VQA , TallyQA , SpatialSense VQA , CountBench
Its technical report adds 5 evaluations 5 new 55 results in all
Gemma 3 Technical Report (arXiv 2503.19786v1)Capabilities
- LMArena (Elo) 1338
- RULER 66
- MRCR (Michelangelo, original) 59.3
- Perception Test MCVQA 58.1
- ActivityNet-QA 52.8
Gemma 3 4B: Its technical report adds 4 evaluations 1 dropped 54 results in all
Gemma 3 Technical Report (arXiv 2503.19786v1)Capabilities
- RULER 46.8
- MRCR (Michelangelo, original) 44.6
- Perception Test MCVQA 50.6
- ActivityNet-QA 46.3
Dropped since the lab’s previous technical report: LMArena (Elo)
- 10 Mar 2025
- GPQA Diamond 61.1
- HumanEval 96.3
- LiveCodeBench v5 62
- MATH-500 90.6
- AIME 2024 70
- AIME 2025 50
- MMLU-Pro 65
- Berkeley Function Calling Leaderboard (BFCL) 77.2
- 6 Mar 2025
- 4 Mar 2025 Aya Vision 32B Cohere Cohere For AI Launches Aya Vision, a Multilingual Multimodal AI Model (Cohere blog)
- AyaVisionBench 64
- m-WildVision 72
Its technical report adds 11 evaluations 13 results in all
Aya Vision: Advancing the Frontier of Multilingual MultimodalityCapabilities
- MaXM 62.3
- xMMMU 45.1
- CVQA 74.1
- MTVQA-Test 23.5
- Kaleidoscope 41.7
- xChatBench 70.1
- m-ArenaHard
- Global MMLU Lite 63.6
- MGSM 79.5
- Flores 37.8
- IFEval 78.5
- 27 Feb 2025
OpenVLA-OFT Stanford University Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (OpenVLA-OFT) - LIBERO 97.1
- 27 Feb 2025
- MMLU 89.5
- MMLU-Pro 79
- GPQA Diamond 57.5
- SimpleQA 22.8
- Chinese SimpleQA 70.8
- BIG-Bench Hard 92.2
- DROP 91.5
- ZebraLogic 46
- MATH 89.7
- AIME 2024 43.3
- HumanEval 91
- LiveCodeBench 32
- C-Eval 90.9
- CMMLU 90.8
- LiveBench 61
- Arena-Hard-Auto 88.6
- IFEval 88.6
- 27 Feb 2025
- SimpleQA 62.5
- SimpleQA Hallucination Rate 37.1
- GPQA Diamond 71.4
- AIME 2024 36.7
- MMMLU 85.1
- MMMU 74.4
- SWE-Lancer 32.6
- SWE-bench Verified 38
Dropped since the last post: Humanity's Last Exam , GAIA
Its system card adds 33 evaluations 7 new 4 dropped 36 results in all
OpenAI GPT-4.5 System CardHallucination
- PersonQA 78
Safety
Robustness
Bio and chem risk
Cyber risk
Autonomy
Dropped since the lab’s previous system card: Deep research prompt injection evaluation , Red Teaming Network pairwise safety comparison , Personal data evaluations , ChangeMyView
- 26 Feb 2025
- MMMU 55.1
- MMBench 86.7
- MMMU-Pro 38.5
- ScienceQA 97.5
- MathVista 62.4
- Inter-GPS (Geometry3K) 48.6
- AI2D 82.3
- ChartQA 81.4
- DocVQA 93.2
- InfographicVQA 72.7
- TextVQA 75.6
- OCRBench 84.4
- POPE 85.6
- BLINK 61.3
- Video-MME 55
Dropped since the last post: Arena-Hard-Auto , BIG-Bench Hard , MMLU , MMLU-Pro , ARC-Challenge , BoolQ , GPQA Diamond , HellaSwag , OpenBookQA , PIQA , SIQA , TruthfulQA , WinoGrande , MMMLU , MGSM , GSM8K , MATH
Its technical report adds 12 evaluations 10 new 30 dropped 27 results in all
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAsCapabilities
- Common Voice 6.8
- FLEURS 4
- Open ASR Leaderboard 6.14
- CoVoST2 39.3
- Golden3 speech summarization (Microsoft) 6.28
- AIR-Bench (Audio Instruction Benchmark) 6.98
- MMAU 55.6
Safety
Dropped since the lab’s previous technical report: BIG-Bench Hard , MMLU , MMLU-Pro , ARC-Challenge , BoolQ , GPQA Diamond , HellaSwag , OpenBookQA , PIQA , SIQA , TruthfulQA , WinoGrande , MMMLU , MGSM , GSM8K , MATH , Qasper , SQuALITY , IFEval , Berkeley Function Calling Leaderboard (BFCL) , HumanEval , MBPP , BigCodeBench , HumanEval+ , LiveCodeBench , LiveBench , MBPP+ , Spider , AIME 2024 , MATH-500
- 26 Feb 2025
- Arena-Hard-Auto 32.8
- BIG-Bench Hard 70.4
- MMLU 67.3
- MMLU-Pro 52.8
- ARC-Challenge 83.7
- BoolQ 81.2
- GPQA Diamond 30.4
- HellaSwag 69.1
- OpenBookQA 79.2
- PIQA 77.6
- SIQA 72.5
- TruthfulQA 66.4
- WinoGrande 67
- MMMLU 49.3
- MGSM 63.9
- GSM8K 88.6
- MATH 64
Dropped since the last post: HumanEval , SimpleQA , DROP , LiveBench , IFEval , PhiBench
Its technical report adds 16 evaluations 12 new 11 dropped 32 results in all
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAsCapabilities
- Qasper 40.4
- SQuALITY 22.8
- IFEval 70.1
- Berkeley Function Calling Leaderboard (BFCL) 70.3
- HumanEval 74.4
- MBPP 65.3
- BigCodeBench 43
- HumanEval+ 68.3
- LiveCodeBench 19.9
- LiveBench
- MBPP+ 63.8
- Spider 42.2
- AIME 2024 10
- MATH-500 71.8
Safety
Dropped since the lab’s previous technical report: SimpleQA , DROP , Arena-Hard-Auto , PhiBench , AMC 10/12 (November 2024) , HELMET , Microsoft RAI benchmark: Grounding , Microsoft RAI benchmark: Third Party Harm (DR-1) , Microsoft RAI benchmark: Harmful Content Continuation (DR-3) , Microsoft RAI benchmark: Harmful Content Summarization (DR-3) , Microsoft RAI benchmark: Jailbreak (DR-1)
- 26 Feb 2025
- Arena-Hard-Auto 55.3
- AlpacaEval 2.0 61.2
- MMLU 66.8
- PopQA 28
- TruthfulQA 66.9
- BIG-Bench Hard 64.8
- DROP 51
- GSM8K 81.7
- HumanEval 89.3
- HumanEval+ 85.7
- IFEval 74.3
- AttaQ 85.4
- 26 Feb 2025Mercury Coder Small Inception Labs Introducing Mercury, the first commercial-scale diffusion large language model (Inception blog)
- HumanEval 90
- MBPP 76.6
- EvalPlus 80.4
- MultiPL-E 76.2
- LiveCodeBench 25
- BigCodeBench 45.5
Its technical report adds 1 evaluation 7 results in all
Mercury: Ultra-Fast Language Models Based on DiffusionCapabilities
- HumanEval Infilling 84.8
- 24 Feb 2025
- GPQA Diamond 84.8
- SWE-bench Verified 70.3
- tau-bench 69.8
- MMMLU 86.1
- MMMU 75
- IFEval 93.2
- MATH-500 96.2
- AIME 2024 80
Dropped since the last post: OSWorld-Verified , MathVista , AI2D , ChartQA , DocVQA , MMLU , MMLU-Pro , MATH , HumanEval , MGSM , DROP , BIG-Bench Hard
Its system card adds 28 evaluations 28 results in all
Claude 3.7 Sonnet System CardSafety
Robustness
Bias
Alignment
Monitorability
Bio and chem risk
Cyber risk
- 22 Feb 2025
- 19 Feb 2025
- AIME 2025 90.8
- AIME 2024 95.8
- GPQA Diamond 84
- LiveCodeBench 80.4
- MMLU-Pro 78.9
- LOFT 83.1
- SimpleQA 21.7
- MMMU 69.4
- EgoSchema 74.3
- 19 Feb 2025
- AIME 2025 93.3
- AIME 2024 93.3
- GPQA Diamond 84.6
- LiveCodeBench 79.4
- MMMU 78
- MMLU-Pro 79.9
- LOFT 83.3
- SimpleQA 43.6
- EgoSchema 74.5
- LMArena (Elo) 1402
Dropped since the last post: MMLU , MATH , HumanEval , MathVista , DocVQA
- 17 Feb 2025
Dropped since the last post: HumanEval , MBPP , CRUXEval , LiveCodeBench , RepoBench , Spider , CanItEdit , HumanEval Multi
- 11 Feb 2025
- Answer Factuality (Perplexity) 85.1
- Readability (Perplexity) 85.9
- IFEval 86.8
- MMLU 87.1
Dropped since the last post: SimpleQA
- 5 Feb 2025
- MMLU-Pro 71.6
- LiveCodeBench v5 28.9
- Bird-SQL 57.4
- GPQA Diamond 51.5
- SimpleQA 21.7
- FACTS Grounding 83.6
- Global MMLU Lite 78.2
- MATH 86.8
- HiddenMath 55.3
- MRCR (Michelangelo, original) 58
- MMMU 68
- CoVoST2 38.4
- EgoSchema 67.2
Dropped since the last post: MMLU , HellaSwag , PIQA , SIQA , BoolQ , WinoGrande , ARC-Easy , ARC-Challenge , TriviaQA , Natural Questions , HumanEval , MBPP , GSM8K , AGIEval , DROP , BIG-Bench Hard
Its model card adds 5 evaluations 18 results in all
- 5 Feb 2025
- MMLU-Pro 77.6
- LiveCodeBench v5 34.5
- Bird-SQL 58.7
- GPQA Diamond 60.1
- SimpleQA 29.9
- FACTS Grounding 84.6
- Global MMLU Lite 83.4
- MATH 90.9
- HiddenMath 63.5
- MRCR (Michelangelo, original) 70.5
- MMMU 71.7
- CoVoST2 39
- EgoSchema 71.1
Its model card adds 5 evaluations 18 results in all
- 5 Feb 2025
- MMLU-Pro 79.1
- LiveCodeBench v5 36
- Bird-SQL 59.3
- GPQA Diamond 64.7
- SimpleQA 44.3
- FACTS Grounding 82.8
- Global MMLU Lite 86.5
- MATH 91.8
- HiddenMath 65.2
- MRCR (Michelangelo, original) 74.7
- MMMU 72.7
- CoVoST2 40.6
- EgoSchema 71.9
Dropped since the last post: AIME 2024
- 4 Feb 2025π0-FAST Physical Intelligence openpi LIBERO example results, released with Open Sourcing π0 (GitHub)
- LIBERO 85.5
- 4 Feb 2025
- LIBERO 94.2
- 2 Feb 2025
- Humanity's Last Exam 26.6
- GAIA 67.4
Dropped since the last post: OSWorld , WebArena , WebVoyager
Its system card adds 32 evaluations 28 new 9 dropped 32 results in all
Deep Research System CardHallucination
- PersonQA 86
Safety
Robustness
Bio and chem risk
Cyber risk
Autonomy
Persuasion
- ChangeMyView 84.1
- MakeMePay 36
- MakeMeSay 24
Dropped since the lab’s previous system card: Production Jailbreaks , Jailbreak Augmented Examples , Human Sourced Jailbreaks , Agentic harmful task refusals , User Confirmations During Computer Use , Proactive refusals of high-risk tasks , Operator prompt injection scenarios , Biotool and Wet Lab Actions , OSWorld
- 31 Jan 2025
- AIME 2024 87.3
- GPQA Diamond 79.7
- FrontierMath 9.2
- Codeforces rating 2130
- SWE-bench Verified 49.3
- LiveBench 84.6
- MMLU 86.9
- MATH-500 97.9
- MGSM 92
- SimpleQA 13.8
Dropped since the last post: DROP , MATH , HumanEval , MMMU , MathVista
Its system card adds 37 evaluations 38 results in all
OpenAI o3-mini System CardCapabilities
Hallucination
- PersonQA 21.7
Safety
Robustness
Bio and chem risk
- Long-form Biological Risk Questions
- Expert comparisons on biothreat information
- Biotool and Wet Lab Actions
- Multimodal Troubleshooting Virology 53
- BioLP-bench 41
- ProtocolQA Open-Ended 23
- Tacit Knowledge and Troubleshooting 58
- Tacit Knowledge Brainstorm (Open-Ended) 51
- Structured expert probing campaign: radiological and nuclear
- Contextual Nuclear Knowledge 73
- Radiological and Nuclear Expert Knowledge 60
Cyber risk
Autonomy
Persuasion
- ChangeMyView 82.9
- Political Persuasion Parallel Generation 35.5
- MakeMePay 1
- MakeMeSay 62
- 30 Jan 2025
- MMLU 87
- PopQA 55.5
- BIG-Bench Hard 88.6
- MATH 67.3
- GSM8K 95.5
- HumanEval 95.9
- HumanEval+ 92.9
- IFEval 86
- AlpacaEval 2.0 51.4
- Tulu 3 Safety Eval 86.7
Its technical report adds 0 evaluations 10 results in all
- 30 Jan 2025 Mistral Small 3 Mistral AI Mistral Small 3 (launch post) + HF model cards Mistral-Small-24B-Instruct-2501 / Base-2501
- MMLU-Pro 66.3
- GPQA Diamond 45.3
- HumanEval 84.8
- MATH 70.6
- MT-Bench 8.35
- WildBench 52.3
- Arena-Hard-Auto 87.3
- IFEval 82.9
- MMLU 80.7
- TriviaQA 80.3
- ARC-Challenge 91.3
- MBPP 69.6
- GSM8K 80.7
- AGIEval 65.8
Dropped since the last post: HellaSwag , WinoGrande , PIQA , ARC-Easy , Natural Questions
- 28 Jan 2025
- MMMU 70.2
- MMMU-Pro 51.1
- MathVista 74.8
- MATH-Vision (MathVision) 38.1
- HallusionBench 55.2
- MMBench-EN v1.1 88
- AI2D 88.4
- ChartQA 89.5
- DocVQA 96.4
- MMStar 70.8
- MM-Vet 76.2
- OCRBench 88.5
- OCRBench v2 61.5
- CC-OCR 79.8
- Video-MME 79.1
- MVBench 70.4
- MMBench-Video 2.02
- LVBench 47.3
- EgoSchema 76.2
- Perception Test MCVQA 73.2
- MLVU 74.6
- TempCompass 74.8
- ScreenSpot 87.1
- ScreenSpot-Pro 43.6
- Android in the Zoo (AitZ) 83.2
- AndroidControl 67.4
- AndroidWorld 35
- MobileMiniWoB++ 68
- OSWorld 8.83
Dropped since the last post: Arena-Hard-Auto , MMLU-Pro , GPQA Diamond , LiveCodeBench , LiveBench , MMLU , BIG-Bench Hard , C-Eval , CMMLU , HumanEval , MBPP , CRUXEval , GSM8K , MATH
Its technical report adds 34 evaluations 25 new 18 dropped 62 results in all
Qwen2.5-VL Technical ReportCapabilities
- MathVerse 57.6
- MEGA-Bench 51.3
- MMBench 88.6
- MME 2448
- MuirBench 70.7
- BLINK 64.4
- CRPE 79.2
- MTVQA-Test 31.7
- RealWorldQA 75.7
- MME-RealWorld 63.2
- MM-MT-Bench 7.6
- MMLU-Pro 71.2
- MMLU-Redux 85.9
- LiveBench 57
- GPQA Diamond 49
- MATH 83
- GSM8K 95.3
- HumanEval 87.8
- MultiPL-E 79.5
- IFEval 86.3
- OmniDocBench
- TextVQA 83.5
- InfographicVQA 87.3
- CharXiv 49.7
- CharXiv (DQ) 87.4
- SEED-Bench-2-Plus 73
- VCR (Visual Caption Restoration) 79.8
- RefCOCO / RefCOCO+ / RefCOCOg
- ODinW-13 43.1
- CountBench 93.6
- Video-MMMU 60.2
- MMVU 62.9
- LongVideoBench 60.7
- Charades-STA 50.9
Dropped since the lab’s previous technical report: MBPP , LiveCodeBench , Arena-Hard-Auto , MT-Bench , Multilingual IFEval , AMMLU , JMMLU , KMMLU , IndoMMLU , TurkishMMLU , Okapi MMLU , MGSM , BLEnD , Qwen in-house English automatic evaluation , Qwen in-house Chinese automatic evaluation , RULER , LV-Eval , LongBench-Chat
- 28 Jan 2025
- Arena-Hard-Auto
- MMLU-Pro
- GPQA Diamond
- LiveCodeBench
- LiveBench
- MMLU
- BIG-Bench Hard
- C-Eval
- CMMLU
- HumanEval
- MBPP
- CRUXEval
- GSM8K
- MATH
Dropped since the last post: MMLU-Redux , MultiPL-E , IFEval , AlignBench , MT-Bench , Multilingual IFEval , AMMLU , JMMLU , KMMLU , IndoMMLU , TurkishMMLU , Okapi MMLU , MGSM , BLEnD , ARC-Challenge , TruthfulQA , WinoGrande , HellaSwag , TheoremQA , MMLU-stem , Multilingual Exams , Multilingual Understanding , Flores
- 27 Jan 2025
Qwen2.5-14B-Instruct-1M Qwen Qwen2.5-1M: Deploy Your Own Qwen with Context Length up to 1M Tokens (blog) - RULER 95.7
- LV-Eval 49.8
- LongBench-Chat 8.76
- Passkey Retrieval
Its technical report adds 13 evaluations 17 results in all
Qwen2.5-1M Technical ReportCapabilities
- MMLU-Pro 63.3
- MMLU-Redux 80.7
- LiveBench 44.6
- GPQA Diamond 39.9
- MATH 79.5
- GSM8K 94.8
- HumanEval 88.4
- MBPP 80.2
- MultiPL-E 77.1
- LiveCodeBench 38.6
- IFEval 84.3
- Arena-Hard-Auto 70.2
- MT-Bench 8.89
- 27 Jan 2025 Janus-Pro-7B DeepSeek Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling (GitHub README and tech report)
- POPE 87.4
- MME 1567.1
- MMBench 79.2
- SEED-Bench 72.1
- GQA 62
- MMMU 41
- MM-Vet 50
- GenEval 80
- DPG-Bench 84.2
Its technical report adds 0 evaluations 9 results in all
- 23 Jan 2025
- OSWorld 38.1
- WebArena 58.1
- WebVoyager 87
Dropped since the last post: GPQA Diamond , MMLU , SWE-bench Verified , LiveBench , MATH-500 , AIME 2024 , MGSM , MMMU , MathVista , SimpleQA , tau-bench
Its system card adds 12 evaluations 4 new 39 dropped 13 results in all
Operator System CardSafety
Robustness
Bio and chem risk
Dropped since the lab’s previous system card: WildChat (toxic conversations) , XSTest , Image input safety evaluations , Regurgitation Evaluations , SimpleQA , SimpleQA Hallucination Rate , PersonQA , BBQ (Bias Benchmark for QA) , Discrim-Eval , Instruction Hierarchy Evaluation , CoT Deception Monitoring , Red Teaming Network pairwise safety comparison , Gray Swan Jailbreak Arena , Apollo Research in-context scheming evaluations , METR general autonomy task suite , Capture the Flag (High School) , Capture the Flag (Collegiate) , Capture the Flag (Professional) , Long-form Biological Risk Questions , Expert comparisons on biothreat information , Multimodal Troubleshooting Virology , ProtocolQA Open-Ended , BioLP-bench , Tacit Knowledge and Troubleshooting , Tacit Knowledge Brainstorm (Open-Ended) , Structured expert probing campaign: chem-bio novel design , Structured expert probing campaign: radiological and nuclear , Contextual Nuclear Knowledge , Radiological and Nuclear Expert Knowledge , ChangeMyView , Political Persuasion Parallel Generation , MakeMePay , MakeMeSay , OpenAI Research Engineer Interview: Multiple Choice , OpenAI Research Engineer Interview: Coding , SWE-bench Verified , MLE-bench , MMMLU , QuantBench
- 21 Jan 2025
- AIME 2024 73.3
- GPQA Diamond 74.2
- MMMU 75.4
Dropped since the last post: MMLU-Pro , Natural2Code , Bird-SQL , LiveCodeBench , FACTS Grounding , MATH , HiddenMath , MRCR (Michelangelo, original) , Vibe-Eval , CoVoST2 , EgoSchema
- 21 Jan 2025
- 20 Jan 2025Kimi k1.5 Moonshot AI Kimi k1.5: Scaling Reinforcement Learning with LLMs (tech report, arXiv 2501.12599, Tables 2 and 3)
- MATH-500 96.2
- AIME 2024 77.5
- Codeforces rating 94
- LiveCodeBench 62.5
- MathVista 74.9
- MMMU 70
- MATH-Vision (MathVision) 38.6
- MMLU 87.4
- IFEval 87.2
- CLUEWSC 91.7
- C-Eval 88.3
- HumanEval Multi 81.5
Its technical report adds 0 evaluations 12 results in all
- 20 Jan 2025
- MMLU 90.8
- MMLU-Redux 92.9
- MMLU-Pro 84
- DROP 92.2
- IFEval 83.3
- GPQA Diamond 71.5
- SimpleQA 30.1
- FRAMES 82.5
- AlpacaEval 2.0 87.6
- Arena-Hard-Auto 92.3
- LiveCodeBench 65.9
- Codeforces rating 2029
- SWE-bench Verified 49.2
- Aider Polyglot 53.3
- AIME 2024 79.8
- MATH-500 97.3
- CNMO 2024 78.8
- CLUEWSC 92.8
- C-Eval 91.8
- Chinese SimpleQA 63.7
Dropped since the last post: LongBench v2 , HumanEval Multi , Aider-Edit , Pile-test , BIG-Bench Hard , ARC-Easy , ARC-Challenge , HellaSwag , PIQA , WinoGrande , RACE-M , RACE-H , TriviaQA , Natural Questions , AGIEval , HumanEval , MBPP , CRUXEval , GSM8K , MATH , MGSM , CMATH , CMMLU , CMRC , C3 , CCPM , MMMLU , Needle In A Haystack
DeepSeek-R1-Distill-Llama-70B: Its technical report adds 0 evaluations 20 dropped 5 results in all
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948)Dropped since the lab’s previous technical report: MMLU , MMLU-Redux , MMLU-Pro , DROP , IFEval , SimpleQA , FRAMES , LongBench v2 , HumanEval Multi , SWE-bench Verified , Aider-Edit , Aider Polyglot , CNMO 2024 , CLUEWSC , C-Eval , Chinese SimpleQA , Arena-Hard-Auto , AlpacaEval 2.0 , RewardBench , Needle In A Haystack
DeepSeek-R1-Distill-Qwen-1.5B: Its technical report adds 0 evaluations 5 results in all
DeepSeek-R1-Distill-Qwen-14B: Its technical report adds 0 evaluations 5 results in all
DeepSeek-R1-Distill-Qwen-32B: Its technical report adds 0 evaluations 5 results in all
Its technical report adds 11 evaluations 11 new 31 results in all
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948)Capabilities
- AMC 12 2024 143.7
- AIME 2025 11.3
Safety
- SimpleSafetyTests 97.5
- Anthropic Red Team 96.2
- XSTest 95.3
- Do-Not-Answer 94.8
- HarmBench 35
- DeepSeek in-house safety benchmark 25.2
- DeepSeek multilingual safety evaluation 74.2
Robustness
Bias
- 15 Jan 2025
- MMLU 88.5
- MMLU-Pro 75.7
- SimpleQA 23.7
- Chinese SimpleQA 67.4
- IFEval 89.1
- Arena-Hard-Auto 89.1
- GPQA Diamond 54.4
- DROP 87.8
- GSM8K 94.8
- MATH 77.4
- MBPP 71.7
- HumanEval 86.9
- RULER 91
- LongBench v2 56.5
- MTOB 51.7
Its technical report adds 11 evaluations 25 results in all
MiniMax-01: Scaling Foundation Models with Lightning AttentionCapabilities
- EvalPlus 71.7
- Needle In A Haystack
- MR-NIAH (Multi-Round Needles-In-A-Haystack)
- General Assistant (MiniMax in-house) 73.9
- Hard Capability (MiniMax in-house) 64.8
- Creative Writing (MiniMax in-house) 81.3
- Knowledge Q&A (MiniMax in-house) 78.6
- Instruction Following (MiniMax in-house) 46.3
- Coding (MiniMax in-house) 90.2
- Long Context (MiniMax in-house) 93.8
Safety
- 13 Jan 2025
- HumanEval 86.6
- MBPP 80.2
- CRUXEval 55.5
- LiveCodeBench 37.9
- RepoBench 38
- Spider 66.5
- CanItEdit 50.5
- HumanEval Multi 71.4
Dropped since the last post: MMLU , MT-Bench , WildBench , Arena-Hard-Auto , MultiPL-E , GSM8K , MATH
- 13 Jan 2025
- CMMLU 83.1
- MMLU 76.6
- MMLU-Pro 57.6
- GPQA Diamond 37.4
- DROP 83.1
- HellaSwag 91.2
- KOR-Bench 56.4
- MATH-500 83
- AIME 2024 20
- LiveCodeBench 17.8
- HumanEval 82.3
- IFEval 79.3
- RULER 87.9
- AlpacaEval 2.0 51.1
- WildBench 33.1
- MT-Bench-101 8.59
2024
- 26 Dec 2024
- MMLU 88.5
- MMLU-Redux 89.1
- MMLU-Pro 75.9
- DROP 91.6
- IFEval 86.1
- GPQA Diamond 59.1
- SimpleQA 24.9
- FRAMES 73.3
- LongBench v2 48.7
- HumanEval Multi 82.6
- LiveCodeBench 40.5
- Codeforces rating 51.6
- SWE-bench Verified 42
- Aider-Edit 79.7
- Aider Polyglot 49.6
- AIME 2024 39.2
- MATH-500 90.2
- CNMO 2024 43.2
- CLUEWSC 90.9
- C-Eval 86.5
- Chinese SimpleQA 64.8
- Arena-Hard-Auto 85.5
- AlpacaEval 2.0 70
- Pile-test 0.548
- BIG-Bench Hard 87.5
- ARC-Easy 98.9
- ARC-Challenge 95.3
- HellaSwag 88.9
- PIQA 84.7
- WinoGrande 84.9
- RACE-M 67.1
- RACE-H 51.3
- TriviaQA 82.9
- Natural Questions 40
- AGIEval 79.6
- HumanEval 65.2
- MBPP 75.4
- CRUXEval 69.8
- GSM8K 89.3
- MATH 61.6
- MGSM 79.8
- CMATH 90.7
- CMMLU 88.8
- CMRC 76.3
- C3 78.6
- CCPM 92
- MMMLU 79.4
- Needle In A Haystack
Dropped since the last post: AlignBench , MT-Bench , DS-FIM-Eval , DS-Arena-Code
Its technical report adds 1 evaluation 1 new 16 dropped 25 results in all
DeepSeek-V3 Technical Report (arXiv 2412.19437)Capabilities
- RewardBench 87
Dropped since the lab’s previous technical report: TriviaQA , Natural Questions , ARC-Easy , ARC-Challenge , BIG-Bench Hard , AGIEval , HumanEval , MBPP , CRUXEval , GSM8K , MATH , CMATH , CMMLU , MT-Bench , AlignBench , SuperCLUE-Math6
- 17 Dec 2024
- GPQA Diamond 75.7
- MMLU 91.8
- SWE-bench Verified 48.9
- LiveBench 76.6
- MATH-500 96.4
- AIME 2024 79.2
- MGSM 89.3
- MMMU 77.3
- MathVista 71
- SimpleQA 42.6
- tau-bench 63.9
Dropped since the last post: Codeforces rating
Its system card adds 45 evaluations 15 new 7 dropped 47 results in all
OpenAI o1 System CardCapabilities
- MMMLU
- QuantBench 60
Hallucination
Safety
Robustness
Bio and chem risk
- Long-form Biological Risk Questions
- Expert comparisons on biothreat information
- Biotool and Wet Lab Actions
- Multimodal Troubleshooting Virology 59
- ProtocolQA Open-Ended 24
- BioLP-bench 33
- Tacit Knowledge and Troubleshooting 66
- Tacit Knowledge Brainstorm (Open-Ended) 51
- Structured expert probing campaign: chem-bio novel design
- Structured expert probing campaign: radiological and nuclear
- Contextual Nuclear Knowledge 74
- Radiological and Nuclear Expert Knowledge 70
Cyber risk
AI R&D risk
- MLE-bench 14
Autonomy
Persuasion
- ChangeMyView 89.1
- Political Persuasion Parallel Generation 47.1
- MakeMePay 27
- MakeMeSay 42
Dropped since the lab’s previous system card: External red team jailbreaks , BirthdayFacts , Open Ended Questions (hallucination) , Apollo Research strategic deception evaluations , RE-Bench , LAB-Bench ProtocolQA , LAB-Bench CloningScenarios
- 12 Dec 2024
- MMLU 84.8
- GPQA Diamond 56.1
- MATH 80.4
- HumanEval 82.6
- MGSM 80.6
- SimpleQA 3
- DROP 75.5
- MMLU-Pro 70.4
- Arena-Hard-Auto 75.4
- LiveBench 47.6
- IFEval 63
- PhiBench 56.2
Dropped since the last post: BIG-Bench Hard , ARC-Challenge , BoolQ , HellaSwag , OpenBookQA , PIQA , SIQA , TruthfulQA , WinoGrande , MMMLU , GSM8K , Qasper , SQuALITY , MBPP
Its technical report adds 8 evaluations 4 new 19 dropped 20 results in all
Phi-4 Technical ReportCapabilities
- HumanEval+ 82.8
- AMC 10/12 (November 2024) 91.8
- HELMET
Hallucination
Safety
Robustness
Dropped since the lab’s previous technical report: HellaSwag , ANLI , GSM8K , MedQA , AGIEval , TriviaQA , ARC-Challenge , ARC-Easy , PIQA , SIQA , BIG-Bench Hard , WinoGrande , OpenBookQA , BoolQ , CommonSenseQA , TruthfulQA , MBPP , MT-Bench , Microsoft RAI benchmark: Ungroundedness
- 11 Dec 2024
- MMLU-Pro 76.4
- Natural2Code 92.9
- Bird-SQL 56.9
- LiveCodeBench 35.1
- FACTS Grounding 83.6
- MATH 89.7
- HiddenMath 63
- GPQA Diamond 62.1
- MRCR (Michelangelo, original) 69.2
- MMMU 70.7
- Vibe-Eval 56.3
- CoVoST2 39.2
- EgoSchema 71.5
Dropped since the last post: PhysicsFinals , Functional MATH , AMC , GSM8K , BIG-Bench Hard , DROP , MMLU , HellaSwag , HumanEval , WMT23 , MGSM , MathVista , AI2D , ChemicalDiagramQA , ChartQA , BetterChartQA , DocVQA , DUDE , TAT-DQA , InfographicVQA , TextVQA , VQAv2 , V* , BLINK , RealWorldQA , VATEX , VATEX ZH , YouCook2 , ActivityNet-QA , OpenEQA
- 6 Dec 2024
- MMLU 86
- MMLU-Pro 68.9
- IFEval 92.1
- GPQA Diamond 50.5
- HumanEval 88.4
- MBPP 87.6
- MATH 77
- Berkeley Function Calling Leaderboard (BFCL) 77.3
- MGSM 91.1
Dropped since the last post: GSM8K , ARC-Challenge , Nexus , ZeroSCROLLS/QuALITY , InfiniteBench/En.MC , Needle In A Haystack
Its model card adds 0 evaluations 16 dropped 9 results in all
Llama 3.3 model card (MODEL_CARD.md, meta-llama/llama-models)Dropped since the lab’s previous model card: ARC-Challenge , MultiPL-E , GSM8K , API-Bank , Gorilla APIBench , Nexus , Multilingual MMLU (Meta) , AGIEval , CommonSenseQA , WinoGrande , BIG-Bench Hard , TriviaQA , SQuAD , QuAC , BoolQ , DROP
- 22 Oct 2024Claude 3.5 Sonnet (October 2024) Anthropic Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet
- OSWorld-Verified 22
- SWE-bench Verified 49
- tau-bench 57.6
- MMMU 70.4
- MathVista 70.7
- AI2D 95.3
- ChartQA 90.8
- DocVQA 94.2
- GPQA Diamond 65
- MMLU 90.5
- MMLU-Pro 78
- MATH 78.3
- HumanEval 93.7
- MGSM 92.5
- DROP 88.3
- BIG-Bench Hard 93.2
- AIME 2024 27.6
- IFEval 90.2
Dropped since the last post: GSM8K
Claude 3.5 Haiku: Its model card adds 7 evaluations 7 dropped 19 results in all
Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 SonnetSafety
Dropped since the lab’s previous model card: GSM8K , MMMU , MathVista , AI2D , ChartQA , DocVQA , Needle In A Haystack
Its model card adds 8 evaluations 1 new 25 results in all
- 19 Sep 2024
- MMLU-Pro 71.1
- MMLU-Redux 86.8
- GPQA Diamond 49
- MATH 83.1
- GSM8K 95.8
- HumanEval 86.6
- MBPP 88.2
- MultiPL-E 75.1
- LiveCodeBench 55.5
- LiveBench 52.3
- IFEval 84.1
- Arena-Hard-Auto 81.2
- AlignBench 8.16
- MT-Bench 9.35
- Multilingual IFEval 87
- AMMLU 72.4
- JMMLU 80.6
- KMMLU 62
- IndoMMLU 69.3
- TurkishMMLU 76.1
- Okapi MMLU 80
- MGSM 88.2
- BLEnD 32.5
- MMLU 86.1
- BIG-Bench Hard 86.3
- ARC-Challenge 72.4
- TruthfulQA 60.4
- WinoGrande 83.9
- HellaSwag 87.6
- TheoremQA 42.4
- MMLU-stem 82.7
- Multilingual Exams 78.7
- Multilingual Understanding 89.6
- Flores 39
Dropped since the last post: EvalPlus , C-Eval , CMMLU , MMMLU , Multilingual human evaluation
Qwen-Turbo: Its technical report adds 6 evaluations 2 new 7 dropped 19 results in all
Qwen2.5 Technical Report (arXiv 2412.15115)Capabilities
Dropped since the lab’s previous technical report: MMLU , TheoremQA , MixEval , AlignBench , NeedleBench , Multilingual human evaluation , Qwen2 multilingual safety evaluation
Qwen2.5-14B: Its technical report adds 5 evaluations 1 dropped 27 results in all
Qwen2.5 Technical Report (arXiv 2412.15115)Capabilities
Dropped since the lab’s previous technical report: Needle In A Haystack
Qwen2.5-32B: Its technical report adds 5 evaluations 27 results in all
Its technical report adds 5 evaluations 27 results in all
Qwen2.5-7B: Its technical report adds 5 evaluations 27 results in all
- 12 Sep 2024
- AIME 2024 56.7
- Codeforces rating 1258
- GPQA Diamond 78.3
- MATH-500 85.5
- MMLU 90.8
- MMMU
- MathVista
Dropped since the last post: MATH , HumanEval , MGSM , DROP , M3Exam , AI2D , ChartQA , DocVQA , ActivityNet-QA , EgoSchema
o1-mini: Its system card adds 37 evaluations 26 new 23 dropped 37 results in all
OpenAI o1 System Card (o1-preview and o1-mini)Capabilities
- MMMLU
- QuantBench 48
Hallucination
Safety
Robustness
Bio and chem risk
Cyber risk
Autonomy
Persuasion
- ChangeMyView 69.5
- Political Persuasion Parallel Generation 46.8
- MakeMePay 0.7
- MakeMeSay 38.9
Dropped since the lab’s previous system card: Unauthorized voice generation , Speaker identification , Disparate performance on voice inputs , Ungrounded inference and sensitive trait attribution , Biological threat creation uplift study , Text persuasion study on political opinions , Voice persuasion study on party preferences , Select machine learning engineering tasks from METR , Situational Awareness Dataset (SAD) , MedQA , MedQA USMLE (5 options) , MedQA Taiwan , MedQA Mainland China , MMLU Clinical Knowledge , MMLU Medical Genetics , MMLU Anatomy , MMLU Professional Medicine , MMLU College Biology , MMLU College Medicine , MedMCQA , Uhura ARC-Easy , Uhura TruthfulQA , Uhura-Eval
Its system card adds 39 evaluations 2 new 39 results in all
OpenAI o1 System Card (o1-preview and o1-mini)Capabilities
- MMMLU
- QuantBench 36
Hallucination
Safety
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
Autonomy
Persuasion
- ChangeMyView 81.8
- Political Persuasion Parallel Generation 53.8
- MakeMePay 11.6
- MakeMeSay 49
- 5 Sep 2024
- LIBERO 76.5
- 5 Sep 2024
- AlpacaEval 2.0 50.5
- Arena-Hard-Auto 76.2
- AlignBench 8.04
- MT-Bench 9.02
- HumanEval 89
- HumanEval Multi 73.8
- LiveCodeBench 41.8
- Aider-Edit 72.2
- SWE-bench Verified 16.8
- DS-FIM-Eval 78.3
- DS-Arena-Code 63.1
Dropped since the last post: MMLU , BIG-Bench Hard , C-Eval , CMMLU , MBPP , GSM8K , MATH , Needle In A Haystack
- 20 Aug 2024
- Arena-Hard-Auto 37.9
- BIG-Bench Hard 79.1
- MMLU 78.9
- MMLU-Pro 54.3
- ARC-Challenge 91
- BoolQ 84.6
- GPQA Diamond 36.8
- HellaSwag 83.8
- OpenBookQA 89.6
- PIQA 88.6
- SIQA 78
- TruthfulQA 77.5
- WinoGrande 81.3
- MMMLU 69.9
- MGSM 58.7
- GSM8K 88.7
- MATH 59.5
- Qasper 40
- SQuALITY 24.1
- HumanEval 70.7
- MBPP 80.8
Dropped since the last post: ANLI , MedQA , AGIEval , TriviaQA , ARC-Easy , CommonSenseQA , MT-Bench
Phi-3.5-mini: Its technical report adds 2 evaluations 23 results in all
Its technical report adds 7 evaluations 5 new 28 results in all
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4)Hallucination
Safety
Robustness
Phi-3.5-vision-instruct: Its technical report adds 14 evaluations 14 new 28 dropped 14 results in all
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4)Capabilities
Hallucination
- POPE 86.1
Safety
- Microsoft internal multimodal RAI benchmark 8.16
- RTVLM 5.44
- VLGuard 9.1
Dropped since the lab’s previous technical report: Arena-Hard-Auto , BIG-Bench Hard , MMLU , MMLU-Pro , ARC-Challenge , BoolQ , GPQA Diamond , HellaSwag , OpenBookQA , PIQA , SIQA , TruthfulQA , WinoGrande , MMMLU , MGSM , GSM8K , MATH , Qasper , SQuALITY , HumanEval , MBPP , RepoQA , RULER , Microsoft RAI benchmark: Ungroundedness , Microsoft RAI benchmark: Third Party Harm (DR-1) , Microsoft RAI benchmark: Harmful Content Continuation (DR-3) , Microsoft RAI benchmark: Harmful Content Summarization (DR-3) , Microsoft RAI benchmark: Jailbreak (DR-1)
- 13 Aug 2024
- GPQA Diamond 56
- MMLU 87.5
- MMLU-Pro 75.5
- MATH 76.1
- HumanEval 88.4
- MMMU 66.1
- MathVista 69
- DocVQA 93.6
- LMArena (Elo)
Dropped since the last post: GSM8K
- 24 Jul 2024 Mistral Large 2 Mistral AI Large Enough (Mistral Large 2 launch post) + HF model card Mistral-Large-Instruct-2407
- MMLU 84
- MT-Bench 8.63
- WildBench 56.3
- Arena-Hard-Auto 73.2
- HumanEval 92
- MBPP 80
- MultiPL-E 76.9
- GSM8K 93
- MATH 71.5
Dropped since the last post: HellaSwag , WinoGrande , ARC-Challenge , TriviaQA , TruthfulQA
- 23 Jul 2024
- MMLU 88.6
- MMLU-Pro 73.3
- IFEval 88.6
- HumanEval 89
- MBPP 88.6
- GSM8K 96.8
- MATH 73.8
- ARC-Challenge 96.9
- GPQA Diamond 51.1
- Berkeley Function Calling Leaderboard (BFCL) 88.5
- Nexus 58.7
- ZeroSCROLLS/QuALITY 95.2
- InfiniteBench/En.MC 83.4
- Needle In A Haystack 98.1
- MGSM 91.6
Dropped since the last post: AGIEval , CommonSenseQA , WinoGrande , BIG-Bench Hard , TriviaQA , SQuAD , QuAC , BoolQ , DROP
Its model card adds 13 evaluations 4 new 25 results in all
Llama 3.1 model card (MODEL_CARD.md, meta-llama/llama-models)Capabilities
- MultiPL-E
- API-Bank 92
- Gorilla APIBench 35.3
- Multilingual MMLU (Meta)
- AGIEval 71.6
- CommonSenseQA 85.8
- WinoGrande 86.7
- BIG-Bench Hard 85.9
- TriviaQA 91.8
- SQuAD 89.3
- QuAC 53.6
- BoolQ 80
- DROP 84.8
Llama 3.1-70B: Its model card adds 13 evaluations 25 results in all
Llama 3.1 model card (MODEL_CARD.md, meta-llama/llama-models)Capabilities
- MultiPL-E
- API-Bank 90
- Gorilla APIBench 29.7
- Multilingual MMLU (Meta)
- AGIEval 64.6
- CommonSenseQA 84.1
- WinoGrande 83.3
- BIG-Bench Hard 81.6
- TriviaQA 89.8
- SQuAD 81.8
- QuAC 51.1
- BoolQ 79.4
- DROP 79.6
Llama 3.1-8B: Its model card adds 13 evaluations 25 results in all
Llama 3.1 model card (MODEL_CARD.md, meta-llama/llama-models)Capabilities
- MultiPL-E
- API-Bank 82.6
- Gorilla APIBench 8.2
- Multilingual MMLU (Meta)
- AGIEval 47.8
- CommonSenseQA 75
- WinoGrande 60.5
- BIG-Bench Hard 64.2
- TriviaQA 77.6
- SQuAD 77
- QuAC 44.9
- BoolQ 75
- DROP 59.5
Its technical report adds 54 evaluations 45 new 10 dropped 69 results in all
The Llama 3 Herd of ModelsCapabilities
- HumanEval+ 82.3
- MultiPL-E
- API-Bank 92.3
- Gorilla APIBench 35.3
- Qasper 49.8
- SQuALITY 15.4
- InfiniteBench/En.QA 30.5
- Multilingual MMLU (Meta) 83.2
- LSAT 81.1
- SAT Reading 74.8
- SAT Math 94.9
- GMAT Quantitative 96
- GMAT Verbal 86.6
- GRE Physics 80
- AP Art History 86.7
- AP Biology 100
- AP Calculus 88.6
- AP Chemistry 90.6
- AP English Language and Composition 94.3
- AP English Literature and Composition 83.3
- AP Environmental Science 93.5
- AP Macroeconomics 98.3
- AP Microeconomics 93.8
- AP Physics 92.9
- AP Psychology 100
- AP Statistics 85.2
- AP US Government 97.6
- AP US History 97.6
- AP World History 100
- GRE Quantitative 162
- GRE Verbal 166
- SQuAD v2 81.8
- QuAC 53.6
- RACE 58.1
- CommonSenseQA 85.8
- PIQA 85.6
- SIQA 53.7
- OpenBookQA 49.2
- WinoGrande 82.2
- DROP 84.8
- WorldSense 63.7
- AGIEval 71.6
- BIG-Bench Hard 85.9
- QuALITY 87.6
- Adversarial SQuAD
- Dynabench SQuAD
- GSM-Plus
- PAWS
- Llama 3 pairwise human evaluation (Meta)
Robustness
Cyber risk
Dropped since the lab’s previous technical report: BoolQ , HellaSwag , ARC-Easy , Natural Questions , TriviaQA , TruthfulQA , ToxiGen , BOLD , Llama 2-Chat helpfulness human evaluation (Meta) , Llama 2-Chat safety human evaluation (Meta)
Llama 3.1-70B: Its technical report adds 53 evaluations 1 dropped 68 results in all
The Llama 3 Herd of ModelsCapabilities
- HumanEval+ 74.4
- MultiPL-E
- API-Bank 90
- Gorilla APIBench 29.7
- Qasper 49
- SQuALITY 16.4
- InfiniteBench/En.QA 36.7
- Multilingual MMLU (Meta) 78.2
- LSAT 74.2
- SAT Reading 71.4
- SAT Math 91.9
- GMAT Quantitative 84
- GMAT Verbal 85.1
- GRE Physics 74.7
- AP Art History 84.4
- AP Biology 100
- AP Calculus 54.3
- AP Chemistry 96.9
- AP English Language and Composition 90.6
- AP English Literature and Composition 79.6
- AP Environmental Science 89.1
- AP Macroeconomics 98.3
- AP Microeconomics 91.7
- AP Physics 78.6
- AP Psychology 100
- AP Statistics 59.3
- AP US Government 97.6
- AP US History 97.6
- AP World History 100
- GRE Quantitative 158
- GRE Verbal 166
- SQuAD v2 81.8
- QuAC 51.1
- RACE 59
- CommonSenseQA 84.1
- PIQA 83.8
- SIQA 52.2
- OpenBookQA 47.6
- WinoGrande 83.5
- DROP 79.6
- WorldSense 61.1
- AGIEval 64.6
- BIG-Bench Hard 81.6
- QuALITY 82.8
- Adversarial SQuAD
- Dynabench SQuAD
- GSM-Plus
- PAWS
Robustness
Cyber risk
Dropped since the lab’s previous technical report: Llama 3 pairwise human evaluation (Meta)
Llama 3.1-8B: Its technical report adds 50 evaluations 3 dropped 65 results in all
The Llama 3 Herd of ModelsCapabilities
- HumanEval+ 67.1
- MultiPL-E
- API-Bank 82.6
- Gorilla APIBench 8.2
- Qasper 39.3
- SQuALITY 15.3
- InfiniteBench/En.QA 27.1
- Multilingual MMLU (Meta) 58.6
- LSAT 53.9
- SAT Reading 57.4
- SAT Math 73.3
- GMAT Quantitative 56
- GMAT Verbal 65.7
- GRE Physics 48
- AP Art History 75.6
- AP Biology 91.7
- AP Calculus 57.1
- AP Chemistry 59.4
- AP English Language and Composition 69.8
- AP English Literature and Composition 59.3
- AP Environmental Science 73.9
- AP Macroeconomics 72.4
- AP Microeconomics 70.8
- AP Physics 57.1
- AP Psychology 94.8
- AP Statistics 66.7
- AP US Government 90.2
- AP US History 78
- AP World History 94.1
- GRE Quantitative 152
- GRE Verbal 149
- SQuAD v2 77
- QuAC 44.9
- RACE 54.3
- CommonSenseQA 75
- PIQA 81
- SIQA 49.5
- OpenBookQA 45
- WinoGrande 75.7
- DROP 59.5
- WorldSense 45.5
- AGIEval 47.8
- BIG-Bench Hard 64.2
- QuALITY 56
- Adversarial SQuAD
- Dynabench SQuAD
- GSM-Plus
- PAWS
Robustness
Dropped since the lab’s previous technical report: CyberSecEval Code Interpreter Abuse , CyberSecEval Spear Phishing , Llama internal safety benchmarks (Meta)
- 18 Jul 2024
- 27 Jun 2024
- MMLU 75.2
- HellaSwag 86.4
- PIQA 83.2
- SIQA 53.7
- BoolQ 84.8
- WinoGrande 83.7
- ARC-Easy 88.6
- ARC-Challenge 71.4
- TriviaQA 83.7
- Natural Questions 34.5
- HumanEval 51.8
- MBPP 62.6
- GSM8K 74
- MATH 42.3
- AGIEval 55.1
- DROP 72.2
- BIG-Bench Hard 74.9
Dropped since the last post: CommonSenseQA , OpenBookQA
Its technical report adds 20 evaluations 13 new 3 dropped 37 results in all
Gemma 2: Improving Open Language Models at a Practical Size (arXiv 2408.00118v1)Capabilities
- LMArena (Elo) 1218
Hallucination
- TruthfulQA 51.6
Safety
- RealToxicityPrompts 8.84
- ToxiGen 38.4
Bias
Cyber risk
Persuasion
Dropped since the lab’s previous technical report: CommonSenseQA , OpenBookQA , BOLD
Gemma 2 9B: Its technical report adds 8 evaluations 12 dropped 25 results in all
Gemma 2: Improving Open Language Models at a Practical Size (arXiv 2408.00118v1)Capabilities
- LMArena (Elo) 1187
Hallucination
- TruthfulQA 50.3
Safety
- RealToxicityPrompts 8.25
- ToxiGen 39.3
Bias
Dropped since the lab’s previous technical report: InterCode-CTF , In-house CTF suite (Phuong et al. 2024) , Hack the Box challenge set (Google DeepMind) , PrimeVul , DiverseVul , SPI (security patch identification) , SecretPatch , Self-proliferation challenges (Phuong et al. 2024) , Charm offensive , Hidden agenda , Money talks , Web of lies
- 20 Jun 2024
- GPQA Diamond 67.2
- MMLU 90.4
- MATH 71.1
- HumanEval 92
- MGSM 91.6
- DROP 87.1
- BIG-Bench Hard 93.1
- GSM8K 96.4
- MMMU 68.3
- MathVista 67.7
- AI2D 94.7
- ChartQA 90.8
- DocVQA 95.2
Dropped since the last post: ARC-Challenge , HellaSwag , PubMedQA , WinoGrande , RACE-H , APPS , MBPP
Its model card adds 8 evaluations 1 new 21 dropped 21 results in all
Claude 3.5 Sonnet Model Card AddendumCapabilities
Safety
Dropped since the lab’s previous model card: GPQA Main , GPQA Extended , ARC-Challenge , HellaSwag , PubMedQA , WinoGrande , RACE-H , APPS , MBPP , LSAT , MBE (Multistate Bar Examination) , AMC 12 2023 , AMC 10 2023 , AMC 8 2023 , Okapi MMLU , 100Q-Hard (Anthropic) , Multi-factual (Anthropic) , QuALITY , Multimodal policy red-teaming (Anthropic) , Discrim-Eval , BBQ (Bias Benchmark for QA)
- 7 Jun 2024
- MMLU 82.3
- MMLU-Pro 64.4
- GPQA Diamond 42.4
- TheoremQA 44.4
- MT-Bench 9.12
- Arena-Hard-Auto 48.1
- IFEval 77.6
- HumanEval 86
- MBPP 80.2
- MultiPL-E 69.2
- EvalPlus 79
- LiveCodeBench 35.7
- GSM8K 91.1
- MATH 59.7
- C-Eval 83.8
- AlignBench 8.27
- BIG-Bench Hard 82.4
- HellaSwag 87.6
- WinoGrande 85.1
- ARC-Challenge 68.9
- TruthfulQA 54.8
- CMMLU 90.1
- Multilingual Exams 76.6
- Multilingual Understanding 80.7
- Flores 37.8
- MMMLU 78
- MGSM 86.6
- Multilingual human evaluation 3.93
Dropped since the last post: AlpacaEval 2.0 , L-Eval , RGB , T-Eval , Chinese Tool-Use Benchmark , Code Interpreter Benchmark
Its technical report adds 7 evaluations 7 new 10 dropped 22 results in all
Qwen2 Technical Report (arXiv 2407.10671)Capabilities
Dropped since the lab’s previous technical report: C-Eval , BIG-Bench Hard , HumanEvalPack , MATH-500 , Math401 , Math23K , Chinese Tool-Use Benchmark , Code Interpreter Benchmark , HuggingFace Agent Benchmark , Qwen Chinese human evaluation
- 14 May 2024 Gemini 1.5 Pro Google DeepMind Gemini 1.5 technical report, May 2024 update with 1.5 Flash (arXiv 2403.05530v3)
- GPQA Diamond 46.2
- MATH 77.9
- PhysicsFinals 63.9
- HiddenMath 20.1
- Functional MATH 64.6
- AMC 46.4
- GSM8K 90.8
- BIG-Bench Hard 89.2
- DROP 74.9
- MMLU 91.7
- HellaSwag 93.3
- HumanEval 84.1
- Natural2Code 82.6
- WMT23 75.3
- MGSM 87.5
- MMMU 62.2
- MathVista 63.9
- AI2D 94.4
- ChemicalDiagramQA 69.7
- ChartQA 87.2
- BetterChartQA 65.8
- DocVQA 93.1
- DUDE 46
- TAT-DQA 37.8
- InfographicVQA 81
- TextVQA 78.7
- VQAv2 80.2
- V* 71.7
- BLINK 61.4
- RealWorldQA 70.4
- VATEX 64.6
- VATEX ZH 55.3
- YouCook2 106.5
- ActivityNet-QA 57.5
- EgoSchema 72.2
- OpenEQA 57.9
- 13 May 2024
- MMLU 88.7
- GPQA Diamond 53.6
- MATH 76.6
- HumanEval 90.2
- MGSM 90.5
- DROP 83.4
- M3Exam
- MMMU 69.1
- MathVista 63.8
- AI2D 94.2
- ChartQA 85.7
- DocVQA 92.8
- ActivityNet-QA 61.9
- EgoSchema 72.2
Its system card adds 34 evaluations 34 results in all
- 7 May 2024
- MMLU 77.8
- BIG-Bench Hard 79.7
- C-Eval 78
- CMMLU 81.6
- HumanEval 81.1
- MBPP 72
- LiveCodeBench 32.5
- GSM8K 92.2
- MATH 53.9
- Needle In A Haystack
- AlpacaEval 2.0
- MT-Bench
- AlignBench 7.91
Dropped since the last post: HellaSwag , TriviaQA , ChineseQA , Hungarian National High-School Exam , IFEval , LeetCode Weekly Contest
Its technical report adds 10 evaluations 2 new 16 dropped 22 results in all
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv 2405.04434)Capabilities
- TriviaQA 86.7
- Natural Questions 53.4
- ARC-Easy 98.1
- ARC-Challenge 92.3
- AGIEval 61.4
- IFEval 63.8
- CRUXEval 61.5
- CMATH 81.9
- CLUEWSC 89.9
- SuperCLUE-Math6 83.3
Dropped since the lab’s previous technical report: HellaSwag , PIQA , WinoGrande , RACE-M , RACE-H , DROP , OpenBookQA , CHID , C3 , CCPM , HumanEval Multi , MGSM , LeetCode Weekly Contest , Hungarian National High-School Exam , DeepSeek LLM in-house safety evaluation , Do-Not-Answer
- 22 Apr 2024
- MMLU 78.2
- HellaSwag 83
- ANLI 58.7
- GSM8K 90.3
- MedQA 69.4
- AGIEval 48.4
- TriviaQA 75.6
- ARC-Challenge 91
- ARC-Easy 97.8
- PIQA 87.7
- SIQA 80.2
- BIG-Bench Hard 81.3
- WinoGrande 81.4
- OpenBookQA 87.2
- BoolQ 86.6
- CommonSenseQA 82.6
- TruthfulQA 75.7
- HumanEval 55.5
- MBPP 74.5
- GPQA Diamond
- MT-Bench 8.91
Dropped since the last post: SQuAD v2
Its technical report adds 6 evaluations 6 new 14 dropped 26 results in all
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4)Capabilities
- MATH 53.1
Hallucination
Safety
Robustness
Dropped since the lab’s previous technical report: MMMU , ScienceQA , MathVista , Inter-GPS (Geometry3K) , MMBench , POPE , AI2D , ChartQA , TextVQA , BLINK , Video-MME , Microsoft internal multimodal RAI benchmark , RTVLM , VLGuard
phi-3-mini 3.8B: Its technical report adds 7 evaluations 1 new 28 results in all
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4)Hallucination
Safety
Robustness
phi-3-small 7.4B: Its technical report adds 6 evaluations 1 dropped 27 results in all
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4)Capabilities
- MATH 34.6
Hallucination
Safety
Robustness
Dropped since the lab’s previous technical report: MMMLU
- 18 Apr 2024
- MMLU 82
- GPQA Diamond 39.5
- HumanEval 81.7
- GSM8K 93
- MATH 50.4
- AGIEval 63
- CommonSenseQA 83.8
- WinoGrande 83.1
- BIG-Bench Hard 81.3
- ARC-Challenge 93
- TriviaQA 89.7
- SQuAD 85.6
- QuAC 51.1
- BoolQ 79
- DROP 79.7
Its model card adds 0 evaluations 2 dropped 15 results in all
Llama 3 model card (MODEL_CARD.md, meta-llama/llama3)Dropped since the lab’s previous model card: TruthfulQA , ToxiGen
Llama 3-8B: Its model card adds 0 evaluations 15 results in all
- 11 Apr 2024
Dropped since the last post: HellaSwag , ARC-Challenge , WinoGrande , GSM8K , VQAv2 , TextVQA , ChartQA , AI2D , DocVQA , InfographicVQA , TVQA , LSMDC
- 28 Mar 2024
Dropped since the last post: Hungarian National High-School Exam
- 4 Mar 2024
- MMLU 88.2
- MATH 73.7
- GSM8K 95
- HumanEval 84.9
- GPQA Diamond 59.5
- MGSM 90.7
- DROP 83.1
- BIG-Bench Hard 86.8
- ARC-Challenge 96.4
- HellaSwag 95.4
- PubMedQA 75.8
- WinoGrande 88.5
- RACE-H 92.9
- APPS 70.2
- MBPP 86.4
- MMMU 59.4
- DocVQA 89.3
- MathVista 50.5
- AI2D 88.1
- ChartQA 80.8
Dropped since the last post: TriviaQA , QuALITY
Claude 3 Haiku: Its model card adds 17 evaluations 14 new 12 dropped 37 results in all
The Claude 3 Model Family: Opus, Sonnet, HaikuCapabilities
- GPQA Main 35.2
- GPQA Extended 34.7
- LSAT 156.3
- MBE (Multistate Bar Examination) 64
- AMC 12 2023 48
- AMC 10 2023 54
- AMC 8 2023 36
- Okapi MMLU 65.2
- QuALITY 80.2
- Needle In A Haystack 95.9
Hallucination
Dropped since the lab’s previous model card: Human feedback win rates (Anthropic) , TruthfulQA , Harmfulness scores on held-out prompts (Anthropic) , Combined HHH evals (Anthropic) , FLORES-200 , TriviaQA , GRE Verbal , GRE Quantitative , GRE Analytical Writing , USMLE Step 1 (sample questions) , USMLE Step 2 (sample questions) , USMLE Step 3 (sample questions)
Its model card adds 23 evaluations 6 new 43 results in all
The Claude 3 Model Family: Opus, Sonnet, HaikuCapabilities
- GPQA Main 49.1
- GPQA Extended 48.8
- LSAT 161
- MBE (Multistate Bar Examination) 85
- AMC 12 2023 63
- AMC 10 2023 72
- AMC 8 2023 84
- GRE Quantitative 159
- GRE Verbal 166
- GRE Analytical Writing 5
- Okapi MMLU 79.1
- QuALITY 90.5
- Needle In A Haystack 99.4
Hallucination
Safety
Claude 3 Sonnet: Its model card adds 21 evaluations 3 new 5 dropped 41 results in all
The Claude 3 Model Family: Opus, Sonnet, HaikuCapabilities
- GPQA Main 38.5
- GPQA Extended 38
- LSAT 158.3
- MBE (Multistate Bar Examination) 71
- AMC 12 2023 27
- AMC 10 2023 24
- AMC 8 2023 54
- Human feedback win rates (Anthropic)
- Okapi MMLU 69
- QuALITY 85.9
- Needle In A Haystack 95.4
Hallucination
Safety
Dropped since the lab’s previous model card: GRE Quantitative , GRE Verbal , GRE Analytical Writing , Autonomous Replication and Adaption (ARA) evaluations (Anthropic) , Expert vulnerability discovery and exploit development evaluations (Anthropic)
- 26 Feb 2024
- MMLU 81.2
- HellaSwag 89.2
- WinoGrande 86.7
- ARC-Challenge 94
- TriviaQA 82.7
- TruthfulQA 50.5
- HumanEval 45.1
- MBPP 73.1
- MATH 45
- GSM8K 91.2
- 21 Feb 2024
- MMLU 64.3
- HellaSwag 81.2
- PIQA 81.2
- SIQA 51.8
- BoolQ 83.2
- WinoGrande 72.3
- CommonSenseQA 71.3
- OpenBookQA 52.8
- ARC-Easy 81.5
- ARC-Challenge 53.2
- TriviaQA 63.4
- Natural Questions 23
- HumanEval 32.3
- MBPP 44.4
- GSM8K 46.4
- MATH 24.3
- AGIEval 41.7
- BIG-Bench Hard 55.1
Gemma 2B: Its technical report adds 8 evaluations 26 results in all
Gemma: Open Models Based on Gemini Research and Technology (arXiv 2403.08295v1)Hallucination
- TruthfulQA 44.8
Safety
- RealToxicityPrompts 6.86
- ToxiGen 29.8
Bias
- BOLD 45.6
- CrowS-Pairs 45.8
- BBQ (Bias Benchmark for QA)
- Winogender 51.3
- WinoBias
Its technical report adds 8 evaluations 26 results in all
Gemma: Open Models Based on Gemini Research and Technology (arXiv 2403.08295v1)Hallucination
- TruthfulQA 31.8
Safety
- RealToxicityPrompts 7.9
- ToxiGen 39.6
Bias
- BOLD 49.1
- CrowS-Pairs 51.3
- BBQ (Bias Benchmark for QA)
- Winogender 54.2
- WinoBias
- 15 Feb 2024 Gemini 1.5 Pro Google DeepMind Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context (technical report, Feb 2024 version = arXiv 2403.05530v1)
- HellaSwag 92.5
- MMLU 81.9
- GSM8K 91.7
- GPQA Diamond 41.5
- MATH 59.4
- PhysicsFinals 60.7
- AMC 37.2
- BIG-Bench Hard 84
- DROP 78.9
- HumanEval 71.9
- Natural2Code 77.7
- WMT23 75.2
- MGSM 88.7
- MMMU 58.5
- AI2D 80.3
- MathVista 52.1
- ChartQA 81.3
- VQAv2 73.2
- TextVQA 73.5
- DocVQA 86.5
- InfographicVQA 72.7
- VATEX 63
- VATEX ZH 54.9
- YouCook2 134.2
- ActivityNet-QA 56.7
- EgoSchema 63.2
- 5 Feb 2024
- MMLU 77.5
- C-Eval 84.1
- GSM8K 79.5
- MATH 34.1
- HumanEval 41.5
- MBPP 53.4
- BIG-Bench Hard 65.5
- CMMLU 83.5
- MT-Bench 8.61
- AlpacaEval 2.0 27.2
- Multilingual Exams 66.3
- Multilingual Understanding 78.2
- MGSM 61.7
- Flores 35.6
- L-Eval 78.1
- RGB
- T-Eval 76.7
- Chinese Tool-Use Benchmark 95.1
- Code Interpreter Benchmark 73.1
Dropped since the last post: VCSUM , HuggingFace Agent Benchmark
- 16 Jan 2024GLM-4 Z.ai GLM-4 launch at Zhipu DevDay (official announcement text and slides, reposted on Volcengine developer community)
- MMLU 81.5
- GSM8K 87.6
- MATH 47.9
- BIG-Bench Hard 82.3
- HellaSwag 85.4
- HumanEval 72
- IFEval 67.7
- AlignBench 7.88
- LongBench 81.1
- Needle In A Haystack 100
Its technical report adds 3 evaluations 9 results in all
2023
- 12 Dec 2023
- 11 Dec 2023
- MMLU 72.5
- HellaSwag 86.7
- ARC-Challenge 85.8
- WinoGrande 81.2
- MBPP 60.7
- GSM8K 58.4
- MT-Bench 8.3
- PIQA 83.6
- ARC-Easy 83.1
- Natural Questions 30.6
- TriviaQA 71.5
- HumanEval 40.2
- MATH 28.4
Its technical report adds 7 evaluations 4 new 3 dropped 20 results in all
Mixtral of ExpertsCapabilities
Bias
Dropped since the lab’s previous technical report: LLMBoxing , Mistral 175 unsafe prompts , Mistral self-reflection moderation set
- 6 Dec 2023 Gemini 1.0 Ultra Google DeepMind Gemini: A Family of Highly Capable Multimodal Models (technical report, arXiv 2312.11805v1)
- MMLU 90
- GSM8K 94.4
- MATH 53.2
- BIG-Bench Hard 83.6
- HumanEval 74.4
- Natural2Code 74.9
- DROP 82.4
- HellaSwag 87.8
- WMT23 74.4
- MMMU 62.4
- TextVQA 82.3
- DocVQA 90.9
- ChartQA 80.8
- InfographicVQA 80.3
- MathVista 53
- AI2D 79.5
- VQAv2 77.8
Dropped since the last post: TriviaQA , Natural Questions , WebQuestions , LAMBADA , StoryCloze , WSC , WinoGrande , Winograd , SQuAD v2 , RACE-H , RACE-M , PIQA , ARC-Challenge , ARC-Easy , OpenBookQA , BoolQ , COPA , RTE , WiC , MultiRC , ReCoRD , CB , ANLI , StrategyQA , CommonSenseQA , XCOPA , MGSM
- 29 Nov 2023
- HellaSwag 84
- TriviaQA 81.5
- MMLU 71.1
- GSM8K 84.1
- HumanEval 73.8
- BIG-Bench Hard 71.7
- C-Eval 65.2
- CMMLU 67.8
- ChineseQA 85.1
- Hungarian National High-School Exam
- IFEval
- LeetCode Weekly Contest
Its technical report adds 23 evaluations 34 results in all
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (arXiv 2401.02954)Capabilities
- PIQA 82.6
- WinoGrande 76
- RACE-M 70.9
- RACE-H 56
- Natural Questions 47
- ARC-Easy 81.6
- ARC-Challenge 64.1
- MATH 32.6
- MBPP 61.4
- DROP 71.9
- OpenBookQA 63.2
- AGIEval 46.4
- CLUEWSC 60
- CHID 72.6
- CMATH 80.3
- C3 77
- CCPM 84.9
- HumanEval Multi 53.3
- MGSM 74
- AlignBench 6.43
- MT-Bench 8.35
Safety
- 3 Nov 2023
- GSM8K 62.9
- MMLU 73
- HumanEval 63.2
- MATH 23.9
- Hungarian National High-School Exam 59
- 27 Oct 2023
Its technical report adds 10 evaluations 16 results in all
ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All ToolsCapabilities
- GPQA Diamond 26.8
- HumanEval 58.5
- BoolQ 87.9
- CommonSenseQA 86.5
- HellaSwag 79.7
- PIQA 80.1
- DROP 70.9
- GAOKAO-Bench 67.3
- C3 73.9
- Berkeley Function Calling Leaderboard (BFCL) 57.9
- 27 Sep 2023
- MMLU 60.1
- HellaSwag 81.3
- WinoGrande 75.3
- PIQA 83
- ARC-Easy 80
- ARC-Challenge 55.5
- Natural Questions 28.8
- TriviaQA 69.9
- HumanEval 30.5
- MBPP 47.5
- MATH 13.1
- GSM8K 52.1
- MT-Bench 6.84
Its technical report adds 6 evaluations 19 results in all
Mistral 7BCapabilities
- 25 Sep 2023
- C-Eval 71.7
- MMLU 66.5
- HumanEval 43.9
- GSM8K 60.1
- MATH 24.8
- MBPP 40.8
- BIG-Bench Hard 53.4
- CMMLU 71
- VCSUM 17.3
- Chinese Tool-Use Benchmark 98
- Code Interpreter Benchmark 58.4
- HuggingFace Agent Benchmark 93.5
Qwen-1_8B: Its technical report adds 0 evaluations 8 results in all
Its technical report adds 5 evaluations 5 new 14 results in all
Qwen Technical Report (arXiv 2309.16609)Capabilities
- HumanEvalPack 33.7
- MATH-500 18.4
- Math401 70.1
- Math23K 67
- Qwen Chinese human evaluation 30.7
Qwen-7B: Its technical report adds 5 evaluations 14 results in all
Qwen Technical Report (arXiv 2309.16609)Capabilities
- HumanEvalPack 24.2
- MATH-500 6.8
- Math401 57.4
- Math23K 51.2
- Qwen Chinese human evaluation 28.4
- 18 Jul 2023
- MMLU 68.9
- BIG-Bench Hard 51.2
- AGIEval 54.2
Llama 2-13B: Its model card adds 2 evaluations 5 results in all
Llama 2 model card (MODEL_CARD.md, meta-llama/llama)Safety
- TruthfulQA 41.9
- ToxiGen 26.1
Its model card adds 2 evaluations 5 results in all
Llama 2 model card (MODEL_CARD.md, meta-llama/llama)Safety
- TruthfulQA 50.2
- ToxiGen 24.6
Llama 2-7B: Its model card adds 2 evaluations 5 results in all
Llama 2 model card (MODEL_CARD.md, meta-llama/llama)Safety
- TruthfulQA 33.3
- ToxiGen 21.3
Llama 2-13B: Its technical report adds 22 evaluations 25 results in all
Llama 2: Open Foundation and Fine-Tuned Chat ModelsCapabilities
- HumanEval 18.3
- MBPP 30.6
- BoolQ 81.7
- PIQA 80.5
- SIQA 50.3
- HellaSwag 80.7
- WinoGrande 72.8
- ARC-Easy 77.3
- ARC-Challenge 49.4
- OpenBookQA 57
- CommonSenseQA 67.3
- Natural Questions 31.2
- TriviaQA 79.6
- SQuAD v2 72.9
- QuAC 42.7
- GSM8K 28.7
- MATH 3.9
- Llama 2-Chat helpfulness human evaluation (Meta)
Safety
Bias
Llama 2-34B: Its technical report adds 22 evaluations 25 results in all
Llama 2: Open Foundation and Fine-Tuned Chat ModelsCapabilities
- HumanEval 22.6
- MBPP 33
- BoolQ 83.7
- PIQA 81.9
- SIQA 50.9
- HellaSwag 83.3
- WinoGrande 76.7
- ARC-Easy 79.4
- ARC-Challenge 54.5
- OpenBookQA 58.2
- CommonSenseQA 74.3
- Natural Questions 32.8
- TriviaQA 84.5
- SQuAD v2 77.4
- QuAC 42.9
- GSM8K 42.2
- MATH 6.24
- Llama 2-Chat helpfulness human evaluation (Meta)
Safety
Bias
Its technical report adds 22 evaluations 25 results in all
Llama 2: Open Foundation and Fine-Tuned Chat ModelsCapabilities
- HumanEval 29.9
- MBPP 45
- BoolQ 85
- PIQA 82.8
- SIQA 50.7
- HellaSwag 85.3
- WinoGrande 80.2
- ARC-Easy 80.2
- ARC-Challenge 57.4
- OpenBookQA 60.2
- CommonSenseQA 78.5
- Natural Questions 39.5
- TriviaQA 87.6
- SQuAD v2 80.7
- QuAC 42.4
- GSM8K 56.8
- MATH 13.5
- Llama 2-Chat helpfulness human evaluation (Meta)
Safety
Bias
Llama 2-7B: Its technical report adds 22 evaluations 25 results in all
Llama 2: Open Foundation and Fine-Tuned Chat ModelsCapabilities
- HumanEval 12.8
- MBPP 20.8
- BoolQ 77.4
- PIQA 78.8
- SIQA 48.3
- HellaSwag 77.2
- WinoGrande 69.2
- ARC-Easy 75.2
- ARC-Challenge 45.9
- OpenBookQA 58.6
- CommonSenseQA 57.8
- Natural Questions 25.7
- TriviaQA 72.1
- SQuAD v2 67.2
- QuAC 39.4
- GSM8K 14.6
- MATH 2.5
- Llama 2-Chat helpfulness human evaluation (Meta)
Safety
Bias
- 11 Jul 2023
Its model card adds 13 evaluations 20 results in all
- 10 May 2023
- TriviaQA 86.1
- Natural Questions 37.5
- WebQuestions 28.2
- LAMBADA 86.9
- HellaSwag 86.8
- StoryCloze 87.4
- WSC 86.9
- WinoGrande 90.9
- Winograd 89.5
- SQuAD v2 80.5
- RACE-H 62.3
- RACE-M 77
- PIQA 85
- ARC-Challenge 95.1
- ARC-Easy 89.7
- OpenBookQA 58.5
- BoolQ 90.9
- COPA 96
- RTE 79.3
- WiC 66.8
- MultiRC 88.2
- ReCoRD 93.8
- CB 87.5
- ANLI 73.1
- DROP 85
- StrategyQA 90.4
- CommonSenseQA 90.4
- XCOPA 94.4
- BIG-Bench Hard 78.1
- MATH 48.8
- GSM8K 91
- MGSM 87
- 14 Mar 2023
- MMLU 86.4
- HellaSwag 95.3
- ARC-Challenge 96.3
- WinoGrande 87.5
- HumanEval 67
- DROP 80.9
- GSM8K 92
- VQAv2 77.2
- TextVQA 78
- ChartQA 78.5
- AI2D 78.2
- DocVQA 88.4
- InfographicVQA 75.1
- TVQA 87.3
- LSMDC 45.7
Its technical report adds 43 evaluations 50 results in all
GPT-4 Technical Report (with appended GPT-4 System Card)Capabilities
- Uniform Bar Exam (MBE+MEE+MPT) 298
- LSAT 163
- SAT Evidence-Based Reading & Writing 710
- SAT Math 700
- GRE Quantitative 163
- GRE Verbal 169
- GRE Analytical Writing 4
- USABO Semifinal Exam 2020 87
- USNCO Local Section Exam 2022 36
- Medical Knowledge Self-Assessment Program (MKSAP) 75
- Codeforces rating 392
- AP Art History 5
- AP Biology 5
- AP Calculus BC 4
- AP Chemistry 4
- AP English Language and Composition 2
- AP English Literature and Composition 2
- AP Environmental Science 5
- AP Macroeconomics 5
- AP Microeconomics 5
- AP Physics 2 4
- AP Psychology 5
- AP Statistics 5
- AP US Government 5
- AP US History 5
- AP World History 4
- AMC 10 2022 30
- AMC 12 2022 60
- Introductory Sommelier (theory knowledge) 92
- Certified Sommelier (theory knowledge) 86
- Advanced Sommelier (theory knowledge) 77
- Leetcode (easy) 75.6
- Leetcode (medium) 26.3
- Leetcode (hard) 6.7
- MMLU translated with Azure Translate
- Hindsight Neglect
- Preference on ChatGPT and API prompts 70.2
Hallucination