Benchmark Almanac

PIQA

Phi-3.5-MoE 88.6% leads since Aug 2024 Dropped

Models pick which of two solutions better achieves an everyday physical goal. Scoring is accuracy.

In the authors’ words

To apply eyeshadow without a brush, should I use a cotton swab or a toothpick? Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems. While recent pretrained models (such as BERT) have made progress on question answering over more abstract domains - such as news articles and encyclopedia entries, where text is plentiful - in more physical domains, text is inherently limited due to reporting bias. Can AI systems learn to reliably answer physical common-sense questions without experiencing the physical world? In this paper, we introduce the task of physical commonsense reasoning and a corresponding benchmark dataset Physical Interaction: Question Answering or PIQA. Though humans find the dataset easy (95% accuracy), large pretrained models struggle (77%). We provide analysis about the dimensions of knowledge that existing models lack, which offers significant opportunities for future research.

Abstract of PIQA: Reasoning about Physical Commonsense in Natural Language (arXiv:1911.11641, CC0)
Made by Yonatan Bisk , Rowan Zellers , Ronan Le Bras , Jianfeng Gao , Yejin Choi Released Nov 2019 Commonsense Public tasks plus a private test set Last in a launch post Sep 2025 , dropped after it saturated Last model added Feb 2026 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
Paper Dataset Leaderboard yonatanbisk.com
33
Models scored in lab cards
15
Launch posts list it, from 5 labs
28
System cards report it, from 6 labs

Phi-3.5-MoE leads with 88.6% among models released by Feb 2026

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

The newest model with a score here came out in Feb 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

33 models. Scroll sideways for the rest.

  1. 88.6 Phi-3.5-MoE
  2. 87.9 phi-3-medium 14B
  3. 86.9 phi-3-small 7.4B
  4. 85.6 Llama 3.1-405B
  5. 85.3 GLM-4.5
  6. 85.0 PaLM 2-L
  7. 84.7 DeepSeek-V3
  8. 84.6 GLM-5
  9. 84.2 phi-3-mini 3.8B
  10. 83.8 Llama 3.1-70B
  11. 83.6 Mixtral 8x7B
  12. 83.3 Gemma 3 27B
  13. 83.2 Gemma 2 27B
  14. 83.0 Mistral 7B v0.1
  15. 82.8 Llama 2-70B
  16. 82.6 DeepSeek LLM 67B
  17. 81.9 Llama 2-34B
  18. 81.9 Apertus 70B
  19. 81.8 Gemma 3 12B
  20. 81.7 Gemma 2 9B
  21. 81.2 Gemma 7B
  22. 81.0 Phi-3.5-mini
  23. 81.0 Llama 3.1-8B
  24. 81.0 Gemma 3n E4B
  25. 80.5 Llama 2-13B
  26. 80.1 ChatGLM3-6B
  27. 79.6 Gemma 3 4B
  28. 78.8 Llama 2-7B
  29. 77.6 Phi-4 Mini
  30. 77.3 Gemma 2B
  31. 77.1 BitNet b1.58 2B4T
  32. 73.8 Gemma 3 1B
  33. 66.2 Gemma 3 270M
  1. 1 Phi-3.5-MoE 88.6%
  2. 2 phi-3-medium 14B 87.9%
  3. 3 phi-3-small 7.4B 86.9%
  4. 4 Llama 3.1-405B 85.6%
  5. 5 GLM-4.5 85.3%
  6. 6 PaLM 2-L 85%
  7. 7 DeepSeek-V3 84.7%
  8. 8 GLM-5 84.6%
  9. 9 phi-3-mini 3.8B 84.2%
  10. 10 Llama 3.1-70B 83.8%
  11. 11 Mixtral 8x7B 83.6%
  12. 12 Gemma 3 27B 83.3%
  13. 13 Gemma 2 27B 83.2%
  14. 14 Mistral 7B v0.1 83%
  15. 15 Llama 2-70B 82.8%
  16. 16 DeepSeek LLM 67B 82.6%
  17. 17 Llama 2-34B 81.9%
  18. 18 Apertus 70B 81.9%
  19. 19 Gemma 3 12B 81.8%
  20. 20 Gemma 2 9B 81.7%
  21. 21 Gemma 7B 81.2%
  22. 22 Phi-3.5-mini 81%
  23. 23 Llama 3.1-8B 81%
  24. 24 Gemma 3n E4B 81%
  25. 25 Llama 2-13B 80.5%
  26. 26 ChatGLM3-6B 80.1%
  27. 27 Gemma 3 4B 79.6%
  28. 28 Llama 2-7B 78.8%
  29. 29 Phi-4 Mini 77.6%
  30. 30 Gemma 2B 77.3%
  31. 31 BitNet b1.58 2B4T 77.1%
  32. 32 Gemma 3 1B 73.8%
  33. 33 Gemma 3 270M 66.2%
#ModelScoreSettingDateSource
1 Phi-3.5-MoE 88.6% Aug 2024 Model card
2 phi-3-medium 14B 87.9% Aug 2024 Model card
3 phi-3-small 7.4B 86.9% Aug 2024 Model card
4 Llama 3.1-405B 85.6% Nov 2024 Model card
5 GLM-4.5 85.3% Aug 2025 Model card
6 PaLM 2-L 85% May 2023 Model card
7 DeepSeek-V3 84.7% Dec 2024 Model card
8 GLM-5 84.6% Feb 2026 Model card
9 phi-3-mini 3.8B 84.2% Aug 2024 Model card
10 Llama 3.1-70B 83.8% Nov 2024 Model card
11 Mixtral 8x7B 83.6% Dec 2023 Model card
12 Gemma 3 27B 83.3% Mar 2025 Model card
13 Gemma 2 27B 83.2% Jun 2024 Model card
14 Mistral 7B v0.1 83% Sep 2023 Model card
15 Llama 2-70B 82.8% Jul 2023 Model card
16 DeepSeek LLM 67B 82.6% Jan 2024 Model card
17 Llama 2-34B 81.9% Jul 2023 Model card
18 Apertus 70B 81.9% Sep 2025 Model card
19 Gemma 3 12B 81.8% Mar 2025 Model card
20 Gemma 2 9B 81.7% Jul 2024 Model card
21 Gemma 7B 81.2% Feb 2024 Model card
22 Phi-3.5-mini 81% Aug 2024 Model card
23 Llama 3.1-8B 81% Nov 2024 Model card
24 Gemma 3n E4B 81% Jun 2025 Model card
25 Llama 2-13B 80.5% Jul 2023 Model card
26 ChatGLM3-6B 80.1% Jun 2024 Model card
27 Gemma 3 4B 79.6% Mar 2025 Model card
28 Llama 2-7B 78.8% Jul 2023 Model card
29 Phi-4 Mini 77.6% Feb 2025 Model card
30 Gemma 2B 77.3% Mar 2024 Model card
31 BitNet b1.58 2B4T 77.1% Apr 2025 Model card
32 Gemma 3 1B 73.8% Mar 2025 Model card
33 Gemma 3 270M 66.2% Aug 2025 Model card

The benchmark's own leaderboard has 8 entries, 1 of them models we track. Epoch AI has scores for 58 models. We plot every source in the charts below.

4 models have held number one since 2023

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for Epoch AI’s runs and what labs claim in their cards.

Microsoft Inflection AI Technology Innovation Institute OpenAI Other labs
Epoch AI runs Passed 85% in Jul 2024, by GPT-4o mini
GPT-4o mini 88.7%
Lab claims Passed 85% in May 2023, by PaLM 2-L
Phi-3.5-MoE 88.6%

The best score passed 85% in May 2023

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Run or collected by Epoch AI Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2020 2021 2022 2023 2024 2025 2026 Near the ceiling (85%) Random guessing GPT-4o mini PaLM 2-L Megatron-Turing NLG 530B GPT-3 175B (davinci) GPT-2 (1.5B) 0% 20% 40% 60% 80% 100% 2020 2022 2024 2026 Near the ceiling (85%) Random guessing GPT-4o mini GPT-3 175B (davinci)

Labs dropped it from launch posts once the best score passed 85%

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Google DeepMind
Mistral AI
Microsoft
DeepSeek
Swiss AI Initiative
DateLaunch postScoreAs the post writes it
2 Sep 2025 Apertus 70B 81.9 PIQA (pretraining evaluation)
14 Aug 2025 Gemma 3 270M 66.2 PIQA 0-shot (IT)
26 Jun 2025 Gemma 3n E4B 81 PIQA 0-shot (PT)
15 Apr 2025 BitNet b1.58 2B4T 77.1 PIQA
12 Mar 2025 Gemma 3 27B 83.3 PIQA 0-shot (PT)
26 Feb 2025 Phi-4 Mini 77.6 PIQA (5-shot)
26 Dec 2024 DeepSeek-V3 84.7 PIQA (Acc.)
20 Aug 2024 Phi-3.5-MoE 88.6 PIQA (5-shot)
27 Jun 2024 Gemma 2 27B 83.2 PIQA (0-shot)
22 Apr 2024 phi-3-medium 14B 87.7 PIQA (5-Shot)
21 Feb 2024 Gemma 7B 81.2 PIQA (0-shot)
12 Dec 2023 Phi-2 PIQA (in Commonsense Reasoning average only)
11 Dec 2023 Mixtral 8x7B 83.6 PIQA [detailed table]
27 Sep 2023 Mistral 7B v0.1 83 PIQA
10 May 2023 PaLM 2-L 85 PIQA, 1-shot

28 system cards report it, from 6 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
17 Feb 2026 GLM-5 technical report 84.6 GLM-5-Base, PIQA (EM) (Table 11) Capabilities
8 Aug 2025 GLM-4.5 technical report 85.3 GLM-4.5-Base, PIQA (EM) (Table 2) Capabilities
16 Apr 2025 BitNet b1.58 2B4T technical report 77.1 PIQA (0-shot; Acc,norm) (Table 1, Table 3) Capabilities
25 Mar 2025 Gemma 3 4B technical report 79.6 PIQA, PT, 0-shot (Table 9) Capabilities
25 Mar 2025 Gemma 3 27B technical report 83.3 PIQA, PT, 0-shot (Table 9) Capabilities
25 Mar 2025 Gemma 3 1B technical report 73.8 PIQA, PT, 0-shot (Table 9) Capabilities
25 Mar 2025 Gemma 3 12B technical report 81.8 PIQA, PT, 0-shot (Table 9) Capabilities
3 Mar 2025 Phi-4 Mini technical report 77.6 PIQA (5-Shot) (Table 7) Capabilities
23 Nov 2024 Llama 3.1-8B technical report 81 PiQA, pretrained (Table 11) Capabilities
23 Nov 2024 Llama 3.1-70B technical report 83.8 PiQA, pretrained (Table 11) Capabilities
23 Nov 2024 Llama 3.1-405B technical report 85.6 PiQA, pretrained (Table 11) Capabilities
30 Aug 2024 phi-3-small 7.4B technical report 86.9 PIQA (5-Shot) (Section 3 table) Capabilities
30 Aug 2024 phi-3-mini 3.8B technical report 84.2 PIQA (5-Shot) (Section 3 table) Capabilities
30 Aug 2024 phi-3-medium 14B technical report 87.9 PIQA (5-Shot) (Section 3 table) Capabilities
30 Aug 2024 Phi-3.5-MoE technical report 88.6 PIQA (5-shot) (Table 3) Capabilities
30 Aug 2024 Phi-3.5-mini technical report 81 PIQA (5-shot) (Table 3) Capabilities
31 Jul 2024 Gemma 2 9B technical report 81.7 PIQA, PT, 0-shot (Table 13) Capabilities
31 Jul 2024 Gemma 2 27B technical report 83.2 PIQA, PT, 0-shot (Table 13) Capabilities
18 Jun 2024 ChatGLM3-6B technical report 80.1 ChatGLM3-6B-Base, PIQA (Table 1) Capabilities
13 Mar 2024 Gemma 7B technical report 81.2 PIQA, PT, 0-shot (Table 6) Capabilities
13 Mar 2024 Gemma 2B technical report 77.3 PIQA, PT, 0-shot (Table 6) Capabilities
8 Jan 2024 Mixtral 8x7B technical report 83.6 PIQA, 0-shot (Table 2) Capabilities
5 Jan 2024 DeepSeek LLM 67B technical report 82.6 PIQA, DeepSeek 67B Chat (Table 6) Capabilities
10 Oct 2023 Mistral 7B v0.1 technical report 83 PIQA, 0-shot (Table 2) Capabilities
19 Jul 2023 Llama 2-7B technical report 78.8 PIQA, 0-shot, pretrained (Table 20) Capabilities
19 Jul 2023 Llama 2-70B technical report 82.8 PIQA, 0-shot, pretrained (Table 20) Capabilities
19 Jul 2023 Llama 2-34B technical report 81.9 PIQA, 0-shot, pretrained (Table 20) Capabilities
19 Jul 2023 Llama 2-13B technical report 80.5 PIQA, 0-shot, pretrained (Table 20) Capabilities

More in Knowledge

All 109 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • KMMLU-Pro In use Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. Solar Pro 4 79.2%
  • BIG-Bench Extra Hard In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. Gemma 4 31B IT 74.4%
  • HLE-Verified In use Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. Seed2.0 Pro 73.6%