Benchmark Almanac

ARC-Challenge

Llama 3.1-405B 96.9% leads since Jul 2024 Dropped

Models answer grade-school science multiple-choice questions that retrieval and word co-occurrence baselines got wrong. Scored by accuracy.

Saturated by frontier models, which score above 95%.

In the authors’ words

We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.

Abstract of Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (arXiv:1803.05457, CC0)
Made by Allen Institute for AI Released Mar 2018 Exam questions All tasks public Last in a launch post Aug 2025 , dropped after it saturated Last model added Nov 2025 Scores refreshed 31 Oct 2026
Level
School Students, from grade school to high school
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
allenai/aristo-leaderboard 42 stars Paper Dataset allenai.org
45
Models scored in lab cards
25
Launch posts list it, from 10 labs
39
System cards report it, from 9 labs

Llama 3.1-405B leads with 96.9% among models released by Sep 2025

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

The newest model with a score here came out in Sep 2025, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

45 models. Scroll sideways for the rest.

  1. 96.9 Llama 3.1-405B
  2. 96.4 Claude 3 Opus
  3. 96.3 GPT-4
  4. 95.3 DeepSeek-V3
  5. 95.1 PaLM 2-L
  6. 95.0 Sarvam-M
  7. 94.8 Llama 3.1-70B
  8. 94.0 Mistral Large
  9. 93.2 Claude 3 Sonnet
  10. 93.0 Llama 3-70B
  11. 92.3 DeepSeek-V2 (MoE-236B, May 2024)
  12. 91.6 phi-3-medium 14B
  13. 91.3 Mistral Small 3
  14. 91.0 Claude 2
  15. 91.0 Phi-3.5-MoE
  16. 90.7 phi-3-small 7.4B
  17. 89.2 Claude 3 Haiku
  18. 85.8 Mixtral 8x7B
  19. 85.0 Apertus 70B
  20. 84.9 phi-3-mini 3.8B
  21. 84.6 Phi-3.5-mini
  22. 83.7 Phi-4 Mini
  23. 83.4 Llama 3.1-8B
  24. 78.6 Llama 3-8B
  25. 72.4 Qwen2.5-72B
  26. 71.4 Gemma 2 27B
  27. 70.6 Gemma 3 27B
  28. 68.9 Qwen2-72B
  29. 68.9 Gemma 3 12B
  30. 68.4 Gemma 2 9B
  31. 64.1 DeepSeek LLM 67B
  32. 61.6 Gemma 3n E4B
  33. 61.0 Falcon-H1-34B
  34. 57.4 Llama 2-70B
  35. 56.2 Gemma 3 4B
  36. 55.5 Mistral 7B v0.1
  37. 54.5 Llama 2-34B
  38. 53.5 Falcon-Arabic-7B
  39. 53.2 Gemma 7B
  40. 49.9 BitNet b1.58 2B4T
  41. 49.4 Llama 2-13B
  42. 45.9 Llama 2-7B
  43. 42.1 Gemma 2B
  44. 38.4 Gemma 3 1B
  45. 28.2 Gemma 3 270M
  1. 1 Llama 3.1-405B 96.9%
  2. 2 Claude 3 Opus 96.4%
  3. 3 GPT-4 Mar 2023 96.3%
  4. 4 DeepSeek-V3 95.3%
  5. 5 PaLM 2-L 95.1%
  6. 6 Sarvam-M 95%
  7. 7 Llama 3.1-70B 94.8%
  8. 8 Mistral Large 94%
  9. 9 Claude 3 Sonnet 93.2%
  10. 10 Llama 3-70B 93%
  11. 11 DeepSeek-V2 (MoE-236B, May 2024) 92.3%
  12. 12 phi-3-medium 14B 91.6%
  13. 13 Mistral Small 3 91.3%
  14. 14 Claude 2 91%
  15. 15 Phi-3.5-MoE 91%
  16. 16 phi-3-small 7.4B 90.7%
  17. 17 Claude 3 Haiku 89.2%
  18. 18 Mixtral 8x7B 85.8%
  19. 19 Apertus 70B 85%
  20. 20 phi-3-mini 3.8B 84.9%
  21. 21 Phi-3.5-mini 84.6%
  22. 22 Phi-4 Mini 83.7%
  23. 23 Llama 3.1-8B 83.4%
  24. 24 Llama 3-8B 78.6%
  25. 25 Qwen2.5-72B 72.4%
  26. 26 Gemma 2 27B 71.4%
  27. 27 Gemma 3 27B 70.6%
  28. 28 Qwen2-72B 68.9%
  29. 29 Gemma 3 12B 68.9%
  30. 30 Gemma 2 9B 68.4%
  31. 31 DeepSeek LLM 67B 64.1%
  32. 32 Gemma 3n E4B 61.6%
  33. 33 Falcon-H1-34B 61%
  34. 34 Llama 2-70B 57.4%
  35. 35 Gemma 3 4B 56.2%
  36. 36 Mistral 7B v0.1 55.5%
  37. 37 Llama 2-34B 54.5%
  38. 38 Falcon-Arabic-7B 53.5%
  39. 39 Gemma 7B 53.2%
  40. 40 BitNet b1.58 2B4T 49.9%
  41. 41 Llama 2-13B 49.4%
  42. 42 Llama 2-7B 45.9%
  43. 43 Gemma 2B 42.1%
  44. 44 Gemma 3 1B 38.4%
  45. 45 Gemma 3 270M 28.2%
#ModelScoreSettingDateSource
1 Llama 3.1-405B 96.9% Jul 2024 Model card
2 Claude 3 Opus 96.4% Mar 2024 Model card
3 GPT-4 Mar 2023 96.3% Mar 2023 Model card
4 DeepSeek-V3 95.3% Dec 2024 Model card
5 PaLM 2-L 95.1% May 2023 Model card
6 Sarvam-M 95% May 2025 Model card
7 Llama 3.1-70B 94.8% Jul 2024 Model card
8 Mistral Large 94% Feb 2024 Model card
9 Claude 3 Sonnet 93.2% Mar 2024 Model card
10 Llama 3-70B 93% Apr 2024 Model card
11 DeepSeek-V2 (MoE-236B, May 2024) 92.3% May 2024 Model card
12 phi-3-medium 14B 91.6% Aug 2024 Model card
13 Mistral Small 3 91.3% Jan 2025 Model card
14 Claude 2 91% Jul 2023 Model card
15 Phi-3.5-MoE 91% Aug 2024 Model card
16 phi-3-small 7.4B 90.7% Aug 2024 Model card
17 Claude 3 Haiku 89.2% Mar 2024 Model card
18 Mixtral 8x7B 85.8% Dec 2023 Model card
19 Apertus 70B 85% Sep 2025 Model card
20 phi-3-mini 3.8B 84.9% Aug 2024 Model card
21 Phi-3.5-mini 84.6% Aug 2024 Model card
22 Phi-4 Mini 83.7% Feb 2025 Model card
23 Llama 3.1-8B 83.4% Jul 2024 Model card
24 Llama 3-8B 78.6% Apr 2024 Model card
25 Qwen2.5-72B 72.4% Sep 2024 Model card
26 Gemma 2 27B 71.4% Jun 2024 Model card
27 Gemma 3 27B 70.6% Mar 2025 Model card
28 Qwen2-72B 68.9% Jun 2024 Model card
29 Gemma 3 12B 68.9% Mar 2025 Model card
30 Gemma 2 9B 68.4% Jul 2024 Model card
31 DeepSeek LLM 67B 64.1% Jan 2024 Model card
32 Gemma 3n E4B 61.6% Jun 2025 Model card
33 Falcon-H1-34B 61% May 2025 Model card
34 Llama 2-70B 57.4% Jul 2023 Model card
35 Gemma 3 4B 56.2% Mar 2025 Model card
36 Mistral 7B v0.1 55.5% Sep 2023 Model card
37 Llama 2-34B 54.5% Jul 2023 Model card
38 Falcon-Arabic-7B 53.5% Nov 2025 Model card
39 Gemma 7B 53.2% Feb 2024 Model card
40 BitNet b1.58 2B4T 49.9% Apr 2025 Model card
41 Llama 2-13B 49.4% Jul 2023 Model card
42 Llama 2-7B 45.9% Jul 2023 Model card
43 Gemma 2B 42.1% Mar 2024 Model card
44 Gemma 3 1B 38.4% Mar 2025 Model card
45 Gemma 3 270M 28.2% Aug 2025 Model card

Epoch AI has scores for 75 models. We plot every source in the charts below.

7 models have held number one since 2023

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for Epoch AI’s runs and what labs claim in their cards.

OpenAI Stability AI Anthropic Microsoft DeepSeek Meta
Epoch AI runs Passed 85% in Mar 2022, by davinci-002
Llama 3.1-405B 95.3%
Lab claims Passed 85% in Mar 2023, by GPT-4
Llama 3.1-405B 96.9%

The best score passed 85% in Mar 2022

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Run or collected by Epoch AI Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2020 2021 2022 2023 2024 2025 2026 Near the ceiling (85%) Random guessing Llama 3.1-405B GPT-4 davinci-002 InstructGPT 175B GPT-J-6B GPT-2 (1.5B) 0% 20% 40% 60% 80% 100% 2020 2022 2024 2026 Near the ceiling (85%) Random guessing Llama 3.1-405B davinci-002

Labs dropped it from launch posts once the best score passed 85%

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

OpenAI
Google DeepMind
Anthropic
Mistral AI
Microsoft
Meta
Qwen
DeepSeek
Technology Innovation Institute
Sarvam AI
DateLaunch postScoreAs the post writes it
14 Aug 2025 Gemma 3 270M 28.2 ARC-c 0-shot (IT)
26 Jun 2025 Gemma 3n E4B 61.6 ARC-c 25-shot (PT)
23 May 2025 Sarvam-M 95 ARC-C (0.95)
20 May 2025 Falcon-H1-34B 61 ARC-C
15 Apr 2025 BitNet b1.58 2B4T 49.9 ARC-Challenge
12 Mar 2025 Gemma 3 27B 70.6 ARC-c 25-shot (PT)
26 Feb 2025 Phi-4 Mini 83.7 ARC Challenge (10-shot)
30 Jan 2025 Mistral Small 3 91.3 ARC-c (0-shot) [base]
26 Dec 2024 DeepSeek-V3 95.3 ARC-Challenge (Acc.)
19 Sep 2024 Qwen2.5-72B 72.4 ARC-C
20 Aug 2024 Phi-3.5-MoE 91 ARC Challenge (10-shot)
23 Jul 2024 Llama 3.1-405B 96.9 ARC Challenge (0-shot)
27 Jun 2024 Gemma 2 27B 71.4 ARC-c (25-shot)
7 Jun 2024 Qwen2-72B 68.9 ARC-C
22 Apr 2024 phi-3-medium 14B 91 Arc-C (10-Shot)
18 Apr 2024 Llama 3-70B 93 ARC-Challenge (25-shot) [pretrained]
4 Mar 2024 Claude 3 Opus 96.4 ARC-Challenge (25-shot)
26 Feb 2024 Mistral Large 94 Best of Arc C (25) [Arc C (5) = 94.2] / French Arc-C / German Arc-C / Spanish Arc-C / Italian Arc-C
21 Feb 2024 Gemma 7B 53.2 ARC-c
12 Dec 2023 Phi-2 ARC challenge (in Commonsense Reasoning average only)
11 Dec 2023 Mixtral 8x7B 85.8 Best of ARC Challenge (25-shot) [detailed table Arc-c: 59.7] / French Arc-c / German Arc-c / Spanish Arc-c / Italian Arc-c
27 Sep 2023 Mistral 7B v0.1 55.5 Arc-c
11 Jul 2023 Claude 2 91 ARC-Challenge (5-shot)
10 May 2023 PaLM 2-L 95.1 ARC-C (4), CoT + SC, Table 5 (Table 2 1-shot = 69.2)
14 Mar 2023 GPT-4 96.3 AI2 Reasoning Challenge (ARC) (25-shot)

39 system cards report it, from 9 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
8 Nov 2025 Falcon-Arabic-7B technical report 53.5 ARC Challenge 25-shot (Table 8) Capabilities
17 Sep 2025 Apertus 70B technical report 85 ARC Challenge Chat, 0-shot CoT, held-out test suite (Table 21). ARC Challenge Multilingual 37.3 Capabilities
30 Jul 2025 Falcon-H1-34B technical report 61 ARC-C, logprobs 0-shot (Table 24) Capabilities
16 Apr 2025 BitNet b1.58 2B4T technical report 49.9 ARC-Challange (0-shot; Acc,norm) (Table 1, Table 3) Capabilities
25 Mar 2025 Gemma 3 4B technical report 56.2 ARC-C, PT, 25-shot (Table 9) Capabilities
25 Mar 2025 Gemma 3 27B technical report 70.6 ARC-C, PT, 25-shot (Table 9) Capabilities
25 Mar 2025 Gemma 3 1B technical report 38.4 ARC-C, PT, 25-shot (Table 9) Capabilities
25 Mar 2025 Gemma 3 12B technical report 68.9 ARC-C, PT, 25-shot (Table 9) Capabilities
3 Mar 2025 Phi-4 Mini technical report 83.7 Arc-C (10-Shot) (Table 7) Capabilities
23 Nov 2024 Llama 3.1-8B technical report 83.4 ARC Challenge (0-shot), post-trained (Table 2) Capabilities
23 Nov 2024 Llama 3.1-70B technical report 94.8 ARC Challenge (0-shot), post-trained (Table 2) Capabilities
23 Nov 2024 Llama 3.1-405B technical report 96.9 ARC Challenge (0-shot), post-trained (Table 2) Capabilities
30 Aug 2024 phi-3-small 7.4B technical report 90.7 Arc-C (10-Shot) (Section 3 table) Capabilities
30 Aug 2024 phi-3-mini 3.8B technical report 84.9 Arc-C (10-Shot) (Section 3 table) Capabilities
30 Aug 2024 phi-3-medium 14B technical report 91.6 Arc-C (10-Shot) (Section 3 table) Capabilities
30 Aug 2024 Phi-3.5-MoE technical report 91 ARC Challenge (10-shot) (Table 3) Capabilities
30 Aug 2024 Phi-3.5-mini technical report 84.6 ARC Challenge (10-shot) (Table 3) Capabilities
31 Jul 2024 Gemma 2 9B technical report 68.4 ARC-C, PT, 25-shot (Table 13) Capabilities
31 Jul 2024 Gemma 2 27B technical report 71.4 ARC-C, PT, 25-shot (Table 13) Capabilities
23 Jul 2024 Llama 3.1-8B model card 83.4 ARC-C, 0-shot, acc, instruction tuned (Instruction tuned models table) Capabilities
23 Jul 2024 Llama 3.1-70B model card 94.8 ARC-C, 0-shot, acc, instruction tuned (Instruction tuned models table) Capabilities
23 Jul 2024 Llama 3.1-405B model card 96.9 ARC-C, 0-shot, acc, instruction tuned (Instruction tuned models table) Capabilities
7 May 2024 DeepSeek-V2 (MoE-236B, May 2024) technical report 92.3 ARC-Challenge 25-shot, DeepSeek-V2 Chat (RL) (Table 3) Capabilities
18 Apr 2024 Llama 3-8B model card 78.6 ARC-Challenge (25-shot), pretrained (Base pretrained models table) Capabilities
18 Apr 2024 Llama 3-70B model card 93 ARC-Challenge (25-shot), pretrained (Base pretrained models table) Capabilities
13 Mar 2024 Gemma 7B technical report 53.2 ARC-c, PT (Table 6). HuggingFace H6 run (Table 7): 61.9 Capabilities
13 Mar 2024 Gemma 2B technical report 42.1 ARC-c, PT (Table 6) Capabilities
4 Mar 2024 Claude 3 Sonnet model card 93.2 ARC-Challenge, 25-shot (Table 1) Capabilities
4 Mar 2024 Claude 3 Opus model card 96.4 ARC-Challenge, 25-shot (Table 1) Capabilities
4 Mar 2024 Claude 3 Haiku model card 89.2 ARC-Challenge, 25-shot (Table 1) Capabilities
8 Jan 2024 Mixtral 8x7B technical report 85.8 ARC Challenge, 25-shot (Table 3). 0-shot 59.7 (Table 2) Capabilities
5 Jan 2024 DeepSeek LLM 67B technical report 64.1 ARC-Challenge, DeepSeek 67B Chat (Table 6) Capabilities
10 Oct 2023 Mistral 7B v0.1 technical report 55.5 Arc-c, 0-shot (Table 2) Capabilities
19 Jul 2023 Llama 2-7B technical report 45.9 ARC-c, 0-shot, pretrained (Table 20) Capabilities
19 Jul 2023 Llama 2-70B technical report 57.4 ARC-c, 0-shot, pretrained (Table 20) Capabilities
19 Jul 2023 Llama 2-34B technical report 54.5 ARC-c, 0-shot, pretrained (Table 20) Capabilities
19 Jul 2023 Llama 2-13B technical report 49.4 ARC-c, 0-shot, pretrained (Table 20) Capabilities
11 Jul 2023 Claude 2 model card 91 ARC-Challenge (5-shot) (4.3) Capabilities
14 Mar 2023 GPT-4 technical report 96.3 AI2 Reasoning Challenge (ARC), Challenge-set, 25-shot (Table 2, base model) Capabilities

More in Knowledge

All 109 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • KMMLU-Pro In use Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. Solar Pro 4 79.2%
  • BIG-Bench Extra Hard In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. Gemma 4 31B IT 74.4%
  • HLE-Verified In use Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. Seed2.0 Pro 73.6%