Benchmark Almanac

LLM Confabulation Leaderboard

GPT-5 10.3% leads since Aug 2025 Not in model cards

Models answer misleading questions about recent articles whose answers are not in the text. The board ranks confabulation rate against non-response rate.

201 human-checked questions as of 2025-02-10. About 250 GitHub stars.

Made by Lech Mazur Released Oct 2024 Faithful to sources All tasks public Last model added Aug 2025 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Graded by a model A language model scores each response, usually against a rubric.
Answers
Short answer A number, word or short string matched to a key.
lechmazur/confabulations 251 stars Leaderboard lechmazur.github.io
60
Entries on its own leaderboard, 58 models we track

GPT-5 leads with 10.3% among models released by Aug 2025

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Aug 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 GPT-5 medium 10.3%
  2. 2 Gemini 2.5 Pro May 2025 10.6%
  3. 3 Gemini 2.5 Pro Mar 2025 10.8%
  4. 4 Grok-3 mini high 10.8%
  5. 5 GLM-4.5 11.3%
  6. 6 o1 medium 11.7%
  7. 7 Qwen3-30B-A3B 12.3%
  8. 8 Gemini 2.5 Pro Jun 2025 12.4%
  9. 9 Grok 4 12.4%
  10. 10 Gemini 2.0 Flash Thinking Jan 2025 12.4%
  11. 11 DeepSeek-R1 12.7%
  12. 12 o1-preview 13%
  13. 13 Claude Sonnet 4 thinking 16K 13.2%
  14. 14 GPT-5 mini medium 13.3%
  15. 15 Gemini 1.5 Pro Sep 2024 13.5%
  16. 16 GPT-4.5 13.6%
  17. 17 Grok 3 no reasoning 14.2%
  18. 18 o3-pro medium 14.2%
  19. 19 o3 high 14.4%
  20. 20 DeepSeek-R1 May 2025 14.6%
  21. 21 Claude 3.7 Sonnet thinking 16K 14.7%
  22. 22 GPT-4o Aug 2024 15.3%
  23. 23 QwQ-32B 15.6%
  24. 24 Qwen3-235B-A22B 15.6%
  25. 25 gpt-oss-120b medium 15.7%
  26. 26 o4-mini high 15.8%
  27. 27 Claude Opus 4 thinking 16K 15.9%
  28. 28 ERNIE-4.5-300B-A47B 16%
  29. 29 GPT-4o Feb 2025 16.6%
  30. 30 Qwen3-235B-A22B-Thinking Jul 2025 16.8%
  31. 31 Gemini 2.5 Flash Preview 24K 16.8%
  32. 32 Claude Opus 4.1 thinking 16K 17.1%
  33. 33 GPT-4o Nov 2024 17.2%
  34. 34 Llama 3.1-405B 17.6%
  35. 35 o3-mini medium 17.9%
  36. 36 Gemini 2.0 Pro 18.4%
  37. 37 o1-mini 18.6%
  38. 38 Qwen2.5-72B 19.1%
  39. 39 Claude 3.5 Sonnet (October 2024) 19.9%
  40. 40 Grok-2 Dec 2024 20.1%
  41. 41 Kimi K2 Jul 2025 20.4%
  42. 42 GPT-4o Mar 2025 20.8%
  43. 43 Mistral Large 2 Nov 2024 21.4%
  44. 44 Qwen2.5-Max 21.8%
  45. 45 Mistral Medium 3 21.9%
  46. 46 Llama 4 Maverick 22.6%
  47. 47 Claude 3 Opus 22.7%
  48. 48 Llama 3.3 70B 22.8%
  49. 49 MiniMax-Text-01 23.9%
  50. 50 Mistral Small 3 25.2%
  51. 51 DeepSeek-V3 Mar 2025 26.1%
  52. 52 Gemini 2.0 Flash Feb 2025 26.9%
  53. 53 Gemma 2 27B 27.1%
  54. 54 GPT-4 Turbo Apr 2024 28.4%
  55. 55 Phi-4 29.4%
  56. 56 Amazon Nova Pro 30.1%
  57. 57 Claude 3 Haiku 34.2%
  58. 58 Claude 3.5 Haiku 36.7%
  59. 59 GPT-4o mini 37.2%
  60. 60 Gemma 3 27B 40.3%
#EntryScoreSettingDateSource
1 GPT-5 10.3% medium Aug 2025 Official leaderboard
2 Gemini 2.5 Pro May 2025 10.6% May 2025 Official leaderboard
3 Gemini 2.5 Pro Mar 2025 10.8% Mar 2025 Official leaderboard
4 Grok-3 mini 10.8% high Apr 2025 Official leaderboard
5 GLM-4.5 11.3% Aug 2025 Official leaderboard
6 o1 11.7% medium Dec 2024 Official leaderboard
7 Qwen3-30B-A3B 12.3% Apr 2025 Official leaderboard
8 Gemini 2.5 Pro Jun 2025 12.4% Jun 2025 Official leaderboard
9 Grok 4 12.4% Jul 2025 Official leaderboard
10 Gemini 2.0 Flash Thinking Jan 2025 12.4% Jan 2025 Official leaderboard
11 DeepSeek-R1 12.7% Jan 2025 Official leaderboard
12 o1-preview 13% Sep 2024 Official leaderboard
13 Claude Sonnet 4 13.2% thinking 16K May 2025 Official leaderboard
14 GPT-5 mini 13.3% medium Aug 2025 Official leaderboard
15 Gemini 1.5 Pro Sep 2024 13.5% Sep 2024 Official leaderboard
16 GPT-4.5 13.6% Feb 2025 Official leaderboard
17 Grok 3 14.2% no reasoning Apr 2025 Official leaderboard
18 o3-pro 14.2% medium Jun 2025 Official leaderboard
19 o3 14.4% high Apr 2025 Official leaderboard
20 DeepSeek-R1 May 2025 14.6% May 2025 Official leaderboard
21 Claude 3.7 Sonnet 14.7% thinking 16K Feb 2025 Official leaderboard
22 GPT-4o Aug 2024 15.3% Aug 2024 Official leaderboard
23 QwQ-32B 15.6% Mar 2025 Official leaderboard
24 Qwen3-235B-A22B 15.6% Apr 2025 Official leaderboard
25 gpt-oss-120b 15.7% medium Aug 2025 Official leaderboard
26 o4-mini 15.8% high Apr 2025 Official leaderboard
27 Claude Opus 4 15.9% thinking 16K May 2025 Official leaderboard
28 ERNIE-4.5-300B-A47B 16% Jun 2025 Official leaderboard
29 GPT-4o Feb 2025 16.6% Official leaderboard
30 Qwen3-235B-A22B-Thinking Jul 2025 16.8% Jul 2025 Official leaderboard
31 Gemini 2.5 Flash Preview 24K 16.8% Official leaderboard
32 Claude Opus 4.1 17.1% thinking 16K Aug 2025 Official leaderboard
33 GPT-4o Nov 2024 17.2% Nov 2024 Official leaderboard
34 Llama 3.1-405B 17.6% Jul 2024 Official leaderboard
35 o3-mini 17.9% medium Jan 2025 Official leaderboard
36 Gemini 2.0 Pro 18.4% Feb 2025 Official leaderboard
37 o1-mini 18.6% Sep 2024 Official leaderboard
38 Qwen2.5-72B 19.1% Sep 2024 Official leaderboard
39 Claude 3.5 Sonnet (October 2024) 19.9% Oct 2024 Official leaderboard
40 Grok-2 Dec 2024 20.1% Dec 2024 Official leaderboard
41 Kimi K2 Jul 2025 20.4% Jul 2025 Official leaderboard
42 GPT-4o Mar 2025 20.8% Mar 2025 Official leaderboard
43 Mistral Large 2 Nov 2024 21.4% Nov 2024 Official leaderboard
44 Qwen2.5-Max 21.8% Jan 2025 Official leaderboard
45 Mistral Medium 3 21.9% May 2025 Official leaderboard
46 Llama 4 Maverick 22.6% Apr 2025 Official leaderboard
47 Claude 3 Opus 22.7% Feb 2024 Official leaderboard
48 Llama 3.3 70B 22.8% Dec 2024 Official leaderboard
49 MiniMax-Text-01 23.9% Jan 2025 Official leaderboard
50 Mistral Small 3 25.2% Jan 2025 Official leaderboard
51 DeepSeek-V3 Mar 2025 26.1% Mar 2025 Official leaderboard
52 Gemini 2.0 Flash Feb 2025 26.9% Feb 2025 Official leaderboard
53 Gemma 2 27B 27.1% Jun 2024 Official leaderboard
54 GPT-4 Turbo Apr 2024 28.4% Apr 2024 Official leaderboard
55 Phi-4 29.4% Dec 2024 Official leaderboard
56 Amazon Nova Pro 30.1% Dec 2024 Official leaderboard
57 Claude 3 Haiku 34.2% Mar 2024 Official leaderboard
58 Claude 3.5 Haiku 36.7% Oct 2024 Official leaderboard
59 GPT-4o mini 37.2% Jul 2024 Official leaderboard
60 Gemma 3 27B 40.3% Mar 2025 Official leaderboard

More effort adds a median 3 points

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
o3-mini
medium
18.4%
Claude Opus 4.1
no reasoning thinking 16K
18.5%
Claude Opus 4
no reasoning thinking 16K
17.1%
Claude 3.7 Sonnet
thinking 16K
19.8%
Claude Sonnet 4
no reasoning thinking 16K
14.8%
Grok-3 mini
low high
14%
ModelEffortHarnessScoreCost per taskOutput tokens
Gemma 3 27B 40.3%
GPT-4o mini 37.2%
Claude 3.5 Haiku 36.7%
Claude 3 Haiku 34.2%
Amazon Nova Pro 30.1%
Phi-4 29.4%
GPT-4 Turbo Apr 2024 28.4%
Gemma 2 27B 27.1%
Gemini 2.0 Flash Feb 2025 26.9%
DeepSeek-V3 Mar 2025 26.1%
Mistral Small 3 25.2%
MiniMax-Text-01 23.9%
Llama 3.3 70B 22.8%
Claude 3 Opus 22.7%
Llama 4 Maverick 22.6%
Mistral Medium 3 21.9%
Qwen2.5-Max 21.8%
Mistral Large 2 Nov 2024 21.4%
GPT-4o Mar 2025 20.8%
Kimi K2 Jul 2025 20.4%
Grok-2 Dec 2024 20.1%
Claude 3.5 Sonnet (October 2024) 19.9%
Claude 3.7 Sonnet 19.8%
Qwen2.5-72B 19.1%
o1-mini 18.6%
Claude Opus 4.1 no reasoning 18.5%
o3-mini high 18.4%
Gemini 2.0 Pro 18.4%
o3-mini medium 17.9%
Llama 3.1-405B 17.6%
GPT-4o Nov 2024 17.2%
Claude Opus 4.1 thinking 16K 17.1%
Claude Opus 4 no reasoning 17.1%
Qwen3-235B-A22B-Thinking Jul 2025 16.8%
ERNIE-4.5-300B-A47B 16%
Claude Opus 4 thinking 16K 15.9%
o4-mini high 15.8%
gpt-oss-120b medium 15.7%
Qwen3-235B-A22B 15.6%
QwQ-32B 15.6%
GPT-4o Aug 2024 15.3%
Claude Sonnet 4 no reasoning 14.8%
Claude 3.7 Sonnet thinking 16K 14.7%
DeepSeek-R1 May 2025 14.6%
o3 high 14.4%
o3-pro medium 14.2%
Grok 3 no reasoning 14.2%
Grok-3 mini low 14%
GPT-4.5 13.6%
Gemini 1.5 Pro Sep 2024 13.5%
GPT-5 mini medium 13.3%
Claude Sonnet 4 thinking 16K 13.2%
o1-preview 13%
DeepSeek-R1 12.7%
Gemini 2.0 Flash Thinking Jan 2025 12.4%
Grok 4 12.4%
Gemini 2.5 Pro Jun 2025 12.4%
Qwen3-30B-A3B 12.3%
o1 medium 11.7%
GLM-4.5 11.3%
Gemini 2.5 Pro Mar 2025 10.8%
Grok-3 mini high 10.8%
Gemini 2.5 Pro May 2025 10.6%
GPT-5 medium 10.3%

5 models have held number one since its release in Oct 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Google DeepMind
Own leaderboard Not tested since Aug 2025
GPT-5 10.3%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 2025 GPT-5 Gemini 2.5 Pro o1-preview Llama 3.1-405B Claude 3 Opus 0% 20% 40% 60% 80% 100% 2024 2025 GPT-5 GPT-4o

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Factuality

All 70 in Factuality
  • SimpleVQA In use Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references. Qwen3.7-Plus 81.7%
  • HallusionBench In use Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. Keye-VL-2.0-30B-A3B 73.5%
  • SimpleQA Verified In use 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge. Gemini 3.1 Pro 77.5%
  • GaRAGe In use Models write long-form RAG answers to 2,366 questions with human-annotated grounding passages. Scoring checks whether answers use the relevant passages. Nemotron 3.5 Bolt E4B 40.7%
  • Kimi Hallu Bench In use Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. Kimi K3.1 Pro 8.30%
  • RAGTruth In use A word-level annotated corpus of hallucinations in retrieval-augmented generation answers. It is used to train and evaluate hallucination detection and RAG faithfulness. Nemotron 3.5 Bolt E4B 77.9%