Benchmark Almanac

Vectara Hallucination Leaderboard (HHEM)

falcon-7B-instruct 29.9% tops its leaderboard Gemma 1.1 2B IT 27.8% leads the models we track Only in system cards

Models summarize short documents. The HHEM-2.1-Open classifier flags summaries that add unsupported facts, giving a hallucination rate.

Labs often report one minus the hallucination rate so that higher is better. The judge moved from HHEM-1.0 to HHEM-2.x in 2024-08. Vectara replaced the 1,000 article set in 2025-11.

Made by Vectara Released Nov 2023 Faithful to sources All tasks public Last in a system card Jul 2025 Last model added Oct 2025 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Graded by a model A language model scores each response, usually against a rubric.
Answers
Open-ended Free-form text, documents, designs or media judged as a whole.
vectara/hallucination-leaderboard 3,323 stars Dataset Leaderboard github.com
166
Entries on its own leaderboard, 148 models we track
1
System card reports it, from 1 lab

falcon-7B-instruct tops the leaderboard with 29.9%

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Gemma 1.1 2B IT, at 27.8%.

The newest model with a score here came out in Oct 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

166 entries. Scroll sideways for the rest.

  1. 29.9 falcon-7B-instruct
  2. 27.8 Gemma 1.1 2B IT
  3. 25.2 Qwen2.5-0.5B
  4. 24.8 OpenELM-3B-Instruct
  5. 20.7 Llama 3.2 1B
  6. 20.1 Mixtral 8x7B
  7. 18.3 Flan-T5-large
  8. 17.4 Claude 2
  9. 17.0 Gemma 1.1 7B IT
  10. 16.5 Granite-3.2-2B-Instruct
  11. 16.3 Claude 3 Sonnet
  12. 16.1 QwQ-32B-Preview
  13. 15.8 Qwen2.5-1.5B
  14. 15.7 Granite-3.1-2B-Instruct
  15. 14.8 Gemma 7B
  16. 14.3 DeepSeek-R1
  17. 14.1 PaLM-2
  18. 13.5 Titan-Express
  19. 12.2 Aya Expanse 8B
  20. 11.7 WizardLM-2 8x22B
  21. 11.4 Mistral Medium 3.1
  22. 11.3 Llama 2-7B
  23. 11.2 Mistral NeMo
  24. 11.1 OLMo-2-7B-Instruct
  25. 10.8 OLMo-2-13B-Instruct
  26. 10.5 Llama 2-13B
  27. 10.1 Claude 3 Opus
  28. 10.1 Gemma 2 9B
  29. 9.50 Mistral 7B v0.3
  30. 9.10 Gemini 1.5 Pro
  31. 8.80 Granite-3.0-2B-Instruct
  32. 8.70 Granite 3.2 8B Instruct
  33. 8.60 Mistral-Small2
  34. 8.60 Granite-3.1-8B-Instruct
  35. 8.50 Aya Expanse 32B
  36. 8.40 Granite-4.0-h-micro
  37. 8.30 DBRX
  38. 8.30 Qwen2-VL-2B-Instruct
  39. 8.30 Ministral 3B
  40. 8.00 DeepSeek-V3
  41. 7.90 Yi-1.5-6B-Chat
  42. 7.90 Llama 3.2 3B
  43. 7.70 Gemini 1.0 Pro
  44. 7.70 DeepSeek-R1
  45. 7.50 Ministral 8B
  46. 7.40 Llama 3-8B
  47. 7.30 Granite-4.0-h-tiny
  48. 7.00 Gemma 2 2B IT
  49. 7.00 Qwen2.5-3B
  50. 6.80 o3
  51. 6.80 o4-mini
  52. 6.70 Phi-2
  53. 6.60 Mistral-Pixtral
  54. 6.60 Gemini 1.5 Flash
  55. 6.50 Granite-3.0-8B-Instruct
  56. 6.50 granite-4.0-micro
  57. 6.20 Kimi K2
  58. 5.90 Llama 2-70B
  59. 5.60 Mistral Small 3.1
  60. 5.50 DeepSeek-TNG-R1T2-Chimera
  61. 5.50 Llama 3.2 11B
  62. 5.50 DeepSeek-V3.1
  63. 5.50 Claude Sonnet 4.5
  64. 5.40 Llama 3.1-8B
  65. 5.40 Command R+
  66. 5.30 Gemma 3 1B
  67. 5.00 Llama 3.1-70B
  68. 4.90 Command R (03-2024)
  69. 4.90 Yi-1.5-9B-Chat
  70. 4.90 Claude 3.5 Haiku
  71. 4.90 GPT-5
  72. 4.80 Claude Opus 4
  73. 4.80 Grok 4
  74. 4.80 Qwen3-30B-A3B-Thinking
  75. 4.70 Mixtral 8x22B
  76. 4.70 Qwen2-72B
  77. 4.70 Phi-4
  78. 4.70 Llama 4 Scout
  79. 4.70 GPT-5 nano
  80. 4.60 Grok
  81. 4.60 Claude 3.5 Sonnet
  82. 4.60 Jamba 1.6 Mini
  83. 4.60 Llama 4 Maverick
  84. 4.60 DeepSeek-V3.2-Exp
  85. 4.50 Claude 3.7 Sonnet
  86. 4.50 Cohere Command A
  87. 4.50 Claude Sonnet 4
  88. 4.50 Claude Haiku 4.5
  89. 4.40 Qwen3-1.7B
  90. 4.30 Qwen2.5-72B
  91. 4.30 Llama 3.2 90B
  92. 4.30 GLM-4.6
  93. 4.20 Qwen2-VL-7B-Instruct
  94. 4.20 Qwen2.5-14B
  95. 4.20 Claude Opus 4.1
  96. 4.10 Llama 3-70B
  97. 4.10 Phi-3.5-mini
  98. 4.10 Mistral Large 2
  99. 4.00 phi-3-mini 3.8B
  100. 4.00 Llama 3.3 70B Instruct
  101. 4.00 InternLM3-8B-Instruct
  102. 3.90 Llama 3.1-405B
  103. 3.90 DeepSeek-V3
  104. 3.80 Qwen3-Max
  105. 3.70 Yi-1.5-34B
  106. 3.70 Gemma 3 4B
  107. 3.70 Qwen3-0.6B
  108. 3.70 GLM-4.5
  109. 3.70 gpt-oss-20b
  110. 3.70 Qwen3-Next-80B-A3B
  111. 3.40 Phi-4 Mini
  112. 3.30 o1-preview
  113. 3.30 Grok-3 mini
  114. 3.20 GPT-5 mini
  115. 3.10 Phi-3-mini-128k-instruct
  116. 3.10 Mistral Small 3
  117. 3.00 Snowflake Arctic Instruct
  118. 3.00 Qwen2.5-32B
  119. 3.00 Gemma 3 27B
  120. 3.00 Qwen3-8B
  121. 2.90 Jamba 1.5 Mini
  122. 2.90 Grok 2 Vision (1212)
  123. 2.90 Qwen2.5-Max
  124. 2.90 Gemini 2.5 Flash-Lite
  125. 2.80 Qwen2.5-7B
  126. 2.80 Gemma 3 12B
  127. 2.80 Qwen3-32B
  128. 2.70 Qwen3-4B
  129. 2.60 Neural-Chat-7B-v3-3
  130. 2.60 Gemini 2.5 Pro
  131. 2.50 Orca-2-13b
  132. 2.50 Phi-3.5-MoE
  133. 2.40 DeepSeek-V2.5
  134. 2.40 o1
  135. 2.40 o1-pro
  136. 2.40 gpt-oss-120b
  137. 2.30 Jamba 1.6 Large
  138. 2.20 GPT-4.1 mini
  139. 2.20 Qwen3-14B
  140. 2.10 Grok 3
  141. 2.00 GPT-4.1
  142. 2.00 GPT-4.1 nano
  143. 1.90 GPT-3.5 Turbo
  144. 1.90 Grok-2
  145. 1.90 GLM-4.5-Air
  146. 1.80 GPT-4
  147. 1.80 Amazon Nova Lite
  148. 1.80 Amazon Nova Pro
  149. 1.80 Gemini 2.0 Flash Thinking
  150. 1.70 GPT-4 Turbo
  151. 1.70 GPT-4o mini
  152. 1.60 Amazon Nova Micro
  153. 1.50 GPT-4o
  154. 1.40 o1-mini
  155. 1.30 GLM-4-9B-Chat
  156. 1.30 Gemini 2.0 Flash
  157. 1.30 Gemini 2.5 Flash
  158. 1.20 Gemini 2.0 Flash-Lite
  159. 1.20 GPT-4.5
  160. 1.10 Gemini 2.5 Pro
  161. 1.10 Kimi K2
  162. 0.90 Mockingbird-2-Echo
  163. 0.80 o3-mini
  164. 0.80 Gemini 2.0 Pro
  165. 0.70 Gemini 2.0 Flash
  166. 0.60 Finix-S1-32B
  1. 1 falcon-7B-instruct 29.9%
  2. 2 Gemma 1.1 2B IT 27.8%
  3. 3 Qwen2.5-0.5B 25.2%
  4. 4 OpenELM-3B-Instruct 24.8%
  5. 5 Llama 3.2 1B 20.7%
  6. 6 Mixtral 8x7B 20.1%
  7. 7 Flan-T5-large 18.3%
  8. 8 Claude 2 17.4%
  9. 9 Gemma 1.1 7B IT 17%
  10. 10 Granite-3.2-2B-Instruct 16.5%
  11. 11 Claude 3 Sonnet 16.3%
  12. 12 QwQ-32B-Preview 16.1%
  13. 13 Qwen2.5-1.5B 15.8%
  14. 14 Granite-3.1-2B-Instruct 15.7%
  15. 15 Gemma 7B 14.8%
  16. 16 DeepSeek-R1 14.3%
  17. 17 PaLM-2 14.1%
  18. 18 Titan-Express 13.5%
  19. 19 Aya Expanse 8B 12.2%
  20. 20 WizardLM-2 8x22B 11.7%
  21. 21 Mistral Medium 3.1 11.4%
  22. 22 Llama 2-7B 11.3%
  23. 23 Mistral NeMo 11.2%
  24. 24 OLMo-2-7B-Instruct 11.1%
  25. 25 OLMo-2-13B-Instruct 10.8%
  26. 26 Llama 2-13B 10.5%
  27. 27 Claude 3 Opus 10.1%
  28. 28 Gemma 2 9B 10.1%
  29. 29 Mistral 7B v0.3 9.50%
  30. 30 Gemini 1.5 Pro Sep 2024 9.10%
  31. 31 Granite-3.0-2B-Instruct 8.80%
  32. 32 Granite 3.2 8B Instruct 8.70%
  33. 33 Mistral-Small2 8.60%
  34. 34 Granite-3.1-8B-Instruct 8.60%
  35. 35 Aya Expanse 32B 8.50%
  36. 36 Granite-4.0-h-micro 8.40%
  37. 37 DBRX 8.30%
  38. 38 Qwen2-VL-2B-Instruct 8.30%
  39. 39 Ministral 3B 8.30%
  40. 40 DeepSeek-V3 Mar 2025 8.00%
  41. 41 Yi-1.5-6B-Chat 7.90%
  42. 42 Llama 3.2 3B 7.90%
  43. 43 Gemini 1.0 Pro 7.70%
  44. 44 DeepSeek-R1 May 2025 7.70%
  45. 45 Ministral 8B 7.50%
  46. 46 Llama 3-8B 7.40%
  47. 47 Granite-4.0-h-tiny 7.30%
  48. 48 Gemma 2 2B IT 7.00%
  49. 49 Qwen2.5-3B 7.00%
  50. 50 o3 6.80%
  51. 51 o4-mini low 6.80%
  52. 52 Phi-2 6.70%
  53. 53 Mistral-Pixtral 6.60%
  54. 54 Gemini 1.5 Flash Sep 2024 6.60%
  55. 55 Granite-3.0-8B-Instruct 6.50%
  56. 56 granite-4.0-micro 6.50%
  57. 57 Kimi K2 Sep 2025 6.20%
  58. 58 Llama 2-70B 5.90%
  59. 59 Mistral Small 3.1 5.60%
  60. 60 DeepSeek-TNG-R1T2-Chimera 5.50%
  61. 61 Llama 3.2 11B 5.50%
  62. 62 DeepSeek-V3.1 5.50%
  63. 63 Claude Sonnet 4.5 5.50%
  64. 64 Llama 3.1-8B 5.40%
  65. 65 Command R+ 5.40%
  66. 66 Gemma 3 1B 5.30%
  67. 67 Llama 3.1-70B 5.00%
  68. 68 Command R (03-2024) 4.90%
  69. 69 Yi-1.5-9B-Chat 4.90%
  70. 70 Claude 3.5 Haiku 4.90%
  71. 71 GPT-5 minimal 4.90%
  72. 72 Claude Opus 4 4.80%
  73. 73 Grok 4 4.80%
  74. 74 Qwen3-30B-A3B-Thinking Jul 2025 4.80%
  75. 75 Mixtral 8x22B 4.70%
  76. 76 Qwen2-72B 4.70%
  77. 77 Phi-4 4.70%
  78. 78 Llama 4 Scout 4.70%
  79. 79 GPT-5 nano 4.70%
  80. 80 Grok 4.60%
  81. 81 Claude 3.5 Sonnet 4.60%
  82. 82 Jamba 1.6 Mini 4.60%
  83. 83 Llama 4 Maverick 4.60%
  84. 84 DeepSeek-V3.2-Exp 4.60%
  85. 85 Claude 3.7 Sonnet thinking 4.50%
  86. 86 Cohere Command A 4.50%
  87. 87 Claude Sonnet 4 4.50%
  88. 88 Claude Haiku 4.5 4.50%
  89. 89 Qwen3-1.7B 4.40%
  90. 90 Qwen2.5-72B 4.30%
  91. 91 Llama 3.2 90B 4.30%
  92. 92 GLM-4.6 4.30%
  93. 93 Qwen2-VL-7B-Instruct 4.20%
  94. 94 Qwen2.5-14B 4.20%
  95. 95 Claude Opus 4.1 4.20%
  96. 96 Llama 3-70B 4.10%
  97. 97 Phi-3.5-mini 4.10%
  98. 98 Mistral Large 2 Nov 2024 4.10%
  99. 99 phi-3-mini 3.8B 4.00%
  100. 100 Llama 3.3 70B Instruct 4.00%
  101. 101 InternLM3-8B-Instruct 4.00%
  102. 102 Llama 3.1-405B 3.90%
  103. 103 DeepSeek-V3 3.90%
  104. 104 Qwen3-Max 3.80%
  105. 105 Yi-1.5-34B 3.70%
  106. 106 Gemma 3 4B 3.70%
  107. 107 Qwen3-0.6B 3.70%
  108. 108 GLM-4.5 3.70%
  109. 109 gpt-oss-20b 3.70%
  110. 110 Qwen3-Next-80B-A3B thinking 3.70%
  111. 111 Phi-4 Mini 3.40%
  112. 112 o1-preview 3.30%
  113. 113 Grok-3 mini 3.30%
  114. 114 GPT-5 mini 3.20%
  115. 115 Phi-3-mini-128k-instruct 3.10%
  116. 116 Mistral Small 3 3.10%
  117. 117 Snowflake Arctic Instruct 3.00%
  118. 118 Qwen2.5-32B 3.00%
  119. 119 Gemma 3 27B 3.00%
  120. 120 Qwen3-8B 3.00%
  121. 121 Jamba 1.5 Mini 2.90%
  122. 122 Grok 2 Vision (1212) 2.90%
  123. 123 Qwen2.5-Max 2.90%
  124. 124 Gemini 2.5 Flash-Lite Jun 2025 2.90%
  125. 125 Qwen2.5-7B 2.80%
  126. 126 Gemma 3 12B 2.80%
  127. 127 Qwen3-32B 2.80%
  128. 128 Qwen3-4B 2.70%
  129. 129 Neural-Chat-7B-v3-3 2.60%
  130. 130 Gemini 2.5 Pro Jun 2025 2.60%
  131. 131 Orca-2-13b 2.50%
  132. 132 Phi-3.5-MoE 2.50%
  133. 133 DeepSeek-V2.5 Sep 2024 2.40%
  134. 134 o1 2.40%
  135. 135 o1-pro 2.40%
  136. 136 gpt-oss-120b 2.40%
  137. 137 Jamba 1.6 Large 2.30%
  138. 138 GPT-4.1 mini 2.20%
  139. 139 Qwen3-14B 2.20%
  140. 140 Grok 3 2.10%
  141. 141 GPT-4.1 2.00%
  142. 142 GPT-4.1 nano 2.00%
  143. 143 GPT-3.5 Turbo Jan 2024 1.90%
  144. 144 Grok-2 Dec 2024 1.90%
  145. 145 GLM-4.5-Air 1.90%
  146. 146 GPT-4 Jun 2023 1.80%
  147. 147 Amazon Nova Lite 1.80%
  148. 148 Amazon Nova Pro 1.80%
  149. 149 Gemini 2.0 Flash Thinking Jan 2025 1.80%
  150. 150 GPT-4 Turbo Apr 2024 1.70%
  151. 151 GPT-4o mini 1.70%
  152. 152 Amazon Nova Micro 1.60%
  153. 153 GPT-4o Mar 2025 1.50%
  154. 154 o1-mini 1.40%
  155. 155 GLM-4-9B-Chat 1.30%
  156. 156 Gemini 2.0 Flash Dec 2024 1.30%
  157. 157 Gemini 2.5 Flash Apr 2025 1.30%
  158. 158 Gemini 2.0 Flash-Lite 1.20%
  159. 159 GPT-4.5 1.20%
  160. 160 Gemini 2.5 Pro Mar 2025 1.10%
  161. 161 Kimi K2 Jul 2025 1.10%
  162. 162 Mockingbird-2-Echo 0.90%
  163. 163 o3-mini 0.80%
  164. 164 Gemini 2.0 Pro 0.80%
  165. 165 Gemini 2.0 Flash Feb 2025 0.70%
  166. 166 Finix-S1-32B 0.60%
#EntryScoreSettingDateSource
1 falcon-7B-instruct 29.9% Official leaderboard
2 Gemma 1.1 2B IT 27.8% Apr 2024 Official leaderboard
3 Qwen2.5-0.5B 25.2% Sep 2024 Official leaderboard
4 OpenELM-3B-Instruct 24.8% Official leaderboard
5 Llama 3.2 1B 20.7% Sep 2024 Official leaderboard
6 Mixtral 8x7B 20.1% Dec 2023 Official leaderboard
7 Flan-T5-large 18.3% Official leaderboard
8 Claude 2 17.4% Jul 2023 Official leaderboard
9 Gemma 1.1 7B IT 17% Apr 2024 Official leaderboard
10 Granite-3.2-2B-Instruct 16.5% Official leaderboard
11 Claude 3 Sonnet 16.3% Feb 2024 Official leaderboard
12 QwQ-32B-Preview 16.1% Nov 2024 Official leaderboard
13 Qwen2.5-1.5B 15.8% Sep 2024 Official leaderboard
14 Granite-3.1-2B-Instruct 15.7% Dec 2024 Official leaderboard
15 Gemma 7B 14.8% Feb 2024 Official leaderboard
16 DeepSeek-R1 14.3% Jan 2025 Official leaderboard
17 PaLM-2 14.1% Official leaderboard
18 Titan-Express 13.5% Official leaderboard
19 Aya Expanse 8B 12.2% Oct 2024 Official leaderboard
20 WizardLM-2 8x22B 11.7% Apr 2024 Official leaderboard
21 Mistral Medium 3.1 11.4% Aug 2025 Official leaderboard
22 Llama 2-7B 11.3% Jul 2023 Official leaderboard
23 Mistral NeMo 11.2% Jul 2024 Official leaderboard
24 OLMo-2-7B-Instruct 11.1% Official leaderboard
25 OLMo-2-13B-Instruct 10.8% Official leaderboard
26 Llama 2-13B 10.5% Jul 2023 Official leaderboard
27 Claude 3 Opus 10.1% Feb 2024 Official leaderboard
28 Gemma 2 9B 10.1% Jun 2024 Official leaderboard
29 Mistral 7B v0.3 9.50% Sep 2023 Official leaderboard
30 Gemini 1.5 Pro Sep 2024 9.10% Sep 2024 Official leaderboard
31 Granite-3.0-2B-Instruct 8.80% Oct 2024 Official leaderboard
32 Granite 3.2 8B Instruct 8.70% Feb 2025 Official leaderboard
33 Mistral-Small2 8.60% Official leaderboard
34 Granite-3.1-8B-Instruct 8.60% Dec 2024 Official leaderboard
35 Aya Expanse 32B 8.50% Oct 2024 Official leaderboard
36 Granite-4.0-h-micro 8.40% Official leaderboard
37 DBRX 8.30% Mar 2024 Official leaderboard
38 Qwen2-VL-2B-Instruct 8.30% Aug 2024 Official leaderboard
39 Ministral 3B 8.30% Oct 2024 Official leaderboard
40 DeepSeek-V3 Mar 2025 8.00% Mar 2025 Official leaderboard
41 Yi-1.5-6B-Chat 7.90% May 2024 Official leaderboard
42 Llama 3.2 3B 7.90% Sep 2024 Official leaderboard
43 Gemini 1.0 Pro 7.70% Dec 2023 Official leaderboard
44 DeepSeek-R1 May 2025 7.70% May 2025 Official leaderboard
45 Ministral 8B 7.50% Oct 2024 Official leaderboard
46 Llama 3-8B 7.40% Apr 2024 Official leaderboard
47 Granite-4.0-h-tiny 7.30% Official leaderboard
48 Gemma 2 2B IT 7.00% Jul 2024 Official leaderboard
49 Qwen2.5-3B 7.00% Sep 2024 Official leaderboard
50 o3 6.80% Apr 2025 Official leaderboard
51 o4-mini 6.80% low Apr 2025 Official leaderboard
52 Phi-2 6.70% Dec 2023 Official leaderboard
53 Mistral-Pixtral 6.60% Official leaderboard
54 Gemini 1.5 Flash Sep 2024 6.60% Aug 2024 Official leaderboard
55 Granite-3.0-8B-Instruct 6.50% Oct 2024 Official leaderboard
56 granite-4.0-micro 6.50% Oct 2025 Official leaderboard
57 Kimi K2 Sep 2025 6.20% Sep 2025 Official leaderboard
58 Llama 2-70B 5.90% Jul 2023 Official leaderboard
59 Mistral Small 3.1 5.60% Mar 2025 Official leaderboard
60 DeepSeek-TNG-R1T2-Chimera 5.50% Official leaderboard
61 Llama 3.2 11B 5.50% Sep 2024 Official leaderboard
62 DeepSeek-V3.1 5.50% Aug 2025 Official leaderboard
63 Claude Sonnet 4.5 5.50% Sep 2025 Official leaderboard
64 Llama 3.1-8B 5.40% Jul 2024 Official leaderboard
65 Command R+ 5.40% Aug 2024 Official leaderboard
66 Gemma 3 1B 5.30% Mar 2025 Official leaderboard
67 Llama 3.1-70B 5.00% Jul 2024 Official leaderboard
68 Command R (03-2024) 4.90% Mar 2024 Official leaderboard
69 Yi-1.5-9B-Chat 4.90% May 2024 Official leaderboard
70 Claude 3.5 Haiku 4.90% Oct 2024 Official leaderboard
71 GPT-5 4.90% minimal Aug 2025 Official leaderboard
72 Claude Opus 4 4.80% May 2025 Official leaderboard
73 Grok 4 4.80% Jul 2025 Official leaderboard
74 Qwen3-30B-A3B-Thinking Jul 2025 4.80% Jul 2025 Official leaderboard
75 Mixtral 8x22B 4.70% Apr 2024 Official leaderboard
76 Qwen2-72B 4.70% Jun 2024 Official leaderboard
77 Phi-4 4.70% Dec 2024 Official leaderboard
78 Llama 4 Scout 4.70% Apr 2025 Official leaderboard
79 GPT-5 nano 4.70% Aug 2025 Official leaderboard
80 Grok 4.60% Official leaderboard
81 Claude 3.5 Sonnet 4.60% Jun 2024 Official leaderboard
82 Jamba 1.6 Mini 4.60% Mar 2025 Official leaderboard
83 Llama 4 Maverick 4.60% Apr 2025 Official leaderboard
84 DeepSeek-V3.2-Exp 4.60% Sep 2025 Official leaderboard
85 Claude 3.7 Sonnet 4.50% thinking Feb 2025 Official leaderboard
86 Cohere Command A 4.50% Mar 2025 Official leaderboard
87 Claude Sonnet 4 4.50% May 2025 Official leaderboard
88 Claude Haiku 4.5 4.50% Oct 2025 Official leaderboard
89 Qwen3-1.7B 4.40% Apr 2025 Official leaderboard
90 Qwen2.5-72B 4.30% Sep 2024 Official leaderboard
91 Llama 3.2 90B 4.30% Sep 2024 Official leaderboard
92 GLM-4.6 4.30% Sep 2025 Official leaderboard
93 Qwen2-VL-7B-Instruct 4.20% Aug 2024 Official leaderboard
94 Qwen2.5-14B 4.20% Sep 2024 Official leaderboard
95 Claude Opus 4.1 4.20% Aug 2025 Official leaderboard
96 Llama 3-70B 4.10% Apr 2024 Official leaderboard
97 Phi-3.5-mini 4.10% Aug 2024 Official leaderboard
98 Mistral Large 2 Nov 2024 4.10% Nov 2024 Official leaderboard
99 phi-3-mini 3.8B 4.00% Apr 2024 Official leaderboard
100 Llama 3.3 70B Instruct 4.00% Dec 2024 Official leaderboard
101 InternLM3-8B-Instruct 4.00% Jan 2025 Official leaderboard
102 Llama 3.1-405B 3.90% Jul 2024 Official leaderboard
103 DeepSeek-V3 3.90% Dec 2024 Official leaderboard
104 Qwen3-Max 3.80% Sep 2025 Official leaderboard
105 Yi-1.5-34B 3.70% May 2024 Official leaderboard
106 Gemma 3 4B 3.70% Mar 2025 Official leaderboard
107 Qwen3-0.6B 3.70% Apr 2025 Official leaderboard
108 GLM-4.5 3.70% Aug 2025 Official leaderboard
109 gpt-oss-20b 3.70% Aug 2025 Official leaderboard
110 Qwen3-Next-80B-A3B 3.70% thinking Sep 2025 Official leaderboard
111 Phi-4 Mini 3.40% Feb 2025 Official leaderboard
112 o1-preview 3.30% Sep 2024 Official leaderboard
113 Grok-3 mini 3.30% Apr 2025 Official leaderboard
114 GPT-5 mini 3.20% Aug 2025 Official leaderboard
115 Phi-3-mini-128k-instruct 3.10% Apr 2024 Official leaderboard
116 Mistral Small 3 3.10% Jan 2025 Official leaderboard
117 Snowflake Arctic Instruct 3.00% Apr 2024 Official leaderboard
118 Qwen2.5-32B 3.00% Sep 2024 Official leaderboard
119 Gemma 3 27B 3.00% Mar 2025 Official leaderboard
120 Qwen3-8B 3.00% Apr 2025 Official leaderboard
121 Jamba 1.5 Mini 2.90% Aug 2024 Official leaderboard
122 Grok 2 Vision (1212) 2.90% Dec 2024 Official leaderboard
123 Qwen2.5-Max 2.90% Jan 2025 Official leaderboard
124 Gemini 2.5 Flash-Lite Jun 2025 2.90% Jun 2025 Official leaderboard
125 Qwen2.5-7B 2.80% Sep 2024 Official leaderboard
126 Gemma 3 12B 2.80% Mar 2025 Official leaderboard
127 Qwen3-32B 2.80% Apr 2025 Official leaderboard
128 Qwen3-4B 2.70% Apr 2025 Official leaderboard
129 Neural-Chat-7B-v3-3 2.60% Official leaderboard
130 Gemini 2.5 Pro Jun 2025 2.60% Jun 2025 Official leaderboard
131 Orca-2-13b 2.50% Official leaderboard
132 Phi-3.5-MoE 2.50% Aug 2024 Official leaderboard
133 DeepSeek-V2.5 Sep 2024 2.40% Sep 2024 Official leaderboard
134 o1 2.40% Dec 2024 Official leaderboard
135 o1-pro 2.40% Mar 2025 Official leaderboard
136 gpt-oss-120b 2.40% Aug 2025 Official leaderboard
137 Jamba 1.6 Large 2.30% Mar 2025 Official leaderboard
138 GPT-4.1 mini 2.20% Apr 2025 Official leaderboard
139 Qwen3-14B 2.20% Apr 2025 Official leaderboard
140 Grok 3 2.10% Apr 2025 Official leaderboard
141 GPT-4.1 2.00% Apr 2025 Official leaderboard
142 GPT-4.1 nano 2.00% Apr 2025 Official leaderboard
143 GPT-3.5 Turbo Jan 2024 1.90% Jan 2024 Official leaderboard
144 Grok-2 Dec 2024 1.90% Dec 2024 Official leaderboard
145 GLM-4.5-Air 1.90% Jul 2025 Official leaderboard
146 GPT-4 Jun 2023 1.80% Jun 2023 Official leaderboard
147 Amazon Nova Lite 1.80% Dec 2024 Official leaderboard
148 Amazon Nova Pro 1.80% Dec 2024 Official leaderboard
149 Gemini 2.0 Flash Thinking Jan 2025 1.80% Jan 2025 Official leaderboard
150 GPT-4 Turbo Apr 2024 1.70% Apr 2024 Official leaderboard
151 GPT-4o mini 1.70% Jul 2024 Official leaderboard
152 Amazon Nova Micro 1.60% Dec 2024 Official leaderboard
153 GPT-4o Mar 2025 1.50% Mar 2025 Official leaderboard
154 o1-mini 1.40% Sep 2024 Official leaderboard
155 GLM-4-9B-Chat 1.30% Jun 2024 Official leaderboard
156 Gemini 2.0 Flash Dec 2024 1.30% Dec 2024 Official leaderboard
157 Gemini 2.5 Flash Apr 2025 1.30% Apr 2025 Official leaderboard
158 Gemini 2.0 Flash-Lite 1.20% Feb 2025 Official leaderboard
159 GPT-4.5 1.20% Feb 2025 Official leaderboard
160 Gemini 2.5 Pro Mar 2025 1.10% Mar 2025 Official leaderboard
161 Kimi K2 Jul 2025 1.10% Jul 2025 Official leaderboard
162 Mockingbird-2-Echo 0.90% Official leaderboard
163 o3-mini 0.80% Jan 2025 Official leaderboard
164 Gemini 2.0 Pro 0.80% Feb 2025 Official leaderboard
165 Gemini 2.0 Flash Feb 2025 0.70% Feb 2025 Official leaderboard
166 Finix-S1-32B 0.60% Official leaderboard

Labs report scores for 1 model in their cards. We plot every source in the charts below.

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
o4-mini
low high
6.80%
GPT-5
minimal high
4.90%
Claude 3.7 Sonnet
thinking
4.50%
ModelEffortHarnessScoreCost per taskOutput tokens
Gemma 1.1 2B IT 27.8%
Qwen2.5-0.5B 25.2%
Llama 3.2 1B 20.7%
Mixtral 8x7B 20.1%
Claude 2 17.4%
Gemma 1.1 7B IT 17%
Claude 3 Sonnet 16.3%
QwQ-32B-Preview 16.1%
Qwen2.5-1.5B 15.8%
Granite-3.1-2B-Instruct 15.7%
Gemma 7B 14.8%
DeepSeek-R1 14.3%
Aya Expanse 8B 12.2%
WizardLM-2 8x22B 11.7%
Mistral Medium 3.1 11.4%
Llama 2-7B 11.3%
Mistral NeMo 11.2%
Llama 2-13B 10.5%
Claude 3 Opus 10.1%
Gemma 2 9B 10.1%
Mistral 7B v0.3 9.50%
Gemini 1.5 Pro Sep 2024 9.10%
Granite-3.0-2B-Instruct 8.80%
Granite 3.2 8B Instruct 8.70%
Granite-3.1-8B-Instruct 8.60%
Aya Expanse 32B 8.50%
DBRX 8.30%
Qwen2-VL-2B-Instruct 8.30%
Ministral 3B 8.30%
DeepSeek-V3 Mar 2025 8.00%
Yi-1.5-6B-Chat 7.90%
Llama 3.2 3B 7.90%
Gemini 1.0 Pro 7.70%
DeepSeek-R1 May 2025 7.70%
Ministral 8B 7.50%
Llama 3-8B 7.40%
Gemma 2 2B IT 7.00%
Qwen2.5-3B 7.00%
o3 6.80%
o4-mini low 6.80%
Phi-2 6.70%
Gemini 1.5 Flash Sep 2024 6.60%
Gemini 1.5 Pro Sep 2024 6.60%
Granite-3.0-8B-Instruct 6.50%
granite-4.0-micro 6.50%
Kimi K2 Sep 2025 6.20%
Llama 2-70B 5.90%
Mistral Small 3.1 5.60%
Llama 3.2 11B 5.50%
DeepSeek-V3.1 5.50%
Claude Sonnet 4.5 5.50%
Llama 3.1-8B 5.40%
Command R+ 5.40%
Gemma 3 1B 5.30%
o4-mini high 5.10%
Llama 3.1-70B 5.00%
Command R (03-2024) 4.90%
Yi-1.5-9B-Chat 4.90%
Claude 3.5 Haiku 4.90%
GPT-5 minimal 4.90%
Claude Opus 4 4.80%
Grok 4 4.80%
Qwen3-30B-A3B-Thinking Jul 2025 4.80%
Mixtral 8x22B 4.70%
Qwen2-72B 4.70%
Phi-4 4.70%
Llama 4 Scout 4.70%
GPT-5 nano 4.70%
Claude 3.5 Sonnet 4.60%
Jamba 1.6 Mini 4.60%
Llama 4 Maverick 4.60%
o4-mini 4.60%
DeepSeek-V3.2-Exp 4.60%
Claude 3.7 Sonnet thinking 4.50%
Cohere Command A 4.50%
Claude Sonnet 4 4.50%
Claude Haiku 4.5 4.50%
Claude 3.7 Sonnet 4.40%
Qwen3-1.7B 4.40%
Qwen2.5-72B 4.30%
Llama 3.2 90B 4.30%
GLM-4.6 4.30%
Qwen2-VL-7B-Instruct 4.20%
Qwen2.5-14B 4.20%
Claude Opus 4.1 4.20%
Llama 3-70B 4.10%
Phi-3.5-mini 4.10%
Mistral Large 2 Nov 2024 4.10%
phi-3-mini 3.8B 4.00%
Llama 3.3 70B Instruct 4.00%
InternLM3-8B-Instruct 4.00%
Llama 3.1-405B 3.90%
DeepSeek-V3 3.90%
Qwen3-Max 3.80%
Yi-1.5-34B 3.70%
Gemma 3 4B 3.70%
Qwen3-0.6B 3.70%
GLM-4.5 3.70%
gpt-oss-20b 3.70%
Qwen3-Next-80B-A3B thinking 3.70%
Gemini 1.5 Flash Sep 2024 3.40%
Phi-4 Mini 3.40%
o1-preview 3.30%
Grok-3 mini 3.30%
GPT-5 mini 3.20%
Phi-3-mini-128k-instruct 3.10%
Mistral Small 3 3.10%
Snowflake Arctic Instruct 3.00%
Qwen2.5-32B 3.00%
Gemma 3 27B 3.00%
Qwen3-8B 3.00%
Jamba 1.5 Mini 2.90%
Grok 2 Vision (1212) 2.90%
Qwen2.5-Max 2.90%
Gemini 2.5 Flash-Lite Jun 2025 2.90%
Qwen2.5-7B 2.80%
Gemma 3 12B 2.80%
Qwen3-32B 2.80%
Qwen3-4B 2.70%
Gemini 2.5 Pro Jun 2025 2.60%
Phi-3.5-MoE 2.50%
DeepSeek-V2.5 Sep 2024 2.40%
o1 2.40%
o1-pro 2.40%
gpt-oss-120b 2.40%
Jamba 1.6 Large 2.30%
GPT-4.1 mini 2.20%
Qwen3-14B 2.20%
Grok 3 2.10%
GPT-4.1 2.00%
GPT-4.1 nano 2.00%
GPT-3.5 Turbo Jan 2024 1.90%
Grok-2 Dec 2024 1.90%
GLM-4.5-Air 1.90%
GPT-4 Jun 2023 1.80%
Amazon Nova Lite 1.80%
Amazon Nova Pro 1.80%
Gemini 2.0 Flash Thinking Jan 2025 1.80%
GPT-4 Turbo Apr 2024 1.70%
GPT-4o mini 1.70%
Amazon Nova Micro 1.60%
GPT-4o Mar 2025 1.50%
o1-mini 1.40%
GPT-5 high 1.40%
GLM-4-9B-Chat 1.30%
Gemini 2.0 Flash Dec 2024 1.30%
Gemini 2.5 Flash Apr 2025 1.30%
Gemini 2.0 Flash-Lite 1.20%
GPT-4.5 1.20%
Gemini 2.5 Pro Mar 2025 1.10%
Kimi K2 Jul 2025 1.10%
o3-mini 0.80%
Gemini 2.0 Pro 0.80%
Gemini 2.0 Flash Feb 2025 0.70%

3 models have held number one since its release in Nov 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic Mistral AI Google DeepMind
Own leaderboard Replaced by 2, Nov 2025
Gemma 1.1 2B IT 27.8%

The best score passed 85% in Jul 2025

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2023 2024 2025 Near the ceiling (85%) Kimi K2 Gemma 1.1 2B IT Mixtral 8x7B GPT-4 0% 20% 40% 60% 80% 100% 2023 2024 2025 Near the ceiling (85%) Kimi K2 Gemma 1.1 2B IT

Labs report it only in system cards

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark. Some system cards do, below.

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
21 Jul 2025 Kimi K2 technical report 98.9 HHEM v2.1 (1-Hallu.), hallucination rate 1.1% (Table 3) Hallucination

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

Anthropic Mistral AI Google DeepMind Moonshot AI OpenAI
Vectara Hallucination Leaderboard (2025 dataset) 6 leaders
GPT-5.4 Nano 3.10%
Vectara Hallucination Leaderboard (HHEM) Passed 85% in Jul 2025, by Kimi K2
Kimi K2 98.9%
Vectara Hallucination Leaderboard (2025 dataset) 2 leaders
GPT-5.4 Nano 3.10%
Replaced by Vectara Hallucination Leaderboard (2025 dataset)

More in Factuality

All 70 in Factuality
  • SimpleVQA In use Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references. Qwen3.7-Plus 81.7%
  • HallusionBench In use Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. Keye-VL-2.0-30B-A3B 73.5%
  • SimpleQA Verified In use 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge. Gemini 3.1 Pro 77.5%
  • GaRAGe In use Models write long-form RAG answers to 2,366 questions with human-annotated grounding passages. Scoring checks whether answers use the relevant passages. Nemotron 3.5 Bolt E4B 40.7%
  • Kimi Hallu Bench In use Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. Kimi K3.1 Pro 8.30%
  • RAGTruth In use A word-level annotated corpus of hallucinations in retrieval-augmented generation answers. It is used to train and evaluate hallucination detection and RAG faithfulness. Nemotron 3.5 Bolt E4B 77.9%