Benchmark Almanac

AlpacaEval

GPT-4 Turbo Nov 2023 97.7% leads since Nov 2023 Not in model cards

Models answer 805 open instructions, and a GPT-4 judge compares each reply with text-davinci-003's reply. The score is the win rate.

AlpacaEval 2.0 replaced the baseline and the judge on 2024-01-03, so 1.0 and 2.0 scores are not comparable.

Made by Stanford Tatsu Lab Released May 2023 Overall chat quality All tasks public Last model added Dec 2023 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Graded by a model A language model scores each response, usually against a rubric.
Answers
Open-ended Free-form text, documents, designs or media judged as a whole.
tatsu-lab/alpaca_eval 2,023 stars Leaderboard
99
Entries on its own leaderboard, 19 models we track

GPT-4 Turbo leads with 97.7% among models released by Dec 2023

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Dec 2023, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

99 entries. Scroll sideways for the rest.

  1. 97.7 GPT-4 Turbo
  2. 97.6 XwinLM 70b V0.3
  3. 96.8 Mistral Medium
  4. 95.6 XwinLM 70b V0.1
  5. 95.4 PairRM 0.4B+Tulu 2+DPO 70B (best-of-16)
  6. 95.0 Tulu 2+DPO 70B
  7. 94.9 Mistral-7B+RAHF-DUAL+LoRA
  8. 94.8 GPT-4
  9. 94.8 Mixtral 8x7B
  10. 94.4 Mistral-7B-ReMax-v0.1
  11. 94.1 Yi-34B
  12. 93.8 GPT-4
  13. 93.4 GPT-3.5 Turbo
  14. 93.4 PairRM 0.4B+Zephyr 7B Beta (best-of-16)
  15. 92.8 UltraLM 13B V2.0 (best-of-16)
  16. 92.8 Mistral 7B v0.2
  17. 92.7 LLaMA2 Chat 70B
  18. 92.2 LMCocktail-10.7B-v1
  19. 91.8 XwinLM 13b V0.1
  20. 91.5 UltraLM 13B (best-of-16)
  21. 91.4 CUT 13B
  22. 91.4 Claude 2
  23. 91.1 PairRM 0.4B+Tulu 2+DPO 13B (best-of-16)
  24. 90.6 Cohere Command
  25. 90.6 Zephyr 7B Beta
  26. 90.1 DEITA 7B v1.0
  27. 89.5 OpenChat V3.1 13B
  28. 89.4 GPT-3.5 Turbo (Mar 2023)
  29. 89.3 Evo v2 7B
  30. 89.2 WizardLM 13B V1.2
  31. 89.0 Vicuna 33B v1.3
  32. 88.3 CausalLM-14B
  33. 88.1 Tulu 2+DPO 13B
  34. 87.9 Humpback LLaMa2 70B
  35. 87.8 XwinLM 7b V0.1
  36. 87.7 OpenBudddy-LLaMA2-70B-v10.1
  37. 87.1 OpenChat V2-W 13B
  38. 87.1 Claude 2.1
  39. 86.5 OpenBuddy-LLaMA-65B-v8
  40. 86.3 WizardLM 13B V1.1
  41. 86.3 UltraLM 13B V2.0
  42. 86.3 GPT-3.5 Turbo
  43. 85.8 Zephyr 7B Alpha
  44. 85.0 OpenChat V2 13B
  45. 84.2 Tulu 2+DPO 7B
  46. 83.7 Humpback LLaMa 65B
  47. 83.5 Recycled WizardLM 7B V2.0
  48. 82.3 Phi-2 DPO
  49. 82.1 Vicuna-13B-v1.3
  50. 81.9 PlatoLM 7B
  51. 81.9 LLaMA2 Chat 7B Evol70k-NEFT
  52. 81.5 OpenBuddy-LLaMA-30B-v7.1
  53. 81.1 LLaMA2 Chat 13B
  54. 80.9 OpenChat-13B
  55. 80.7 OpenBuddy-Falcon-40B-v9
  56. 80.6 UltraLM 13B
  57. 79.7 Gemini 1.0 Pro
  58. 79.5 OpenChat8192-13B
  59. 79.2 Evo 7B
  60. 78.9 Claude2 Alpaca 13B
  61. 78.9 Recycled WizardLM 7B V1.0
  62. 78.7 OpenCoderPlus-15B
  63. 78.5 MiniChat 1.5 3B
  64. 77.5 OpenBudddy-LLaMA2-13B-v11.1
  65. 76.8 Vicuna 7B v1.3
  66. 75.3 WizardLM 13B
  67. 74.1 JinaChat
  68. 73.9 airoboros 65B
  69. 73.3 airoboros 33B
  70. 71.8 Guanaco 65B
  71. 71.4 LLaMA2 Chat 7B
  72. 70.4 Ghost 7B Alpha
  73. 70.4 Vicuna 13B
  74. 70.4 OpenBuddy-Falcon-7b-v6
  75. 68.5 Phi-2 SFT
  76. 67.0 Baize-v2 13B
  77. 66.5 LLaMA 33B OASST RLHF
  78. 66.0 Minotaur 13B
  79. 66.0 Guanaco 33B
  80. 65.5 Nous Hermes 13B
  81. 64.4 Vicuna 7B
  82. 63.9 Baize-v2 7B
  83. 61.6 Alpaca-7B-NEFT
  84. 55.0 LLaMA 33B OASST SFT
  85. 52.6 Guanaco 13B
  86. 50.0 davinci-003
  87. 48.8 MiniChat 3B
  88. 47.1 chatglm2-6b
  89. 46.6 Guanaco 7B
  90. 45.7 Falcon 40B Instruct
  91. 45.7 Falcon 7B Instruct
  92. 44.1 Alpaca Farm PPO Sim (GPT-4) 7B
  93. 41.9 Pythia 12B SFT
  94. 41.2 Alpaca Farm PPO Human 7B
  95. 30.7 Phi-2
  96. 26.5 Alpaca 7B
  97. 26.0 Pythia 12B OASST SFT
  98. 21.8 Baichuan-13B-Chat
  99. 15.2 Davinci001
  1. 1 GPT-4 Turbo Nov 2023 97.7%
  2. 2 XwinLM 70b V0.3 97.6%
  3. 3 Mistral Medium 96.8%
  4. 4 XwinLM 70b V0.1 95.6%
  5. 5 PairRM 0.4B+Tulu 2+DPO 70B (best-of-16) 95.4%
  6. 6 Tulu 2+DPO 70B 95%
  7. 7 Mistral-7B+RAHF-DUAL+LoRA 94.9%
  8. 8 GPT-4 Mar 2023 94.8%
  9. 9 Mixtral 8x7B 94.8%
  10. 10 Mistral-7B-ReMax-v0.1 94.4%
  11. 11 Yi-34B 94.1%
  12. 12 GPT-4 Jun 2023 93.8%
  13. 13 GPT-3.5 Turbo Jun 2023 93.4%
  14. 14 PairRM 0.4B+Zephyr 7B Beta (best-of-16) 93.4%
  15. 15 UltraLM 13B V2.0 (best-of-16) 92.8%
  16. 16 Mistral 7B v0.2 92.8%
  17. 17 LLaMA2 Chat 70B 92.7%
  18. 18 LMCocktail-10.7B-v1 92.2%
  19. 19 XwinLM 13b V0.1 91.8%
  20. 20 UltraLM 13B (best-of-16) 91.5%
  21. 21 CUT 13B 91.4%
  22. 22 Claude 2 91.4%
  23. 23 PairRM 0.4B+Tulu 2+DPO 13B (best-of-16) 91.1%
  24. 24 Cohere Command 90.6%
  25. 25 Zephyr 7B Beta 90.6%
  26. 26 DEITA 7B v1.0 90.1%
  27. 27 OpenChat V3.1 13B 89.5%
  28. 28 GPT-3.5 Turbo (Mar 2023) 89.4%
  29. 29 Evo v2 7B 89.3%
  30. 30 WizardLM 13B V1.2 89.2%
  31. 31 Vicuna 33B v1.3 89%
  32. 32 CausalLM-14B 88.3%
  33. 33 Tulu 2+DPO 13B 88.1%
  34. 34 Humpback LLaMa2 70B 87.9%
  35. 35 XwinLM 7b V0.1 87.8%
  36. 36 OpenBudddy-LLaMA2-70B-v10.1 87.7%
  37. 37 OpenChat V2-W 13B 87.1%
  38. 38 Claude 2.1 87.1%
  39. 39 OpenBuddy-LLaMA-65B-v8 86.5%
  40. 40 WizardLM 13B V1.1 86.3%
  41. 41 UltraLM 13B V2.0 86.3%
  42. 42 GPT-3.5 Turbo Nov 2023 86.3%
  43. 43 Zephyr 7B Alpha 85.8%
  44. 44 OpenChat V2 13B 85%
  45. 45 Tulu 2+DPO 7B 84.2%
  46. 46 Humpback LLaMa 65B 83.7%
  47. 47 Recycled WizardLM 7B V2.0 83.5%
  48. 48 Phi-2 DPO 82.3%
  49. 49 Vicuna-13B-v1.3 82.1%
  50. 50 PlatoLM 7B 81.9%
  51. 51 LLaMA2 Chat 7B Evol70k-NEFT 81.9%
  52. 52 OpenBuddy-LLaMA-30B-v7.1 81.5%
  53. 53 LLaMA2 Chat 13B 81.1%
  54. 54 OpenChat-13B 80.9%
  55. 55 OpenBuddy-Falcon-40B-v9 80.7%
  56. 56 UltraLM 13B 80.6%
  57. 57 Gemini 1.0 Pro 79.7%
  58. 58 OpenChat8192-13B 79.5%
  59. 59 Evo 7B 79.2%
  60. 60 Claude2 Alpaca 13B 78.9%
  61. 61 Recycled WizardLM 7B V1.0 78.9%
  62. 62 OpenCoderPlus-15B 78.7%
  63. 63 MiniChat 1.5 3B 78.5%
  64. 64 OpenBudddy-LLaMA2-13B-v11.1 77.5%
  65. 65 Vicuna 7B v1.3 76.8%
  66. 66 WizardLM 13B 75.3%
  67. 67 JinaChat 74.1%
  68. 68 airoboros 65B 73.9%
  69. 69 airoboros 33B 73.3%
  70. 70 Guanaco 65B 71.8%
  71. 71 LLaMA2 Chat 7B 71.4%
  72. 72 Ghost 7B Alpha 70.4%
  73. 73 Vicuna 13B 70.4%
  74. 74 OpenBuddy-Falcon-7b-v6 70.4%
  75. 75 Phi-2 SFT 68.5%
  76. 76 Baize-v2 13B 67%
  77. 77 LLaMA 33B OASST RLHF 66.5%
  78. 78 Minotaur 13B 66%
  79. 79 Guanaco 33B 66%
  80. 80 Nous Hermes 13B 65.5%
  81. 81 Vicuna 7B 64.4%
  82. 82 Baize-v2 7B 63.9%
  83. 83 Alpaca-7B-NEFT 61.6%
  84. 84 LLaMA 33B OASST SFT 55%
  85. 85 Guanaco 13B 52.6%
  86. 86 davinci-003 50%
  87. 87 MiniChat 3B 48.8%
  88. 88 chatglm2-6b 47.1%
  89. 89 Guanaco 7B 46.6%
  90. 90 Falcon 40B Instruct 45.7%
  91. 91 Falcon 7B Instruct 45.7%
  92. 92 Alpaca Farm PPO Sim (GPT-4) 7B 44.1%
  93. 93 Pythia 12B SFT 41.9%
  94. 94 Alpaca Farm PPO Human 7B 41.2%
  95. 95 Phi-2 30.7%
  96. 96 Alpaca 7B 26.5%
  97. 97 Pythia 12B OASST SFT 26%
  98. 98 Baichuan-13B-Chat 21.8%
  99. 99 Davinci001 15.2%
#EntryScoreSettingDateSource
1 GPT-4 Turbo Nov 2023 97.7% Nov 2023 Official leaderboard
2 XwinLM 70b V0.3 97.6% Official leaderboard
3 Mistral Medium 96.8% Dec 2023 Official leaderboard
4 XwinLM 70b V0.1 95.6% Official leaderboard
5 PairRM 0.4B+Tulu 2+DPO 70B (best-of-16) 95.4% Official leaderboard
6 Tulu 2+DPO 70B 95% Official leaderboard
7 Mistral-7B+RAHF-DUAL+LoRA 94.9% Official leaderboard
8 GPT-4 Mar 2023 94.8% Mar 2023 Official leaderboard
9 Mixtral 8x7B 94.8% Dec 2023 Official leaderboard
10 Mistral-7B-ReMax-v0.1 94.4% Official leaderboard
11 Yi-34B 94.1% Nov 2023 Official leaderboard
12 GPT-4 Jun 2023 93.8% Jun 2023 Official leaderboard
13 GPT-3.5 Turbo Jun 2023 93.4% Jun 2023 Official leaderboard
14 PairRM 0.4B+Zephyr 7B Beta (best-of-16) 93.4% Official leaderboard
15 UltraLM 13B V2.0 (best-of-16) 92.8% Official leaderboard
16 Mistral 7B v0.2 92.8% Dec 2023 Official leaderboard
17 LLaMA2 Chat 70B 92.7% Official leaderboard
18 LMCocktail-10.7B-v1 92.2% Official leaderboard
19 XwinLM 13b V0.1 91.8% Official leaderboard
20 UltraLM 13B (best-of-16) 91.5% Official leaderboard
21 CUT 13B 91.4% Official leaderboard
22 Claude 2 91.4% Jul 2023 Official leaderboard
23 PairRM 0.4B+Tulu 2+DPO 13B (best-of-16) 91.1% Official leaderboard
24 Cohere Command 90.6% Official leaderboard
25 Zephyr 7B Beta 90.6% Oct 2023 Official leaderboard
26 DEITA 7B v1.0 90.1% Official leaderboard
27 OpenChat V3.1 13B 89.5% Official leaderboard
28 GPT-3.5 Turbo (Mar 2023) 89.4% Mar 2023 Official leaderboard
29 Evo v2 7B 89.3% Official leaderboard
30 WizardLM 13B V1.2 89.2% Official leaderboard
31 Vicuna 33B v1.3 89% Official leaderboard
32 CausalLM-14B 88.3% Official leaderboard
33 Tulu 2+DPO 13B 88.1% Official leaderboard
34 Humpback LLaMa2 70B 87.9% Official leaderboard
35 XwinLM 7b V0.1 87.8% Official leaderboard
36 OpenBudddy-LLaMA2-70B-v10.1 87.7% Official leaderboard
37 OpenChat V2-W 13B 87.1% Official leaderboard
38 Claude 2.1 87.1% Nov 2023 Official leaderboard
39 OpenBuddy-LLaMA-65B-v8 86.5% Official leaderboard
40 WizardLM 13B V1.1 86.3% Official leaderboard
41 UltraLM 13B V2.0 86.3% Official leaderboard
42 GPT-3.5 Turbo Nov 2023 86.3% Nov 2023 Official leaderboard
43 Zephyr 7B Alpha 85.8% Oct 2023 Official leaderboard
44 OpenChat V2 13B 85% Official leaderboard
45 Tulu 2+DPO 7B 84.2% Official leaderboard
46 Humpback LLaMa 65B 83.7% Official leaderboard
47 Recycled WizardLM 7B V2.0 83.5% Official leaderboard
48 Phi-2 DPO 82.3% Official leaderboard
49 Vicuna-13B-v1.3 82.1% Jun 2023 Official leaderboard
50 PlatoLM 7B 81.9% Official leaderboard
51 LLaMA2 Chat 7B Evol70k-NEFT 81.9% Official leaderboard
52 OpenBuddy-LLaMA-30B-v7.1 81.5% Official leaderboard
53 LLaMA2 Chat 13B 81.1% Official leaderboard
54 OpenChat-13B 80.9% Official leaderboard
55 OpenBuddy-Falcon-40B-v9 80.7% Official leaderboard
56 UltraLM 13B 80.6% Official leaderboard
57 Gemini 1.0 Pro 79.7% Dec 2023 Official leaderboard
58 OpenChat8192-13B 79.5% Official leaderboard
59 Evo 7B 79.2% Official leaderboard
60 Claude2 Alpaca 13B 78.9% Official leaderboard
61 Recycled WizardLM 7B V1.0 78.9% Official leaderboard
62 OpenCoderPlus-15B 78.7% Official leaderboard
63 MiniChat 1.5 3B 78.5% Official leaderboard
64 OpenBudddy-LLaMA2-13B-v11.1 77.5% Official leaderboard
65 Vicuna 7B v1.3 76.8% Official leaderboard
66 WizardLM 13B 75.3% Official leaderboard
67 JinaChat 74.1% Official leaderboard
68 airoboros 65B 73.9% Official leaderboard
69 airoboros 33B 73.3% Official leaderboard
70 Guanaco 65B 71.8% Official leaderboard
71 LLaMA2 Chat 7B 71.4% Official leaderboard
72 Ghost 7B Alpha 70.4% Official leaderboard
73 Vicuna 13B 70.4% Official leaderboard
74 OpenBuddy-Falcon-7b-v6 70.4% Official leaderboard
75 Phi-2 SFT 68.5% Official leaderboard
76 Baize-v2 13B 67% Official leaderboard
77 LLaMA 33B OASST RLHF 66.5% Official leaderboard
78 Minotaur 13B 66% Official leaderboard
79 Guanaco 33B 66% Official leaderboard
80 Nous Hermes 13B 65.5% Official leaderboard
81 Vicuna 7B 64.4% Official leaderboard
82 Baize-v2 7B 63.9% Official leaderboard
83 Alpaca-7B-NEFT 61.6% Official leaderboard
84 LLaMA 33B OASST SFT 55% Official leaderboard
85 Guanaco 13B 52.6% Official leaderboard
86 davinci-003 50% Nov 2022 Official leaderboard
87 MiniChat 3B 48.8% Official leaderboard
88 chatglm2-6b 47.1% Jun 2023 Official leaderboard
89 Guanaco 7B 46.6% Official leaderboard
90 Falcon 40B Instruct 45.7% Official leaderboard
91 Falcon 7B Instruct 45.7% Official leaderboard
92 Alpaca Farm PPO Sim (GPT-4) 7B 44.1% Official leaderboard
93 Pythia 12B SFT 41.9% Official leaderboard
94 Alpaca Farm PPO Human 7B 41.2% Official leaderboard
95 Phi-2 30.7% Dec 2023 Official leaderboard
96 Alpaca 7B 26.5% Official leaderboard
97 Pythia 12B OASST SFT 26% Official leaderboard
98 Baichuan-13B-Chat 21.8% Official leaderboard
99 Davinci001 15.2% Official leaderboard

2 models have held number one since its release in May 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard Passed 85% in May 2023, by GPT-4
GPT-4 Turbo 97.7%

The best score passed 85% in Mar 2023

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2023 2024 Near the ceiling (85%) GPT-4 Turbo GPT-4 GPT-3.5 Turbo (Mar 2023) davinci-003 0% 20% 40% 60% 80% 100% 2023 2024 Near the ceiling (85%) GPT-4 Turbo GPT-3.5 Turbo (Mar 2023)

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

OpenAI DeepSeek
AlpacaEval 2.0 Passed 85% in Jan 2025, by DeepSeek-R1
DeepSeek-R1 87.6%
AlpacaEval Passed 85% in May 2023, by GPT-4
GPT-4 Turbo 97.7%
AlpacaEval 2.0 Passed 85% in Jan 2025, by DeepSeek-R1
DeepSeek-R1 87.6%
Replaced by AlpacaEval 2.0

More in Instructions and chat

All 76 in Instructions and chat
  • IFEval In use Models answer about 500 prompts containing verifiable instructions such as word limits or required keywords. Code checks each instruction for strict and loose accuracy. Gemma 4 31B IT 98.9%
  • IFBench In use 58 new out-of-domain verifiable output constraints. Scored by programmatic constraint checks (strict or loose). Nemotron-Cascade 2 30B-A3B 82.9%
  • MultiChallenge In use Multi-turn conversations testing instruction retention, inference memory, versioned editing and self-coherence. Graded by LLM judge with rubrics. GPT-6.1 Sol 82%
  • Multi-IF In use Models follow verifiable instructions over three-turn conversations in eight languages, extending IFEval. Scored by instruction accuracy at each turn. LFM3-1.7B 89.1%
  • AdvancedIF In use Models respond to over 1,600 prompts with complex, multi-turn, and system-level instructions. A judge model checks each response against expert-written rubrics. MAI-Thinking-1 85%
  • IFStruct In use Models generate JSON or YAML that must follow a requested schema and phrasing constraints, without constrained decoding. A response passes only if every structural constraint holds. LFM3-1.7B 93.7%