Benchmark Almanac

ProtocolQA Open-Ended

GPT-6 Sol 44.4% leads since Sep 2026 Only in system cards

Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1.

OpenAI open-ended adaptation of the ProtocolQA multiple-choice set.

Made by OpenAI Released Dec 2024 Lab protocols Tasks kept private Last in a system card Oct 2026 Last model added Oct 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Graded by a model A language model scores each response, usually against a rubric.
Answers
Short answer A number, word or short string matched to a key.
25
Models scored in lab cards
35
System cards report it, from 1 lab

GPT-6 Sol leads with 44.4%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

25 models. Scroll sideways for the rest.

  1. 44.4 GPT-6 Sol
  2. 44.1 GPT-5.6 Sol
  3. 44.0 GPT-5.3 Codex
  4. 42.5 GPT-5.4
  5. 41.4 GPT-6 Astra
  6. 40.7 GPT-6.1 Sol
  7. 40.0 GPT-5.2
  8. 39.0 GPT-5.2 Codex
  9. 37.3 GPT-5.5
  10. 37.0 GPT-5 mini
  11. 36.7 GPT-5.5 Pro
  12. 36.7 GPT-5.5 Instant
  13. 35.0 GPT-5.1-Codex-Max
  14. 34.9 GPT-6 Luna
  15. 33.6 GPT-5.4 Mini
  16. 30.0 o3
  17. 30.0 GPT-5
  18. 29.9 GPT-5.6 Luna
  19. 28.0 OpenAI deep research
  20. 27.0 o4-mini
  21. 27.0 gpt-oss-120b
  22. 24.0 o1
  23. 23.0 o3-mini
  24. 18.0 GPT-4.5
  25. 16.0 gpt-oss-20b
  1. 1 GPT-6 Sol 44.4%
  2. 2 GPT-5.6 Sol 44.1%
  3. 3 GPT-5.3 Codex 44%
  4. 4 GPT-5.4 42.5%
  5. 5 GPT-6 Astra 41.4%
  6. 6 GPT-6.1 Sol 40.7%
  7. 7 GPT-5.2 40%
  8. 8 GPT-5.2 Codex 39%
  9. 9 GPT-5.5 37.3%
  10. 10 GPT-5 mini 37%
  11. 11 GPT-5.5 Pro 36.7%
  12. 12 GPT-5.5 Instant 36.7%
  13. 13 GPT-5.1-Codex-Max 35%
  14. 14 GPT-6 Luna 34.9%
  15. 15 GPT-5.4 Mini 33.6%
  16. 16 o3 30%
  17. 17 GPT-5 30%
  18. 18 GPT-5.6 Luna 29.9%
  19. 19 OpenAI deep research 28%
  20. 20 o4-mini 27%
  21. 21 gpt-oss-120b 27%
  22. 22 o1 24%
  23. 23 o3-mini 23%
  24. 24 GPT-4.5 18%
  25. 25 gpt-oss-20b 16%
#ModelScoreSettingDateSource
1 GPT-6 Sol 44.4% Sep 2026 Model card
2 GPT-5.6 Sol 44.1% Aug 2026 Model card
3 GPT-5.3 Codex 44% Feb 2026 Model card
4 GPT-5.4 42.5% Mar 2026 Model card
5 GPT-6 Astra 41.4% Sep 2026 Model card
6 GPT-6.1 Sol 40.7% Sep 2026 Model card
7 GPT-5.2 40% Dec 2025 Model card
8 GPT-5.2 Codex 39% Dec 2025 Model card
9 GPT-5.5 37.3% Apr 2026 Model card
10 GPT-5 mini 37% Aug 2025 Model card
11 GPT-5.5 Pro 36.7% Apr 2026 Model card
12 GPT-5.5 Instant 36.7% May 2026 Model card
13 GPT-5.1-Codex-Max 35% Nov 2025 Model card
14 GPT-6 Luna 34.9% Sep 2026 Model card
15 GPT-5.4 Mini 33.6% Mar 2026 Model card
16 o3 30% Apr 2025 Model card
17 GPT-5 30% Aug 2025 Model card
18 GPT-5.6 Luna 29.9% Aug 2026 Model card
19 OpenAI deep research 28% Feb 2025 Model card
20 o4-mini 27% Apr 2025 Model card
21 gpt-oss-120b 27% Aug 2025 Model card
22 o1 24% Dec 2024 Model card
23 o3-mini 23% Jan 2025 Model card
24 GPT-4.5 18% Feb 2025 Model card
25 gpt-oss-20b 16% Aug 2025 Model card

8 models have held number one since its release in Dec 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Lab claims 8 leaders
GPT-6 Sol 44.4%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) GPT-6 Sol GPT-5.3 Codex GPT-5 mini o3 o1 0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) GPT-6 Sol GPT-5 mini

Labs report it only in system cards

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark. Some system cards do, below.

35 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
29 Oct 2026 GPT-6 Luna system card addendum 31.8 ProtocolQA Open-Ended, pass@1 (Table 8) Bio and chem risk
7 Oct 2026 GPT-6 Sol system card addendum 25.6 ProtocolQA Open-Ended, pass@1 (Table 8) Bio and chem risk
7 Oct 2026 GPT-6 Luna system card addendum 34.6 ProtocolQA Open-Ended, pass@1 (Table 8) Bio and chem risk
29 Sep 2026 GPT-6.1 Sol system card addendum 40.7 ProtocolQA Open-Ended, refusal-adjusted pass@1 (9.1.1.1) Bio and chem risk
22 Sep 2026 GPT-6 Sol system card 44.4 ProtocolQA Open-Ended, pass@1 (Table 33, A.8.1.1.1) Bio and chem risk
22 Sep 2026 GPT-6 Luna system card 34.9 ProtocolQA Open-Ended, pass@1 (Table 33, A.8.1.1.1) Bio and chem risk
3 Sep 2026 GPT-6 Astra system card 41.4 ProtocolQA Open-Ended, pass@1 (refusal-adjusted 45.37%) (10.1.1.1.2) Bio and chem risk
6 Aug 2026 GPT-5.6 Sol system card addendum 44.1 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 4, 7.1.1.2) Bio and chem risk
6 Aug 2026 GPT-5.6 Luna system card addendum 29.9 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 4, 7.1.1.2) Bio and chem risk
9 Jul 2026 GPT-5.6 Terra system card ProtocolQA Open-Ended, pass@1 by cost, chart without printed numbers (Figures 24, 25, 9.1.1.2) Bio and chem risk
9 Jul 2026 GPT-5.6 Sol system card 43.5 ProtocolQA Open-Ended, pass@1, top score stated in text (9.1.1.2) Bio and chem risk
9 Jul 2026 GPT-5.6 Luna system card ProtocolQA Open-Ended, pass@1 by cost, chart without printed numbers (Figures 24, 25, 9.1.1.2) Bio and chem risk
25 Jun 2026 GPT-5.6 Terra system card ProtocolQA Open-Ended, pass@1 by cost, chart without printed numbers (Figures 24, 25, 9.1.1.2) Bio and chem risk
25 Jun 2026 GPT-5.6 Sol system card 43.5 ProtocolQA Open-Ended, pass@1, top score stated in text (9.1.1.2) Bio and chem risk
25 Jun 2026 GPT-5.6 Luna system card ProtocolQA Open-Ended, pass@1 by cost, chart without printed numbers (Figures 24, 25, 9.1.1.2) Bio and chem risk
4 May 2026 GPT-5.5 Instant system card 36.7 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 4, 8.1.1.2) Bio and chem risk
23 Apr 2026 GPT-5.5 system card 37.3 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 13, 9.1.1.2) Bio and chem risk
23 Apr 2026 GPT-5.5 Pro system card 36.7 ProtocolQA Open-Ended, pass@1, GPT-5.5 Pro, printed chart label (Figure 13, 9.1.1.2) Bio and chem risk
17 Mar 2026 GPT-5.4 Mini system card 33.6 ProtocolQA Open-Ended, pass@1 (34.24% counting refusals) (Table 18) Bio and chem risk
5 Mar 2026 GPT-5.4 system card 42.5 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 9, 5.1.1.2) Bio and chem risk
5 Feb 2026 GPT-5.3 Codex system card 44 ProtocolQA Open-Ended, pass@1, no browsing, printed chart label (Figure 2, 5.1.1.2) Bio and chem risk
18 Dec 2025 GPT-5.2 Codex system card addendum 39 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 2, 5.1.1.2) Bio and chem risk
11 Dec 2025 GPT-5.2 system card addendum 40 ProtocolQA Open-Ended, pass@1, gpt-5.2 browsing, printed chart label (Figure 6) Bio and chem risk
18 Nov 2025 GPT-5.1-Codex-Max system card 35 ProtocolQA Open-Ended, pass@1, printed chart label (39% with refusals counted) (Figure 3, 5.1.1.3) Bio and chem risk
7 Aug 2025 GPT-5 system card 30 ProtocolQA Open-Ended, pass@1, browsing launch candidate (Figure 11) Bio and chem risk
7 Aug 2025 GPT-5 mini system card 37 ProtocolQA Open-Ended, pass@1, gpt-5-thinking-mini browsing launch candidate (Figure 11) Bio and chem risk
5 Aug 2025 gpt-oss-20b model card 16 ProtocolQA Open-Ended, pass@1, gpt-oss 20b browsing launch candidate (Figure 7) Bio and chem risk
5 Aug 2025 gpt-oss-120b model card 27 ProtocolQA Open-Ended, pass@1, gpt-oss 120b browsing launch candidate (Figure 7) Bio and chem risk
17 Jul 2025 ChatGPT agent system card ProtocolQA Open-Ended, pass@1, chart shows only ChatGPT Agent browsing helpful-only at 40%, no released-model value (Figure 3, 5.1.1.3) Bio and chem risk
16 Apr 2025 o4-mini system card 27 ProtocolQA Open-Ended, pass@1, o4-mini no browsing launch candidate, printed chart label (Figure 5) Bio and chem risk
16 Apr 2025 o3 system card 30 ProtocolQA Open-Ended, pass@1, o3 no browsing launch candidate, printed chart label (Figure 5) Bio and chem risk
27 Feb 2025 GPT-4.5 system card 18 ProtocolQA Open-Ended, pre- and post-mitigation (4.3.4) Bio and chem risk
25 Feb 2025 OpenAI deep research system card 28 ProtocolQA Open-Ended, pass@1, post-mitigation with browsing, text and printed chart label (without browsing 27%) (3.4.4) Bio and chem risk
31 Jan 2025 o3-mini system card 23 ProtocolQA Open-Ended, post-mitigation (5.4.7) Bio and chem risk
5 Dec 2024 o1 system card 24 ProtocolQA Open-Ended, pass@1, o1 (post-mitigation), printed chart label (5.5.6) Bio and chem risk

More in Biology and chemistry

All 96 in Biology and chemistry
  • SUPERChem In use Models solve 500 expert-written chemistry reasoning problems, offered in multimodal and text-only forms. Expert solution paths also allow a reasoning path score. Tencent Hy4 preview 66.4%
  • GeneBench Pro In use Agents analyze messy simulated genomics and biology datasets to estimate a decision-relevant quantity across 129 problems. Scored automatically against known answers. GPT-6 Astra 37.1%
  • LifeSciBench In use Models complete 750 expert-written life science research tasks, often with attached figures, tables, or sequence files. Graded against expert rubrics as score and pass rate. GPT-6 Astra 60.3%
  • MolecularIQ In use Models answer questions about molecular graphs, such as counting, indexing and constrained generation, and a symbolic checker verifies each answer. Intern-S2-397B 62.4%
  • TOMG-Bench In use Models generate molecules from natural language requests in three tasks: molecule editing, molecule optimization and customized generation. Each output is checked for validity and for meeting the requested properties. Intern-S2-397B 66.8%
  • BABE Fading Models reason from experimental results in peer-reviewed biology papers to draw conclusions, with causal and cross-scale inference. Seed2.0 Lite (0428) 57.9%