MedQA
Models answer multiple-choice questions from medical licensing exams, usually the 12,723-question English USMLE set. Scoring is accuracy.
The full dataset also has simplified and traditional Chinese exam sets.
In the authors’ words
Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a machine comprehension model. Through experiments, we find that even the current best method can only achieve 36.7%, 42.0%, and 70.1% of test accuracy on the English, traditional Chinese, and simplified Chinese questions, respectively. We expect MedQA to present great challenges to existing OpenQA systems and hope that it can serve as a platform to promote much stronger OpenQA models from the NLP community in the future.
- Level
- Expert Specialists with years of training, often over hours of work
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Multiple choice
GPT-4o leads with 89.4% among models released by May 2024
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
The newest model with a score here came out in May 2024, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6 Astra
- Claude Opus 5.5
- GPT-6.1 Terra
- GPT-6.1 Sol
- and 16 more
- 1 GPT-4o May 2024 89.4%
- 2
phi-3-medium 14B 69.9% - 3 phi-3-small 7.4B 65.4%
- 4 phi-3-mini 3.8B 53.8%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GPT-4o May 2024 | 89.4% | Aug 2024 | Model card | |
| 2 | phi-3-medium 14B | 69.9% | Aug 2024 | Model card | |
| 3 | phi-3-small 7.4B | 65.4% | Aug 2024 | Model card | |
| 4 | phi-3-mini 3.8B | 53.8% | Aug 2024 | Model card |
2 models have held number one since 2023
Lab claims count from the day the card came out, independent runs from the model’s release.
The best score passed 85% in Aug 2024
All the scores we have, one dot each, so a model tested at several settings appears more than once.
No launch post has listed it since Apr 2024
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 22 Apr 2024 | phi-3-medium 14B | 69.4 | MedQA (2-Shot) |
4 system cards report it, from 2 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 30 Aug 2024 | phi-3-small 7.4B technical report | 65.4 | MedQA (2-Shot) (Section 3 table) | Capabilities |
| 30 Aug 2024 | phi-3-mini 3.8B technical report | 53.8 | MedQA (2-Shot) (Section 3 table) | Capabilities |
| 30 Aug 2024 | phi-3-medium 14B technical report | 69.9 | MedQA (2-Shot) (Section 3 table) | Capabilities |
| 8 Aug 2024 | GPT-4o system card | 89.4 | MedQA USMLE 4 Options (0-shot), 89.4% in text (Table 7 shows 0.89, 5-shot 0.89) | Health |
More in Law, finance and medicine
- HealthBench Professional In use 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. Claude Sonnet 5.5 69.2%
- MedXpertQA-MM In use Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy.
Qwen 3.8 Max 80.4% - HealthBench In use 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. GPT-5 67.2%
- KBL In use Models answer Korean legal knowledge and reasoning tasks and Korean bar exam questions. It has 7 knowledge tasks, 4 reasoning tasks and 2,510 bar exam items.
Solar Pro 4 77.5% - KorMedMCQA In use Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. Solar Pro 4 93.2%
- PRBench In use Model answers 1,100 expert-written law and finance questions. LLM judge grades against weighted expert rubrics of 10-30 criteria each.
Muse Spark 1.3 61.6%