Knowledge
We track 109 benchmarks in knowledge, or 114 if you count each version. They run from everyday level to research frontier level.
OpenAI held the top knowledge score 23% of the contested time
That's about 11 of the 46 months since January 2023, averaged over the benchmarks below. The easiest benchmarks are at the top. Versions of one test share a row, with a dark tick at each new version. We stop the clock, and fade the rest of the row, when the best score passes 85%, or at the newest score if 45 days go by without another.
Google DeepMind OpenAI Anthropic Shanghai AI Lab Microsoft Other labs Meta Baidu DeepSeek Tencent Moonshot AI Meituan ByteDance Seed Qwen xAI
Everyday Most adults, without training
School Students, from grade school to high school
University Undergraduates and entry-level professionals
Expert Specialists with years of training, often over hours of work
Research frontier Researchers, on open problems or tasks that take days
Share of contested time at number one
OpenAI 23% 
Qwen 21% 
Google DeepMind 13% Anthropic 12% 
Meta 8% Other labs 7% 
Microsoft 4% xAI 4% 
ByteDance Seed 3% 
DeepSeek 2% Moonshot AI 1%
Qwen 21% Google DeepMind 13% Anthropic 12% Meta 8% Other labs 7% Microsoft 4% xAI 4% ByteDance Seed 3% DeepSeek 2% Moonshot AI 1% Closed out: 8 benchmarks passed 85%
OpenAI 2 Other labs 2 Google DeepMind 1 DeepSeek 1 Anthropic 1 Meta 1
Google DeepMind 1 DeepSeek 1 Anthropic 1 Meta 1 - Nov 2025 Extended NYT Connections Gemini 3 Pro
- Dec 2024 C-Eval DeepSeek-V3
- Dec 2024 MMLU-Pro o1
- Mar 2024 BIG-Bench Hard Claude 3 Opus
- May 2023 PIQA PaLM 2-L
- May 2023 DROP PaLM 2-L
- Mar 2023 MMLU GPT-4 Mar 2023
- Feb 2023 TriviaQA LLaMA-65B
Scores come from each benchmark’s own leaderboard, from Epoch AI’s runs and from labs’ reports on their own models. We date Epoch runs by the model’s release and lab reports by when they came out. Labs pick their own settings, so a lab’s claim can beat an independent run of the same model. Turn off lab claims to see independent runs only.
Google DeepMind OpenAI Microsoft Inflection AI Technology Innovation Institute Anthropic Meta Stability AI DeepSeek Mistral AI Meituan
Everyday Most adults, without training
School Students, from grade school to high school
University Undergraduates and entry-level professionals
Expert Specialists with years of training, often over hours of work
Research frontier Researchers, on open problems or tasks that take days
Share of contested time at number one, independent runs only
Google DeepMind 30% OpenAI 30% 
Technology Innovation Institute 11% Meta 10% Microsoft 6% Anthropic 6% DeepSeek 4% Inflection AI 3% 
Mistral AI 1% Only each benchmark’s own leaderboard and Epoch AI’s runs, dated by the model’s release. We leave out benchmarks that only labs report.
From everyday to research frontier level
Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.
Everyday
Most adults, without training
- HellaSwag
- WinoGrande
- BoolQ
- PIQA
- TriviaQA
- WideSearch
- SIQA
- LAMBADA
- Natural Questions
- CommonSenseQA
- SkateBench v2
- ANLI
- FRAMES
- PopQA
- SQuAD v2
- SuperGLUE
- CommonsenseQA 2.0
- SQuAD
- CB
- WorldSense
- WSC
- RTE
- StrategyQA
- ChineseQA
- ComplexNLU (Tencent)
- COPA
- QuAC
- ReCoRD
- StoryCloze
- WebQuestions
- WiC
- Winograd
- Word-Task (Tencent)
- Bamboogle
- DeepSeek search Q&A evaluation
- GLM-4 All Tools information seeking (Z.ai)
- HotpotQA
- Knowledge Q&A (MiniMax in-house)
- Temporal reasoning with search
- Adversarial SQuAD
- BLiMP
- CoQA
- Dynabench SQuAD
- Hindsight Neglect
- Paloma
- PAWS
School
Students, from grade school to high school
- BIG-Bench Hard
- ARC-Challenge
- Extended NYT Connections
- DROP
- OpenBookQA
- AGIEval
- ARC-Easy
- RACE-H
- RACE-M
- RACE
- AP Calculus
- AP Physics
- GAOKAO-Bench
- SAT Reading
- MultiRC
- AP Art History
- AP Biology
- AP Calculus BC
- AP Chemistry
- AP English Language and Composition
- AP English Literature and Composition
- AP Environmental Science
- AP Macroeconomics
- AP Microeconomics
- AP Physics 2
- AP Psychology
- AP Statistics
- AP US Government
- AP US History
- AP World History
- SAT Evidence-Based Reading & Writing
- SnorkelSpatial
University
Undergraduates and entry-level professionals
Expert
Specialists with years of training, often over hours of work
By kind
Exam questions
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| MMLU-Pro Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. | University | Checked automatically |  89.8% | In use |
| MMLU 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. | University | Checked automatically | 92.3% | Fading |
| ARC-Challenge Models answer grade-school science multiple-choice questions that retrieval and word co-occurrence baselines got wrong. Scored by accuracy. | School | Checked automatically | 96.9% | Dropped |
| C-Eval 13,948 Chinese multiple-choice exam questions across 52 disciplines and four difficulty levels. Accuracy. | University | Checked automatically | 93.7% | Fading |
| MMLU-Redux Manually re-annotated subset of MMLU that fixes wrong labels. Multiple-choice accuracy. | University | Checked automatically | 95% | Fading |
| OpenBookQA Models answer elementary science multiple-choice questions that need a given core science fact plus common knowledge. Scored by accuracy. | School | Checked automatically | 96.6% | Fading |
| AGIEval Questions from human standardized exams (gaokao, SAT, LSAT, bar). Multiple-choice and cloze accuracy. | School | Checked automatically |  89.2% | Fading |
| MMLU-CF Models answer new MMLU-style multiple-choice questions written to avoid training data contamination. A 10,000 question test set stays private. | University | Checked automatically | 73.4% | Not in cards |
| ARC-Easy Models answer grade-school science multiple-choice questions from the ARC Easy Set, which has 5,197 questions. Scoring is accuracy. | School | Checked automatically | 98.9% | Dropped |
| CMMLU Chinese multiple-choice questions across 67 subjects including China-specific topics. Accuracy. | University | Checked automatically |  90.8% | Fading |
| AP Calculus Models answer an official AP Calculus practice exam without image questions. Scored as accuracy. | School | Checked automatically | 88.6% | System cards only |
| AP Physics Models answer an official AP Physics practice exam without image questions. Scored as accuracy. | School | Checked automatically | 92.9% | System cards only |
| GAOKAO-Bench Models answer Chinese national college entrance exam questions from 2010 to 2022 across nine subjects, with objective and subjective questions. | School | Checked automatically | 94.5% | Dropped |
| GRE Analytical Writing Models write GRE issue and argument essays graded by trained contractors. Scored on the 0 to 6 scale. | University | Graded by people | 5.00 | System cards only |
| MMLU-stem Models answer the STEM subjects of MMLU, such as physics, chemistry, mathematics, and computer science, as multiple choice. Scoring is accuracy. | University | Checked automatically | 83.6% | Dropped |
| KMMLU-Redux Models answer multiple-choice questions from Korean National Technical Qualification exams. The set is rebuilt from KMMLU with erroneous questions removed. | University | Checked automatically |  72.7% | Dropped |
| AP Art History Models take a simulated AP Art History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Biology Models take a simulated AP Biology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Calculus BC Models take a simulated AP Calculus BC exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 4.00 | System cards only |
| AP Chemistry Models take a simulated AP Chemistry exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 4.00 | System cards only |
| AP English Language and Composition Models take a simulated AP English Language and Composition exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 2.00 | System cards only |
| AP English Literature and Composition Models take a simulated AP English Literature and Composition exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 2.00 | System cards only |
| AP Environmental Science Models take a simulated AP Environmental Science exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Macroeconomics Models take a simulated AP Macroeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Microeconomics Models take a simulated AP Microeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Physics 2 Models take a simulated AP Physics 2 exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 4.00 | System cards only |
| AP Psychology Models take a simulated AP Psychology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Statistics Models take a simulated AP Statistics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP US Government Models take a simulated AP US Government exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP US History Models take a simulated AP US History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP World History Models take a simulated AP World History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 4.00 | System cards only |
| MMLU College Biology Models answer MMLU multiple-choice questions from the College Biology subject. Scoring is accuracy. | University | Checked automatically | 95% | System cards only |
| MMLU-Redux 2.0 Models answer 5,700 MMLU questions from all 57 subjects that experts re-annotated for errors. Flawed questions are marked so scoring can skip or fix them. | University | Checked automatically | Not in cards |
Expert-written questions
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| Humanity's Last Exam 2,500 expert-written closed-ended questions across 100+ subjects. Exact-match or multiple-choice answers graded by an LLM judge. | Expert | Checked automatically | 54.8% | In use |
| SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. | Expert | Checked automatically | 73.9% | In use |
| HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. | Research frontier | Checked automatically | 65.8% | In use |
| KMMLU-Pro Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. | Expert | Checked automatically |  79.2% | In use |
| HLE-Verified Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. | Expert | Checked automatically | 73.6% | In use |
| LPFQA Models answer 430 long-tail professional questions drawn from real forum discussions in 7 academic and industrial domains. | Expert | Checked automatically | 52.6% | Fading |
| Encyclo-K Models judge questions composed at test time from knowledge statements extracted from authoritative textbooks. Random composition is meant to resist contamination and test several facts at once. | University | Checked automatically | 65.7% | Fading |
| KINA Models answer 899 knowledge questions across 261 fine-grained disciplines, each with ten lettered options. Scored by accuracy against the key. | Expert | Checked automatically | 48.3% | In use |
| HLE Extended Models answer closed-ended frontier academic questions. Mercor adds 447 new expert-written tasks to the 2158 public ones and reports both scores. | Expert | Checked automatically | Not in cards | |
| HLE-Rolling A dynamic fork of Humanity's Last Exam where the makers remove, fix and re-add questions over time and publish an update log. | Research frontier | Checked automatically | Not in cards |
Commonsense
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| HellaSwag Commonsense sentence completion with adversarially filtered wrong endings. Multiple-choice accuracy. | Everyday | Checked automatically | 97% | Fading |
| WinoGrande 44K Winograd-style pronoun-resolution problems. Binary-choice accuracy. | Everyday | Checked automatically | 91.7% | Fading |
| PIQA Models pick which of two solutions better achieves an everyday physical goal. Scoring is accuracy. | Everyday | Checked automatically | 88.6% | Dropped |
| SIQA Models answer multiple-choice questions about people's motives, reactions, and emotions in everyday social situations. Scored by accuracy. | Everyday | Checked automatically | 80.2% | Dropped |
| CommonSenseQA Models answer five-way multiple-choice questions built from ConceptNet relations that need everyday background knowledge. Scored by accuracy. | Everyday | Checked automatically | 90.4% | Dropped |
| CommonsenseQA 2.0 Models answer 14,343 yes or no commonsense questions that players wrote to fool an AI in a game. Scored by accuracy. | Everyday | Checked automatically | 67.8% | Not in cards |
| WSC Models decide whether a pronoun refers to a given noun phrase in sentences that need commonsense to resolve. Scored by accuracy. | Everyday | Checked automatically | 86.9% | Dropped |
| COPA Models choose which of two alternatives is the more plausible cause or effect of a premise, across 1,000 questions. Scored by accuracy. | Everyday | Checked automatically | 96% | Dropped |
| StoryCloze Models pick the correct ending for a four-sentence everyday story from two candidate endings. Scored by accuracy. | Everyday | Checked automatically | 87.4% | Dropped |
| Winograd Models pick which noun an ambiguous pronoun refers to in Winograd schema sentence pairs that need world knowledge. Scored by accuracy. | Everyday | Checked automatically | 89.5% | Dropped |
| BLiMP Language models must assign higher probability to the grammatical sentence in each of 67,000 minimal pairs covering English syntax, morphology and semantics. | Everyday | Checked automatically | Not in cards | |
| Paloma Language models are measured by perplexity on held-out text from 585 domains, from news sites to individual subreddits. | Everyday | Checked automatically | Not in cards |
Reading comprehension
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| BoolQ Models read a Wikipedia passage and answer a naturally occurring yes or no question about it. Scored by accuracy. | Everyday | Checked automatically | 90.9% | Dropped |
| DROP Reading comprehension requiring discrete operations (counting, sorting, arithmetic) over paragraphs. Scored by F1. | School | Checked automatically | 93.5% | Fading |
| LAMBADA Models predict the last word of narrative passages where humans can guess it only from the whole passage. Scored by accuracy. | Everyday | Checked automatically | 87.2% | Dropped |
| RACE-H Models answer multiple-choice reading questions from English exams for Chinese high school students. The test split has 3,498 questions, scored by accuracy. | School | Checked automatically | 92.9% | Dropped |
| SQuAD v2 Models extract answers from Wikipedia paragraphs and must abstain on over 50,000 adversarial unanswerable questions. Scored by exact match and F1. | Everyday | Checked automatically | 81.8% | Dropped |
| GRE Verbal Models take the GRE verbal reasoning section. Scored on the 130 to 170 scale. | University | Checked automatically | 169.0 | System cards only |
| SQuAD Models extract answer spans from Wikipedia paragraphs for over 100,000 crowdsourced questions. Scored by exact match and F1. | Everyday | Checked automatically | 89.3% | Dropped |
| RACE-M Models read passages from English exams for Chinese middle school students and answer multiple-choice questions. Scored by accuracy. | School | Checked automatically | 77% | Dropped |
| RACE Models answer reading comprehension questions from Chinese middle and high school English exams. Reported here without a subset split. | School | Checked automatically | 94.2% | System cards only |
| GMAT Verbal Models answer the verbal section of the official GMAT online exam. Scored as accuracy. | University | Checked automatically | 86.6% | System cards only |
| SAT Reading Models answer the reading section of official SAT practice tests. Scored as accuracy. | School | Checked automatically | 74.8% | System cards only |
| ComplexNLU (Tencent) Models answer Tencent in-house natural language understanding tasks with complex inputs. Task and scoring details are not published. | Everyday | Checked automatically | 61.2% | Dropped |
| MultiRC Models pick all correct answers to questions that need reasoning over several sentences of a paragraph. Scored by F1 over answer options. | School | Checked automatically | 88.2% | Dropped |
| QuAC Models answer free-form questions in information-seeking dialogs about a hidden Wikipedia section, across 100K questions. Scored by word-level F1. | Everyday | Checked automatically | 51.1% | Dropped |
| ReCoRD Models fill in a masked entity in a query about a news article, using the article as context. Scoring is exact match and token F1. | Everyday | Checked automatically | 93.8% | Dropped |
| Word-Task (Tencent) Models solve Tencent in-house word level language tasks. Task and scoring details are not published. | Everyday | Checked automatically | 62.9% | Dropped |
| HotpotQA Models answer 113k Wikipedia questions that need facts from several documents. The distractor setting supplies gold and distractor paragraphs. Scored by exact match or F1. | Everyday | Checked automatically |  92.1% | System cards only |
| SAT Evidence-Based Reading & Writing Models take the SAT reading and writing section. Scored on the 200 to 800 scale. | School | Checked automatically | 710.0 | System cards only |
| Adversarial SQuAD Models answer SQuAD questions whose passages contain an added distracting sentence. | Everyday | Checked automatically | System cards only | |
| CoQA Models answer a series of linked conversational questions about a passage, where later questions depend on earlier turns. Scored by word-level F1. | Everyday | Checked automatically | Not in cards | |
| Dynabench SQuAD Models answer reading comprehension questions written by people to fool earlier models in the loop. | Everyday | Checked automatically | System cards only |
Trivia and open questions
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| TriviaQA Trivia questions with evidence documents. Scored by exact match. | Everyday | Checked automatically | 91.8% | Fading |
| Natural Questions Models answer real Google search questions. Model cards usually use the open-domain version, scored by exact match against short answers from Wikipedia. | Everyday | Checked automatically | 53.4% | Dropped |
| SkateBench v2 Models name skateboard tricks from technical trick definitions. The score is the share named correctly. | Everyday | Checked automatically | 96.9% | Not in cards |
| PopQA Models answer about 14k entity questions built from Wikidata, spread from popular to rare entities. Scored by whether the answer matches a gold alias. | Everyday | Checked automatically | 55.5% | Fading |
| ChineseQA Models answer Chinese trivia questions in a closed-book format modeled on TriviaQA. | Everyday | Checked automatically | 85.1% | Dropped |
| WebQuestions Models answer factual questions mined from Google Suggest, with answers taken from the Freebase knowledge base. Scored by exact match. | Everyday | Checked automatically | 28.2% | Dropped |
| Knowledge Q&A (MiniMax in-house) Models answer knowledge questions drawn from real Hailuo AI user queries. | Everyday | Graded by a model |  78.6% | System cards only |
Questions that need search
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| WideSearch 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. | Everyday | Checked automatically | 83.9% | In use |
| FRAMES Models answer 824 multi-hop questions that need facts from 2 to 15 Wikipedia articles. Answers are graded against gold answers for accuracy. | Everyday | Checked automatically | 87.6% | Fading |
| Seal-0 Models answer 111 fact-seeking questions where web search returns conflicting or unhelpful results. Answers are graded for accuracy, usually with search enabled. | University | Checked automatically | 57.4% | Fading |
| Bamboogle Models answer 125 two-hop questions written so that a search engine cannot answer them directly. Models usually search and compose the facts. | Everyday | Checked automatically | 76% | System cards only |
| DeepSeek search Q&A evaluation Models answer objective and subjective Chinese search questions with retrieval or agentic search. Pairwise judging gives win, loss and tie rates. | Everyday | Ranked by matches | 28.1% | System cards only |
| GLM-4 All Tools information seeking (Z.ai) Models use a web browser to find information for user questions. The ChatGLM report gives accuracy but does not describe the test set. | Everyday | Graded by a model | 78.1% | System cards only |
| Multi-hop search factuality Models with search answer multi-hop and tail-fact questions that need several sources. Reports accuracy and unfactual rate. | Expert | Graded by a model | 94.3% | System cards only |
| Temporal reasoning with search Models with search answer real queries that test date arithmetic, stale facts and time-sensitive context. Reports error rate. | Everyday | Graded by a model | 11.8% | System cards only |
| BRIGHT Retrieval systems find documents for queries from StackExchange, LeetCode and math competitions where relevance needs reasoning. Scored by nDCG@10. | Expert | Checked automatically | Not in cards | |
| OBLIQ-Bench Retrievers search large real corpora for documents that match a latent pattern, such as an implicit stance. The suite has five such oblique search problems. | Expert | Checked automatically | Not in cards |
Language reasoning
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| BIG-Bench Hard Models solve 23 BIG-Bench tasks where earlier models trailed average human raters, such as logic puzzles and date math. Scoring is exact match. | School | Checked automatically |  94.3% | Fading |
| Extended NYT Connections Models solve New York Times Connections puzzles with up to four extra trick words added. Each puzzle scores by the square of the share of groups found. | School | Checked automatically | 98.6% | Not in cards |
| BIG-Bench Extra Hard Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. | University | Checked automatically | 74.4% | In use |
| ANLI Models label premise and hypothesis pairs as entailment, neutral, or contradiction. People wrote the pairs over three rounds to fool strong models. | Everyday | Checked automatically | 58.1% | Dropped |
| ToRR Models answer questions over tables given in several formats, and the board checks how much accuracy changes when the table format or prompt changes. | University | Checked automatically | 49.9 | Not in cards |
| MuSR Models answer questions about long generated narratives such as murder mysteries, object placement and team allocation. Each item needs several soft reasoning steps over the story. | University | Checked automatically | 86.7% | Fading |
| SuperGLUE Models solve eight English language understanding tasks, including BoolQ, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Scored by the average task score. | Everyday | Checked automatically | 88.9% | Not in cards |
| CB Models label whether a text commits its speaker to an embedded clause as entailment, contradiction, or neutral. Scored by accuracy and F1. | Everyday | Checked automatically | 87.5% | Dropped |
| WorldSense Models answer synthetic questions that test consistent reasoning about simple arrangements of entities described in text. | Everyday | Checked automatically | 67.3% | Fading |
| LSAT Models take a simulated Law School Admission Test. Scored on the 120 to 180 scale. | University | Checked automatically | 163.0 | System cards only |
| RTE Models decide whether a premise sentence entails a hypothesis, using data from the PASCAL RTE challenges. Scored by accuracy. | Everyday | Checked automatically | 79.3% | Dropped |
| StrategyQA Models answer 2,780 yes or no questions that need implicit multi-step reasoning, such as whether Aristotle used a laptop. Scoring is accuracy. | Everyday | Checked automatically | 90.4% | Dropped |
| WiC Models decide whether a target word has the same meaning in two sentence contexts. Scoring is binary accuracy. | Everyday | Checked automatically | 66.8% | Dropped |
| ACPBench Models answer boolean and multiple-choice questions on 7 planning reasoning tasks over 13 formally described planning domains. Answers are provably correct. | University | Checked automatically |  62.9% | System cards only |
| BIG-bench Language models do over 200 tasks contributed by hundreds of authors, from linguistics and math to social bias. Most tasks are scored by exact match or multiple choice. | University | Checked automatically | Not in cards | |
| Hindsight Neglect Models judge whether a bet was worth taking based on expected value rather than its outcome. Scoring is accuracy. | Everyday | Checked automatically | System cards only | |
| LogiQA Models answer reading comprehension questions that test logical reasoning, taken from Chinese civil service exams and translated to English. | University | Checked automatically | Not in cards | |
| PAWS Models decide whether two sentences with heavy word overlap are paraphrases. | Everyday | Checked automatically | System cards only | |
| PlanningBench Models solve generated planning problems that coordinate goals, constraints and resources. Instances are scalable and come with verifiers. | Expert | Checked automatically | Not in cards | |
| RuleTrek Agents must find and correctly apply domain-specific rules when the scenario's facts are spread across many files, unlike rule benchmarks built from a few statements. | Expert | Checked automatically | Not in cards | |
| SnorkelSpatial Models track particles on a 2D grid through 10 to 200 text-described moves, then report final positions and orientations. The 330 samples test allocentric and egocentric spatial reasoning. | School | Checked automatically | Not in cards |