Benchmark Almanac

Math

We track 105 benchmarks in math, or 122 if you count each version. They run from everyday level to research frontier level.

16 in use 41 past 85% 57 with public code

OpenAI held the top math score 57% of the contested time

That's about 26 of the 46 months since January 2023, averaged over the benchmarks below. The easiest benchmarks are at the top. Versions of one test share a row, with a dark tick at each new version. We stop the clock, and fade the rest of the row, when the best score passes 85%, or at the newest score if 45 days go by without another.

OpenAI Google DeepMind Anthropic Meta Other labs Qwen Tencent Deep Cogito DeepSeek Z.ai Meituan xAI Moonshot AI Ant Group
School Students, from grade school to high school
GSM8K Passed 85% in Mar 2023, by GPT-4
Llama 3.1-405B 96.8%
Competition The strongest students, in olympiad-style contests
MATH Passed 85% in Dec 2024, by Gemini 2.0 Flash
Cogito v2 Preview 671B MoE 98.2%
MATH Level 5 Passed 85% in Sep 2024, by o1-mini
GPT-5 98.1%
MATH-500 Passed 85% in Sep 2024, by o1-preview
LongCat-Flash-Thinking 99.2%
AIME 2026 Passed 85% in Feb 2026, by GPT-5.2
GPT-5.5 100%
OTIS Mock AIME 2024-2025 Passed 85% in Jul 2025, by Qwen3-235B-A22B-Thinking
GPT-5.5 100%
HMMT 2026 Passed 85% in Feb 2026, by GPT-5.2
GPT-5.5 98.5%
IMO-AnswerBench Passed 85% in Jan 2026, by LongCat-Flash-Thinking-2601
GLM-5.2 91%
Expert Specialists with years of training, often over hours of work
FrontierMath Tiers 1-3 v2 Passed 85% in Jun 2026, by GPT-5.5 Pro
GPT-6 Astra 93.7%
ProofBench v1.1 Passed 85% in Aug 2026, by Claude Opus 5
Claude Fable 5.1 100%
Research frontier Researchers, on open problems or tasks that take days
FrontierMath Tier 4 v2 Passed 85% in Jun 2026, by Claude Fable 5
GPT-6.1 Sol 100%

Scores come from each benchmark’s own leaderboard, from Epoch AI’s runs and from labs’ reports on their own models. We date Epoch runs by the model’s release and lab reports by when they came out. Labs pick their own settings, so a lab’s claim can beat an independent run of the same model. Turn off lab claims to see independent runs only.

OpenAI DeepSeek Anthropic xAI Google DeepMind Qwen Z.ai
School Students, from grade school to high school
GSM8K Passed 85% in Mar 2023, by GPT-4
DeepSeek-Coder-V2 236B 94.5%
Competition The strongest students, in olympiad-style contests
MATH Level 5 Passed 85% in Sep 2024, by o1-mini
GPT-5 98.1%
OTIS Mock AIME 2024-2025 Passed 85% in Jul 2025, by Qwen3-235B-A22B-Thinking
GPT-5.5 100%
AIME 2026 Passed 85% in Feb 2026, by GPT-5.2
GPT-5.5 100%
HMMT 2026 Passed 85% in Feb 2026, by GPT-5.2
GPT-5.5 98.5%
PolyMath Not tested since Apr 2025
Qwen3-235B-A22B 54.6%
MathArena (Apex, IMO, USAMO) Not tested since Jul 2026
Claude Opus 4.8 81.3%
Expert Specialists with years of training, often over hours of work
FrontierMath Tiers 1-3 v2 Passed 85% in Jun 2026, by GPT-5.5 Pro
GPT-6 Astra 93.7%
ProofBench v1.1 Passed 85% in Aug 2026, by Claude Opus 5
Claude Fable 5.1 100%
Research frontier Researchers, on open problems or tasks that take days
FrontierMath Tier 4 v2 Passed 85% in Jun 2026, by Claude Fable 5
GPT-6.1 Sol 100%

Only each benchmark’s own leaderboard and Epoch AI’s runs, dated by the model’s release. We leave out benchmarks that only labs report.

From everyday to research frontier level

Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.

Everyday

Most adults, without training

School

Students, from grade school to high school

University

Undergraduates and entry-level professionals

Competition

The strongest students, in olympiad-style contests

Expert

Specialists with years of training, often over hours of work

Research frontier

Researchers, on open problems or tasks that take days

By kind

Word problems

BenchmarkLevelScoredBest scoreStatus
GSM8K 8.5K grade-school math word problems. Final numeric answer scored by exact match. School Checked automatically 96.8% Fading
U-MATH Models solve 1,100 unpublished open-ended university math problems from teaching materials, balanced across six subjects. About 20% include images, and an LLM judge checks each solution. University Graded by a model 86.8% Not in cards
MathBench Models answer 3,709 math questions in five stages from basic arithmetic to college, in Chinese and English. Each stage also tests theory knowledge, and Circular Evaluation reshuffles answer options to check consistency. University Checked automatically 70.9% Not in cards
GSM1k Models solve new grade school math word problems written to mirror GSM8K in style and difficulty. The gap between GSM8K and GSM1k accuracy shows overfitting or contamination. School Checked automatically 96.6% Not in cards
CMATH 1.7K Chinese elementary-school math word problems. Final-answer accuracy. School Checked automatically 96.7% Fading
DeepMind Mathematics Models answer generated school-level math questions across algebra, arithmetic, calculus, probability and other modules. Answers are checked for an exact match. School Checked automatically 81.9% Fading
GRE Quantitative Models take the GRE quantitative reasoning section. Scored on the 130 to 170 scale. University Checked automatically 163.0 System cards only
GSM-Symbolic Models solve grade-school word problems generated from symbolic templates of GSM8K questions, with names and numbers changed. P1 and P2 add one or two extra clauses. School Checked automatically 87.4% Fading
HRM8K Models solve 8,011 math problems given as English and Korean parallel pairs. It measures how much math reasoning drops when problems are in Korean. School Checked automatically 92.2% Fading
Hungarian National High-School Exam Models answer the 33 questions of the 2023 Hungarian national high school mathematics finals. Answers are graded by hand with the official rubric. School Graded by people 59% Dropped
GMAT Quantitative Models answer the quantitative section of the official GMAT online exam. Scored as accuracy. University Checked automatically 96% System cards only
GSM-Plus Models solve GSM8K problems after perturbations such as changed numbers, added distractors and reversed questions. School Checked automatically 64.8% Dropped
Math23K Models solve Chinese elementary-school math word problems from a set of about 23,000. Scored by final-answer accuracy. School Checked automatically 88.5% System cards only
Math401 Models compute 401 arithmetic expressions covering addition, multiplication, exponents, logarithms and trigonometry. Scored by exact numeric accuracy. Everyday Checked automatically 70.1% System cards only
GSM8K-Platinum Models solve the GSM8K test problems after MIT researchers fixed wrong labels and removed badly written questions. Errors on it are more likely real model mistakes. School Checked automatically 74.6% System cards only
MathQA Models answer multiple-choice math word problems drawn from AQuA. Each problem is annotated with an operation program. Scored by accuracy. School Checked automatically 33.9% System cards only
SAT Math Models take the SAT math section. Scored on the 200 to 800 scale. School Checked automatically 700.0 System cards only
SuperCLUE-Math6 Models solve thousands of multi-step Chinese math word problems. Scores combine reasoning-step quality and accuracy into a comprehensive score and a reasoning level. School Graded by a model 83.3 System cards only
UGMathBench Models solve 5,062 undergraduate math problems across 16 subjects. Each problem has three randomized versions, and effective accuracy counts problems solved in all versions. University Checked automatically 76.5% System cards only
SnorkelGraph Models solve 200 procedurally generated graph problems, such as minimum vertex cover, given as node and edge lists in text. Answers are checked against verifiable ground truth. University Checked automatically Not in cards
SnorkelSequences Models compute values over integer sequences built from chains of filters, transforms and operators. The 250 procedurally generated questions test compositional multi-step math. School Checked automatically Not in cards

Competition problems with answers

BenchmarkLevelScoredBest scoreStatus
AIME 2025 Models answer the 30 problems from the 2025 AIME I and II exams. Each answer is an integer from 0 to 999, scored by match. Competition Checked automatically 100% In use
OTIS Mock AIME 2024-2025 Models solve 45 problems from three OTIS Mock AIME exams written by students of the OTIS program. Integer answers are scored by exact match. Competition Checked automatically 100% Not in cards
AIME 2024 Models solve the 30 problems of the 2024 AIME I and II. Answers are integers from 0 to 999, scored by exact match. Competition Checked automatically 96.6% Fading
MATH 12,500 competition math problems with step-by-step solutions. Final answer scored by exact match. Competition Checked automatically 98.2% Fading
HMMT 2025 Models answer the 30 problems from the February 2025 Harvard-MIT Mathematics Tournament. Final answers are checked against the official answers. Competition Checked automatically 100% Fading
MATH-500 Models solve 500 competition math problems drawn from the MATH test set. Final answers are checked against the reference answers. Competition Checked automatically 99.2% Fading
MATH Level 5 Models solve the 1,324 hardest MATH test problems, drawn from competitions such as AMC and AIME. Scored by final answer match. Competition Checked automatically 98.1% Dropped
AIME 2026 Models solve the 30 integer-answer problems from the 2026 AIME I and II math competitions. Scored by exact answer match. Competition Checked automatically 100% In use
IMO-AnswerBench 400 olympiad problems with short verifiable answers, robustified against memorization. Scored by answer match. Competition Checked automatically 91% In use
MathArena (Apex, IMO, USAMO) Evaluates models on newly released math competitions after release. Final-answer or human/LLM-judged proof scores. Competition Checked automatically 81.3% In use
HMMT 2026 Models solve the 33 final-answer problems from the February 2026 Harvard-MIT Mathematics Tournament. Scored by exact answer match. Competition Checked automatically 98.5% In use
PolyMath Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. Competition Checked automatically 86.5% In use
HMMT Nov 2025 Models solve the 30 problems of the November 2025 Harvard-MIT Mathematics Tournament. Final answers are checked automatically against the answer key. Competition Checked automatically 99.2% Fading
BRUMO 2025 Models answer the 30 final-answer problems of the 2025 Brown University Math Olympiad. MathArena scores the average accuracy over four attempts. Competition Checked automatically 100% Not in cards
SMT 2025 Models answer 53 final-answer problems from the 2025 Stanford Math Tournament. MathArena scores the average accuracy over four attempts. Competition Checked automatically 93.4% Not in cards
CMIMC 2025 Models answer 40 final-answer problems from the 2025 Carnegie Mellon Informatics and Mathematics Competition. MathArena scores the average accuracy over four attempts. Competition Checked automatically 95.6% Not in cards
AMO-Bench Models solve 50 original math problems at or above IMO difficulty that need only a final answer. Answers are graded automatically for accuracy. Competition Checked automatically 52.4% Fading
Beyond AIME Models answer 100 original competition-style math problems at or above the hardest AIME level. Each answer is a positive integer checked by exact match. Competition Checked automatically 87% Fading
HiddenMath Models solve 179 competition-level math problems that experts wrote from scratch. Answers are graded automatically. Competition Checked automatically 80.6% Dropped
Omni-MATH Models solve 4,428 Olympiad-level competition math problems across 33 subdomains and 10 difficulty levels. A GPT-4 or Omni-Judge grader checks answers. Competition Checked automatically 60.5% Dropped
OlympiadBench Models solve bilingual olympiad-level math and physics problems in Chinese and English, many with images. Answers are checked against a key. Competition Checked automatically 25.9% Dropped
CNMO 2024 Models answer problems from the 2024 Chinese National High School Mathematics Olympiad. DeepSeek reports Pass@1 averaged over 16 samples. Competition Checked automatically 88.5% Dropped
AMC Models solve problems from the AMC 8, 10, and 12 competitions. Gemini 1.5 used 250 problems from 2022 and 2023, scored by accuracy. Competition Checked automatically 69.4% Fading
HRM8K-KSM Models solve the 1,428 problems of the Korean School Math subset of HRM8K. Problems come from Korean olympiad and competition exams. Competition Checked automatically 91.1% Fading
Ko-AIME 2025 (Upstage) Models solve AIME 2025 competition problems in Korean. Upstage built the set in house. Competition Checked automatically 97.7% Fading
OMEGA Models solve generated competition-style math problems that test exploratory, compositional and transformative generalization beyond training patterns. Answers are checked automatically. Competition Checked automatically 53.4% Fading
Project Euler (MathArena) Models solve recent Project Euler problems that need both mathematical insight and code. MathArena adds problems as Project Euler releases them, starting from problem 943. Expert Checked automatically 89% Not in cards
AMC 10 2023 Models take the 2023 AMC 10 A and B exams. Scored out of 150 points. Competition Checked automatically 72.0 System cards only
AMC 12 2023 Models take the 2023 AMC 12 A and B exams. Scored out of 150 points. Competition Checked automatically 63.0 System cards only
AMC 8 2023 Models take the 25 question 2023 AMC 8 exam. Scored out of 150 points. Competition Checked automatically 84.0 System cards only
QuantBench Models answer 25 multiple choice versions of quant trading firm reasoning puzzles. Scored as cons@16 accuracy. Competition Checked automatically 60% System cards only
AIME 1983-2024 Models solve 949 past AIME problems from 1983 to 2024. Integer answers are checked exactly. Competition Checked automatically 89.4% System cards only
CSAT Math 2025 Models solve the mathematics sections of South Korea's 2025 College Scholastic Ability Test in Korean. The score averages the three elective subjects: calculus, statistics and geometry. School Checked automatically 94.5% Dropped
Functional MATH Models solve new instances of MATH problems that code generates for each snapshot. The gap from static MATH accuracy measures memorization. Competition Checked automatically 64.6% Dropped
Gaokao Math 2026 Models sit the five 2026 Chinese Gaokao mathematics papers (National I, National II, Beijing, Shanghai and Tianjin). Each paper is graded out of 150 points. School Checked automatically 134.9 In use
HiddenMath-Hard Models solve the harder part of an expert-written, never-published set of AIME and AMC style competition problems. Competition Checked automatically 80.5% System cards only
Hy-Math Models solve Tencent in-house math problems without tools. Task and scoring details are not published. Expert Checked automatically 60.9% In use
AMC 10 2022 Models take a 2022 AMC 10 exam. Scored out of 150 points. Competition Checked automatically 30.0 System cards only
AMC 10/12 (November 2024) Models take the four November 2024 AMC 10 and 12 contests, 78 questions in all. Scored in contest points out of 150, averaged over tests. Competition Checked automatically 91.8 System cards only
AMC 12 2022 Models take a 2022 AMC 12 exam. Scored out of 150 points. Competition Checked automatically 60.0 System cards only
AMC 12 2024 Models answer the 25 multiple-choice problems of the 2024 AMC 12 contests. Scored on the official 150-point scale. Competition Checked automatically 143.7 System cards only
Ko-AIME 2024 (Upstage) Models solve AIME 2024 competition problems translated into Korean. Upstage built the set in house. Competition Checked automatically 80.3% System cards only
AIME 2023 Models solve the 30 problems of the 2023 AIME I and II. Answers are integers from 0 to 999, scored by exact match. Competition Checked automatically Not in cards
HMMT February 2024 Models answer the problems from the February 2024 Harvard-MIT Mathematics Tournament. Final answers are checked against the official answers. Competition Checked automatically Not in cards
LiveMath Models solve 250 math problems written with six professional mathematicians. The sets are 150 regular high school problems, 50 at about AIME level and 50 multimodal problems with figures. Competition Checked automatically Not in cards
LiveMathBench Models solve recent competition math problems, sampled many times. The G-Pass@k metric rewards both peak accuracy and stable answers across samples. Competition Checked automatically Not in cards
ProcessBench Models read step-by-step solutions to GSM8K, MATH, OlympiadBench and Omni-MATH problems and name the earliest wrong step, or say none is wrong. Scored by F1. Competition Checked automatically Not in cards

Written proofs

BenchmarkLevelScoredBest scoreStatus
IMO 2025 Models write full proofs for the six problems of the 2025 International Mathematical Olympiad. Graders score proofs out of 42 points, mapped to medal cutoffs. Competition Graded by people 83.3% Fading
IMO-ProofBench Models write proofs for 60 olympiad problems, split into a basic set and an advanced set up to IMO-hard. Each proof is graded from 0 to 7 by experts or a Gemini autograder. Competition Graded by people 88.1% Fading
USAMO 2026 Models write full proofs for the six problems of the 2026 USA Mathematical Olympiad. Judges grade each proof from 0 to 7 against rubrics. Competition Graded by people 98.2% In use
USAMO 2025 Models write full proofs for the 6 problems of the 2025 USA Mathematical Olympiad. Expert human graders score each proof. Competition Graded by people 30.1% Dropped
Putnam 2025 Models solve the 12 problems of the 2025 William Lowell Putnam competition with full proofs. Scored out of 120 points. Competition Graded by people 120.0 Fading
CMO 2025 Models write full proofs for the six problems of the 2025 China Mathematical Olympiad. Graders score proofs out of 126 points. Competition Graded by people 114.0 Fading
AdvancedMathBench Models write proofs for 245 advanced problems at undergraduate and doctoral qualifying exam levels. A verifier trained on expert annotations grades each proof. Expert Graded by a model 58.1% In use
Putnam 2024 Models write full proofs for the 12 problems of the December 2024 William Lowell Putnam competition. Proofs are graded out of 120 points. Competition Graded by people 98.3% Dropped
CMO 2024 Models write full proofs for the six problems of the 2024 China Mathematical Olympiad. Experts grade the proofs for points. Competition Graded by people 73.8% System cards only
IMC 2025 Models write proofs for the 10 problems of the 2025 International Mathematics Competition for University Students. Human judges grade each proof. University Graded by people 94.5% Not in cards
IMO 2026 Models write full proofs for the six problems of the 2026 International Mathematical Olympiad. Proofs are scored out of 42 points. Competition Graded by people 100% System cards only
IMO Shortlist 2024 Models write proofs for the 31 problems of the 2024 IMO Shortlist. DeepSeek scores the proofs with its own verifier model. Competition Graded by a model 27% System cards only
Miklós Schweitzer 2025 Models write proofs for the 2025 Miklós Schweitzer competition, a hard Hungarian university-level contest. MathArena runs and grades it. Research frontier Graded by people 90% Not in cards
BrokenMath Models are asked to prove false versions of 2025 competition problems. An LLM judge counts how often a model writes a proof instead of rejecting the statement. Competition Graded by a model Not in cards
CNML-Level Problems (DeepSeek) Models prove 91 theorems in algebra, geometry, number theory, combinatorics and inequalities at Chinese National High School Mathematics League level. A verifier model scores the proofs. Competition Graded by a model System cards only
IMO-GradingBench Models grade 1,000 olympiad proof attempts, and their grades are compared with human expert grades. Tests automatic evaluation of long-form math answers. Competition Checked automatically Not in cards
ProofRank Models write proofs for hard competition problems. An LLM judge compares correct proofs pairwise on conciseness, computational ease, simplicity, diversity and adaptivity, giving Elo ratings. Competition Graded by a model Not in cards
Putnam-like Models write proofs for 96 university problems in 8 sets of 12, in the style of the easier and middle Putnam tiers. Each problem has a Lean statement and a point rubric for grading. University Graded by a model Not in cards
USAMO 2024 Models write full proofs for problems from the 2024 USA Mathematical Olympiad, graded by human experts. Competition Graded by people Not in cards

Formal proofs

BenchmarkLevelScoredBest scoreStatus
ProofBench Models write Lean 4 proofs of graduate-level theorems, with Mathlib search and code tools for up to 40 turns. Scored by the share that compile. Expert Checked automatically 78% Not in cards
ProofBench v1.1 Models write Lean 4 proofs of graduate-level theorems from paired natural-language and formal statements. Scored by the share the Lean checker accepts. Expert Checked automatically 100% Not in cards
PutnamBench Models write formal Lean proofs of Putnam competition problems. A proof counts only when the verifier accepts it. Competition Checked automatically 87.5% Fading
miniF2F Models write formal proofs of olympiad and high school competition statements, such as AMC, AIME and IMO problems. A proof assistant checks each proof, and scores are pass@k on the 244 test problems. Competition Checked automatically 100% Fading
ArXivLean Models prove or disprove theorems from recent arXiv papers, stated in Lean 4. A strict comparator checks each submitted proof against the fixed reference statement. Research frontier Checked automatically 47.2% System cards only
miniCTX-v2 Provers prove Lean theorems from real projects using long repository context, with theorems taken after a November 28, 2024 cutoff. Expert Checked automatically 81.8% Dropped
CombiBench Provers solve 100 combinatorics problems formalized in Lean 4, from middle school to IMO and university level. Lean checks each proof. Competition Checked automatically 48% Fading
FATE-H Provers write Lean proofs for 100 abstract and commutative algebra problems at honors course or graduate exam level. The Lean compiler checks each proof. Expert Checked automatically 87% Fading
FATE-X Provers write Lean proofs for 100 algebra problems at or beyond PhD qualifying exam level, some beyond what Mathlib covers. The Lean compiler checks each proof. Research frontier Checked automatically 34% Fading
FLTEval Agents complete the formal Lean proofs and define the new mathematical concepts in real pull requests to the Fermat's Last Theorem (FLT) formalization project. Lean and SafeVerify check each submitted diff. Research frontier Checked automatically 43.2 Fading
Past IMO (Seed-Prover formal set) Provers write Lean proofs for 155 past IMO problems, mostly adapted from Compfiles and MiniF2F. Competition Checked automatically 78.1% Dropped
ProverBench (DeepSeek) Provers write Lean 4 proofs for 325 formalized problems: 15 from AIME 2024 and 2025 and 310 from textbooks across number theory, algebra, calculus and analysis. Lean checks each proof. University Checked automatically 59.1% Dropped
FormalMATH Provers write Lean 4 proofs for 5,560 formal statements from high-school olympiad problems to undergraduate theorems. The Lean checker scores each proof. Competition Checked automatically 28.3% System cards only
ProofNet Provers write formal proofs for 371 undergraduate theorems from analysis, algebra and topology textbooks, each with a formal statement and a natural-language proof. The proof checker grades them. University Checked automatically 37.1% System cards only
IMO-LeanProofBench Models write Lean 4 proofs for the 60 IMO-ProofBench problems, formalized by hand and checked by Lean experts. The Lean compiler verifies each proof. Competition Checked automatically Not in cards
OpenFrontiers-Bench Agents try to rediscover proofs of research math problems, most resolved during 2026, stated in Lean 4. Two independent checkers grade the formal proofs, with closed-book, frozen-retrieval and live-web settings. Research frontier Checked automatically Not in cards

Research mathematics

BenchmarkLevelScoredBest scoreStatus
FrontierMath Tiers 1-3 v2 Models solve 295 unpublished math problems from undergraduate to advanced graduate level and submit answers that code checks automatically. Scored by share correct. Expert Checked automatically 93.7% In use
FrontierMath Unpublished research-level math problems with automatically verifiable answers. Tier 4 is the hardest tier. Expert Checked automatically 52.4% Fading
FrontierMath Tier 4 v2 Models solve 43 unpublished research-level math problems and submit answers that code checks automatically. Scored by share correct. Research frontier Checked automatically 100% In use
FrontierMath Tier 4 Models solve research-level math problems that take experts days, submitting answers through a Python function. Each answer is verified automatically as correct or incorrect. Research frontier Checked automatically 47.9% Fading
Leipzig Benchmark Models answer 100 research-level math questions with known answers, written by 49 mathematicians at a Max Planck Institute workshop in April and May 2026. Research frontier Checked automatically 91% Not in cards
ArXivMath Final-answer research-level math problems extracted monthly from recent arXiv papers and filtered to be self-contained. Research frontier Checked automatically 94.2% In use
FrontierMath Erdős Models try to prove or disprove 68 open Erdős problems in Lean 4. A result counts only if the Lean checker accepts the full proof. Research frontier Checked automatically 4.41% Not in cards
IMProofBench Agents with web search and SageMath write proofs for private PhD-level problems written by mathematicians. Experts grade the proofs, and final-answer subquestions are graded automatically. Research frontier Graded by people 49.3% Not in cards
BrokenArXiv Models are asked to prove statements from recent arXiv papers that were changed to be plausible but false. Full credit goes to models that say the statement is false. Research frontier Graded by a model 89.7% In use
Riemann-bench 25 private research-mathematics problems from professors and IMO medalists. Agents with tools, 100 runs per problem. Research frontier Checked automatically 80% In use
HorizonMath Models attack 113 mostly unsolved problems in eight areas of mathematics and the mathematical sciences. A program checks each proposed closed form, construction or bound. Research frontier Checked automatically 8.80% In use
Hy-Euler pro Models with tools solve Tencent in-house STEM problems. Task and scoring details are not published. Expert Checked automatically 24.2% In use
LiveMathematicianBench Models answer multiple choice questions about theorems from arXiv papers published after their training cutoff. Wrong options follow plausible but invalid proof sketches. Research frontier Checked automatically 20.9% System cards only
First Proof AI systems attempt ten unpublished research problems that arose in the authors' own work. Mathematicians judge whether each proof is correct. Research frontier Graded by people Not in cards
First Proof Second Batch The editors run AI harnesses once each, with 24 hours, on ten unpublished research problems with human proofs of up to eight pages. Referees grade each answer blind. Research frontier Graded by people Not in cards
First Proof Third Batch The editors run five AI systems once each on ten unpublished research problems with proofs of up to 12 pages. About 30 referees score correctness, exposition and attribution, with negative marks for wrong answers. Research frontier Graded by people Not in cards
FrontierMath: Open Problems Models attack unsolved research math problems proposed by working mathematicians. Each problem has a program that checks a proposed solution. Epoch tracks which problems AI has solved. Research frontier Checked automatically Not in cards
Soohak Models solve 439 new research-level and graduate math problems written by mathematicians. A 340-item Challenge subset tests solving and a 99-item Refusal subset tests spotting ill-posed problems. Research frontier Checked automatically Not in cards
TCS-Bench Models write proofs of target results from STOC, FOCS and SODA papers, given the needed context. A verification agent, checked against expert judgments, grades each proof. Research frontier Graded by a model Not in cards