How we compare scores

Three kinds of score

Lab reports

522 launch posts and 346 system cards, model cards and technical reports. Labs choose which benchmarks to show, the settings, the harness and often the task subset. These are self-reported numbers.

Leaderboards

1,746 boards that benchmarks' makers publish, with 40,393 scores for models we track. Some boards run every model themselves, and others accept submissions, sometimes from the labs.

Epoch AI

1,744 scores from Epoch AI’s own runs, retrieved November 12th, 2026. Epoch also collects scores from elsewhere. We count 496 it copied from leaderboards we don’t read ourselves as independent runs, and the 1,818 it took from labs’ reports, papers and unnamed sources as claims. Its Epoch Capabilities Index (ECI) covers 256 models.

Our charts and tables say which kind of score they show. Filled squares and dots are leaderboard scores and Epoch AI’s runs, and filled diamonds are leaderboards Epoch AI copied. Rings are numbers a lab reports about its own model, and hollow diamonds are claims Epoch AI collected.

One score per model on each board

A model tested at several effort levels or in several harnesses keeps one place on a board, at its best setting. The setting and harness stay on the score as labels, so GPT-5 at high effort is still GPT-5. Charts of every run show each setting as its own dot.

Leaderboard entries that aren't in our model list, such as fine-tunes and agent products, appear on their benchmark's board under the board's own name. They never count toward leaders or a lab's credit.

Which board a page leads with

A benchmark page leads with the benchmark's own leaderboard, or with a board a third party such as Stanford's HELM runs for it, when the board has five or more models and its newest score is within six months of the newest score from any source. Otherwise it leads with Epoch AI's scores on the same terms, and then with lab reports. A board that stopped at last year's models shouldn't headline a benchmark labs still report. Who leads a benchmark, and who first took it past 85%, comes from that one source, here and on the leaders page. A switch on each page ranks every independent run together, or every score with claims included.

When two scores compare

Two scores on one benchmark compare only if the runs match on all of these. Where we can tell they don't, we say so on the score or in the benchmark's notes.

Version

Each version is its own record, linked to the others as a family. We never mix ARC-AGI-1 and ARC-AGI-2 scores.

Unit

We fold a lab's unit as written ("Elo", "ANLS %") into the benchmark's unit and leave out any score in another unit. Charts that combine sources do so only for percentages.

Tasks

Labs sometimes run a subset, such as 477 of SWE-bench Verified's 500 tasks. The lab's own label for the result, which we keep, usually says so.

Harness and effort

Labs often use their own agent harness and their highest effort setting, while a leaderboard runs one harness for every model. Leaderboard rows carry both as labels.

Attempts

pass@1 counts the first try. pass@k counts a success in any of k tries and runs higher.

Grader

A benchmark graded by a model can change its judge, and scores move with it.

Date

Benchmarks fix questions, regrade and change their rules. We log those changes on each benchmark's page.

Changes to benchmarks

40 benchmark pages list 71 dated changes by their maintainers. We log seven kinds: scores recomputed, a new grader, tasks removed or fixed, a new official harness, a new headline metric, corrected questions, and new leaderboard rules. Each entry links to the maintainers' own source, such as a commit, a changelog, a paper revision or their announcement. We log changes that can move a headline score or change what a benchmark tests, and leave out fixes to single questions. New versions aren't changes, because each has its own record. So far we've researched the 120 benchmarks labs cite most, so other pages may have changes we haven't logged.

Lab claims next to leaderboards

The claims page pairs each lab's best number for its model with the same model's best row on the benchmark's own leaderboard. Where the benchmark has no leaderboard, it uses Epoch AI's score, but only for benchmarks Epoch runs itself, since the scores Epoch collects are often the lab's own. We have 1,307 pairs across 306 benchmarks.

Numbers within 0.15 points count as the same, as do dollars and ratings within half a unit, since labs often print them whole. 512 of the 1232 pairs in percent match. A match usually means one side copied the other. Of the rest, the lab's number is better in 456 and worse in 264. Most gaps come from setup, not error. Where we've checked a large gap, the table says what differs.

The Self-Reported Index

The ECI comes from Epoch AI's data, which includes its own runs, and it covers 256 models. Labs publish numbers for many more, often months before anyone else tests them. The Self-Reported Index puts 404 models on one scale using only those numbers: 6,770 scores on 225 benchmarks, from 792 lab documents.

Read it as a claim. Labs pick the benchmarks they show and how they run them, and they rarely publish a weak result. The index ranks what labs say about their models. The ECI is the better guide wherever a model has both.

What goes in

Scores in percent on capability benchmarks, from launch posts and from the capability sections of system cards. Safety, alignment, cyber, bio-chem and robotics results stay out, along with lower-is-better scores. Where a lab reports a benchmark more than once, at several settings or in several documents, the best number counts, as it does everywhere on this site. A benchmark joins once 8 models report it. A model gets a score once it reports 5 of those benchmarks, at least 3 of them outside vision.

How it's fit

We use the same kind of model as the ECI, from item response theory. Each model has a capability, and each benchmark a difficulty and a slope. The expected score is a logistic curve in the gap between the two. Scores count from chance, so 25% on a four-choice test is zero, using Epoch AI's chance levels where it lists them. We fit every capability, difficulty and slope at once by least squares, with a light penalty that fixes the scale. The fit has no randomness, so the same data gives the same index.

Then we stretch and shift the result to the ECI's scale, matching the average and spread of the 177 models that have both. A model at 150 on one sits near 150 on the other.

How well it holds up

Across those 177 models it agrees with the ECI at r = 0.977, and the rank correlation is 0.983. The typical model sits 2.2 points from its ECI. In a five-fold check, where we hid a fifth of the scores and predicted them from the rest, the fit missed by 9.2 points (root mean square) against 15.3 for guessing each benchmark's average.

The 90% intervals come from how tightly a model's own scores pin it down. Benchmarks near a model's level tell the most, and saturated ones tell almost nothing, so a model whose scores all sit near 100% gets a wide interval. The intervals don't cover differences in how labs ran their tests.

Where it's weak

It's least reliable for older and smaller models, whose labs reported mostly easy benchmarks, and for models with only five or six scores. Two labs can report the same benchmark under different settings, such as with and without tools, and the index treats both as the same test. When labs stop reporting a saturated benchmark, the newest models lose the easy points of comparison.

The benchmarks it leans on

The 16 benchmarks the most models report, with the difficulty the fit gives each on the index scale. A model at a benchmark's difficulty is expected to get half the room above chance.

What we leave out

We are currently unable to feature Artificial Analysis benchmarks due to their terms of service. A lab that quotes an Artificial Analysis number in its own launch post keeps it, credited to the lab. Every leaderboard we import is credited and linked on its benchmark's page. The Sources page lists what we read.

Corrections

If a score, a pairing or a change entry is wrong, open an issue on GitHub with the source, and we'll fix the data file it came from. The code for everything on this page is in src/lib/self-reported.ts, src/lib/irt.ts and src/lib/claims.ts.