Benchmark Almanac

Submit a benchmark

If you made a benchmark or found one we're missing, send it in. You'll get an automatic reply within a minute, and we review every submission by hand.

Open the form on GitHub See past submissions

The form needs a free GitHub account. Everything you write there is public.

What gets cataloged

We list 2,721 benchmarks so far. A new one gets in if it passes at least one of four tests.

  1. A major lab reports it A lab that ships frontier or widely used models reports it in a launch post, model card, system card or technical report.
  2. An independent leaderboard runs it Its makers or a third party keep a leaderboard with scores for five or more current models.
  3. An established evaluation group made it For example Epoch AI, METR, Scale, Vals, ARC Prize, Apollo Research, UK AISI, US CAISI, SecureBio, Arena, HELM or Kaggle.
  4. People use it About 100 GitHub stars or 30 Hugging Face upvotes, coverage in a widely read newsletter, or other papers and model cards that run it.

What stays out: test sets made only to evaluate one paper’s own method, training datasets, surveys, renamed copies of a test we already have, and narrow tasks outside general AI ability, such as crowd counting or defect detection. A new version that changes the tasks or the scoring gets its own record. One that only fixes broken tasks doesn’t.

What to include

The form asks for all of these. You only have to give the name, what it tests, its area, a paper or report, and the evidence, but the rest makes our review faster.

The name
And the version, when there is more than one.
What it tests
One or two sentences about the tasks and how a model is judged on them. Leave the motivation for the paper.
The paper or report
arXiv, OpenReview, a technical report or a launch post.
The code and the data
Links, with their licenses. Say whether the test set is public, held out or partly held out.
The scores
A leaderboard, if there is one, and a file of scores we can read, such as a CSV, JSON or a Hugging Face dataset. If each score has a date, we can chart who held number one.
The evidence
Links that show which of the four tests it passes, such as a model card that reports it, a leaderboard that runs it, or papers that use it.
What it replaces
If it’s a new version of a benchmark we already list, say which one and what changed in the tasks or the scoring.

What happens next

  1. Within a minute, you get an automatic reply on your submission. It tells you whether we already have the benchmark or left it out before, with the reason. It also lists any items that are missing.
  2. Within a few days, we review it against the four tests. If it passes, we add a record for it, and it shows up on the New page after the next daily update.
  3. If it doesn’t pass yet, we say which test it missed. You can add evidence to the same submission at any time, for example once a lab reports it.

To report a mistake in one of our records, use the correction form. Include a link to the page and a source for the right value.