Where new benchmarks appear
We watch 23 sources for new benchmarks. Every morning we read 19 of them automatically and send the best finds to our review queue.
Newsletters
People who read everything and name the benchmarks worth knowing.
- AINews Latent Space
Every issue's recap has a cluster of evals, often under a New evals heading. Good for launches that never become papers, such as BallotBench and Immigration Bench.
- Don't Worry About the Vase Zvi Mowshowitz
The weekly AI roundup has an On Your Marks section on benchmarks and evals, with Zvi's take on what they show.
- Import AI Jack Clark
Half of recent issues lead with a named benchmark, most of them new, and Clark says why he thinks it matters.
- The Epoch Brief Epoch AI
Announces Epoch's own new benchmarks and its Benchmark Reviews.
- Interconnects Nathan Lambert
Mostly about open models and post-training, with the occasional long post on a new eval.
Paper feeds
Most benchmarks start as a paper.
- Daily Papers Hugging Face
About one paper in five is a new benchmark. Each comes with upvotes, the authors' organization and a code link, so they're easy to rank.
- arXiv Cornell University
No other feed catches as many new benchmarks. But the bulk of hits only mention a benchmark, so we filter hard.
- NeurIPS Evaluations and Datasets track OpenReview
Every accepted paper is a peer-reviewed dataset or evaluation. They arrive in one batch each fall.
OpenReview's API now needs a signed-in account to list papers. We'll read this track automatically once we have one.
Registries and hubs
Curated lists. When a benchmark lands here, someone has already judged it worth running.
- Benchmarking Hub Epoch AI
A curated list with independent runs and a capability index. A new row in its metadata means a benchmark worth tracking. Our independent runs come from here.
- Inspect Evals UK AI Security Institute
Each new eval gets its own folder, so a new folder means someone has packaged a benchmark to run.
- Harbor Hub The Terminal-Bench team
Labs and vendors publish agent task sets here first, ready to run. Some entries are only tests, so finds from here need a closer look.
- Environments Hub Prime Intellect
New eval and training environments every day. Most are for reinforcement learning, and only a few are benchmarks.
- Every Eval Ever EvalEval Coalition
A shared store of results on thousands of benchmarks. A new collection folder means a new benchmark has been added.
- Official benchmark datasets Hugging Face
Hugging Face's own allow-list of evaluation datasets that model cards can report results on. Every entry is a real benchmark.
Leaderboards
Groups that build and run their own benchmarks.
- Vals AI Vals AI
Vals builds its own benchmarks for law, finance, medicine and tax, and announces each one on its blog.
- SEAL leaderboards Scale AI
Scale publishes its own new benchmarks here, each with a dated post.
New Scale boards are added by hand. Once added, each board's scores are read from its public page (scripts/lib/scale.ts).
- Artificial Analysis Artificial Analysis
Its changelog announces each evaluation it adopts.
Its terms ban automated queries and bar its data from other ranking products, so we link to it and read nothing from it.
- Arena LMArena
Adds new arenas a few times a year. We read its published dataset instead of the site.
Labs
A benchmark in a frontier lab’s headline table is the strongest signal we have.
- Lab model cards and system cards Anthropic, Google DeepMind, OpenAI, Qwen, DeepSeek, Moonshot and others
Labs with closed models publish results in launch posts and system cards. Open-weight labs put their tables in Hugging Face READMEs.
We read launch posts from Google DeepMind, Google and Mistral automatically, along with new open-weight model cards on Hugging Face. Anthropic, OpenAI, Meta, DeepSeek and Kimi bar automated reading of their own sites, so we check those by hand.
Code and lists
Toolkits that add support for new benchmarks, and lists people keep by hand.
- VLMEvalKit OpenCompass
New multimodal and video benchmarks land here within weeks of their papers.
- awesome-ai-leaderboard SAIL Research
The most active general list of AI leaderboards.
- Awesome-CUA reacher-z
Structured, dated records of computer-use agent benchmarks, kept current by a scanner.
Social
- Benchmark authors on X Researchers and labs
Most launches that are not papers start as a thread. AINews links the threads it covers.
X's terms forbid scraping without written consent, so we never scrape it.
Harnesses, registries and lists
Tools for running benchmarks, and collections that list them.
| Name | Kind | What it is |
|---|---|---|
| AI Agent Benchmark Compendium | Lists | 50+ agent benchmarks grouped by function calling, reasoning, coding, computer use. |
| Arize Phoenix | Tools | Open-source tracing plus evaluation for LLM apps. |
| Artificial Analysis | Leaderboards | Independent runs of standard benchmarks into an Intelligence Index, plus price and speed tracking. |
| Artificial Analysis Intelligence Index | Harnesses | Composite score that aggregates Artificial Analysis' independently run benchmark suite into one number per model. |
| Atropos | Environment hubs | Nous Research's RL environments framework for collecting and scoring rollouts. |
| awesome-ai-benchmarks | Lists | Curated list of 100+ AI benchmarks across domains. |
| Awesome-LLM-Eval | Lists | Curated list of eval tools, datasets, leaderboards and papers. |
| BenchFlow | Environment hubs | Infra for building RL environments and evals (SkillsBench, FrontierPhysics). Skill evals with/without the skill. |
| Benchgen | Registries | Agent benchmarking platform: create a benchmark from a repo (questions, answer key, asker and scorer programs) and get a live ranking. |
| BenchLM.ai | Registries | Aggregator listing 668 LLM benchmarks with scores (Oct 2026). |
| Benchmark Registry | Registries | Collects model results from primary sources (95 models, 84 benchmarks, 942 records). |
| benchmarks.bio | Harnesses | 14 agentic biology benchmarks built from real experimental data snapshots. Deterministic graders check recovery of the real result. Scored per model and harness. |
| BetterBench | Registries | 46-criterion quality assessment of benchmarks across their lifecycle. Living repository of scored benchmarks and a minimum-quality checklist. |
| BIG-bench | Harnesses | 204-task collaborative benchmark (2022) that introduced many task styles. BIG-Bench Hard subset still cited. |
| Bloom | Harnesses | Generates evaluation suites for any described behavior automatically. |
| Braintrust autoevals | Tools | Library of LLM-judge and heuristic scorers. |
| ControlArena | Harnesses | Settings, model organisms and protocols for AI control experiments. |
| Decision Index | Harnesses | Index of 37 benchmarks framed as multiple-choice decisions for decision models. Chance-normalized scores (0 = random, 100 = perfect). |
| DeepEval | Tools | Pytest-style LLM eval framework with many built-in metrics. |
| Design Arena | Harnesses | Crowdsourced arena for AI-generated websites, games, images and video. Four anonymous models per battle, votes feed Bradley-Terry ratings. |
| EleutherAI LM Evaluation Harness | Harnesses | Standard few-shot harness for base and chat models. Hundreds of task configs (MMLU, ARC, HellaSwag, GSM8K...). Backend of the old Open LLM Leaderboard. |
| Embodied Arena | Harnesses | Platform integrating 22 embodied benchmarks (embodied QA, navigation, task planning) with leaderboards for 30+ models. |
| Epoch AI Benchmarking Hub | Registries | Independent runs and curated external results for ~93 benchmarks. CSV download, Python client, and the IRT-based Epoch Capabilities Index. |
| Epoch Capabilities Index (ECI) | Harnesses | Combines 50+ benchmark results into one capability scale using fitted benchmark difficulty and slope. Anchored Claude 3.5 Sonnet = 130, GPT-5 = 150. |
| Every Eval Ever | Registries | Shared JSON schema and crowdsourced database of eval results (~229k results, 2,200+ benchmarks on HF). Converters for Inspect, HELM, lm-eval. |
| Harbor | Harnesses | Framework and dataset registry for agent evals and RL rollouts from the Terminal-Bench team. `harbor run -d <dataset>` runs any registered task set. |
| HELM | Harnesses | Holistic Evaluation of Language Models: framework plus public leaderboards (Capabilities, Safety, AIR-Bench, MMLU, VHELM...). |
| Holistic Agent Leaderboard (HAL) | Leaderboards | Standardized, cost-aware agent leaderboard over 9 benchmarks with public trajectories. Now a reliability dashboard. |
| Hugging Face Open LLM Leaderboard | Leaderboards | Open-weight model leaderboard on lm-eval (v2: IFEval, BBH, MATH-Hard, GPQA, MuSR, MMLU-Pro). |
| Inspect AI | Harnesses | Open-source eval framework: datasets, solvers, scorers, sandboxed agent tools, log viewer. Used by UK AISI, METR and many labs. |
| Inspect Evals | Harnesses | Largest open collection of runnable eval implementations (171 benchmarks, 200+ tasks as of Sep 2026) on the Inspect framework. |
| Inspect Harbor | Harnesses | Runs Harbor-format tasks (Terminal-Bench 2.0, SWE-Bench Pro, ...) inside Inspect. |
| Kaggle Benchmarks | Leaderboards | Kaggle's platform and Python library for publishing open benchmarks with model leaderboards. |
| lighteval | Harnesses | Hugging Face's eval toolkit across inference backends. Powers HF leaderboards and Open-R1 evals. |
| LiveBench | Leaderboards | Monthly-refreshed questions with objective ground truth (math, code, reasoning, data analysis) to resist contamination. |
| LLM-Agent-Benchmark-List | Lists | Actively updated list of agent benchmarks. |
| LLM-eval-survey | Lists | Paper list for 'A Survey on Evaluation of Large Language Models'. |
| LMArena | Leaderboards | Crowdsourced pairwise human-preference arena (text, vision, WebDev, search...) with Elo/Bradley-Terry ratings. |
| lmms-eval | Harnesses | One harness for text, image, video and audio benchmarks of large multimodal models. |
| Manifold (Bifrost) | Harnesses | Commercial service that runs robot policies in GPU-parallel simulation across LIBERO, RoboCasa, RoboTwin and other benchmarks, with subgoal scores and failure clustering. |
| METR eval-analysis-public | Harnesses | DVC pipeline behind METR's time-horizon plots. Shows how to fit logistic success-vs-task-length curves. |
| METR task suite and tooling | Harnesses | METR's public autonomy tasks, Task Standard and Vivaria runner (all archived 2026) and Hawk, the current Inspect-based eval platform. |
| Munari | Tools | Scores human robot demonstrations and operators 0-10 on kinematic quality against same-task, same-session peers to guide data collection. |
| OLMES | Harnesses | Ai2's reproducible evaluation standard and suite used for OLMo reports. |
| OpenAI Evals | Harnesses | Older eval framework plus open registry of community-contributed evals. |
| OpenAI Frontier Evals | Harnesses | OpenAI's open-sourced Preparedness evals (e.g. PaperBench, SWE-Lancer) with their agent scaffolds. |
| OpenAI simple-evals | Harnesses | Lightweight reference implementations OpenAI used for headline numbers (MMLU, GPQA, MATH, HumanEval, SimpleQA, BrowseComp, HealthBench). |
| openbench | Harnesses | Provider-agnostic open eval runner bundling many standard benchmarks on top of Inspect. |
| OpenCompass | Harnesses | Large Chinese-led evaluation platform and CompassRank leaderboard covering 100+ datasets. |
| OpenEnv | Environment hubs | Gym-style interface and hub for agentic RL environments, integrated with TRL. |
| Petri | Harnesses | Automated alignment-auditing agent that probes target models across many seeded scenarios. |
| Prime Intellect agentic RL tasksets (SWE, terminal, search) | Harnesses | 23 revalidated SWE, terminal and search tasksets (~365k tasks) behind one taskset API and sandbox lifecycle for evals and RL training. |
| Prime Intellect Environments Hub | Harnesses | Community hub to publish, share and evaluate RL environments and eval reports across models. Environments run on prime-rl and Prime sandboxes. |
| Prime Intellect Environments Hub + verifiers | Environment hubs | Library and hub of RL environments that double as evals. 365k+ SWE/terminal/search tasks in 23 tasksets (Jul 2026). |
| promptfoo | Tools | CLI for prompt/agent test suites and red-teaming in CI. |
| Reasoning Gym | Environment hubs | 100+ procedurally generated reasoning tasks with verifiers for RLVR and eval. |
| Scale Labs leaderboards (SEAL) | Leaderboards | Expert-built private-holdout leaderboards (HLE, SWE-Bench Pro, MultiChallenge, EnigmaEval, HSS...). |
| SWE-Gym / R2E-Gym / SWE-smith | Environment hubs | Training environments for SWE agents: real repos with executable tests, procedurally generated tasks at scale. |
| Vals AI | Leaderboards | Independent domain benchmarks (legal, finance, tax, medical, coding) with a Vals Index. Runs a benchmark fellowship. |
| Vals Index | Harnesses | Combines agentic finance, coding, legal and tax benchmarks, weighted by each sector's share of US GDP, into one score. |
| VLMEvalKit | Harnesses | Eval toolkit for multimodal models: 220+ models and 80+ image/video benchmarks. Open VLM Leaderboard. |
| ZeroEval | Harnesses | Simple unified zero-shot framework for reasoning benchmarks (MMLU-Redux, ZebraLogic, CRUX...). |
How to build a good benchmark
What to read before you design an evaluation.
- Adding Error Bars to Evals Evan Miller (Anthropic)
How to report eval results with standard errors, clustered SEs and paired differences.
arxiv.org - Agentic Benchmark Checklist (ABC) UIUC Kang Lab (Zhu et al.)
Checklist for outcome validity, task validity and reporting in agent benchmarks. Audits of 10 popular benchmarks with repro scripts.
arxiv.org - AI Agents That Matter Princeton (Kapoor, Narayanan et al.)
Argues for cost-controlled agent evals and warns about overfitting and holdout leakage.
arxiv.org - AI and the Everything in the Whole Wide World Benchmark Raji et al.
Critique of 'general' benchmarks as claims about general ability.
arxiv.org - AISI Engineering Playbook UK AI Security Institute
UK AISI's engineering practices for running evals at scale on Inspect.
aisi.gov.uk - Automating eval design and hillclimbing with Claude Anthropic
Guide to eval design (production-like tasks, headroom, low variance, cheapest adequate grader) plus Claude Code build-eval and hillclimb commands.
github.com - Behavior specs (agentbehavior) Braintrust + Basis
Open standard: BEHAVIOR.md files describe how an agent should act across a trajectory. An LLM judge grades each spec true, false or NA.
github.com - Benchmark Map of the US Economy Mercor
Maps prompts from 29 public benchmarks to O*NET tasks with an LLM judge to show which US occupations benchmarks cover.
mercor.com - Challenges in evaluating AI systems Anthropic
Anthropic's account of practical problems with MMLU, BBQ, BIG-bench, HELM and red teaming.
anthropic.com - CUBE standard
Proposed standard and registry for packaging agent benchmarks so any harness can run them.
arxiv.org - Demystifying evals for AI agents Anthropic
Anthropic's practical guide: tasks, trials, graders, pass@k vs pass^k, capability vs regression suites, 8-step roadmap.
anthropic.com - Dynabench Meta (Kiela et al.)
Human-and-model-in-the-loop dynamic dataset creation to keep benchmarks hard.
arxiv.org - Every Eval Ever (paper) EvalEval Coalition
Schema and community repository for comparable eval results.
arxiv.org - Fluid Language Model Benchmarking Ai2
Uses item response theory to pick test items adaptively per model.
arxiv.org - How reliably can AI audit AI? (Free Systems) Free Systems (Andy Hall, Dan Thompson)
Study scoring LLMs as auditors of agent transcripts (METR MALT, R-Judge) for reward hacking, sandbagging and unsafe actions, by recall at fixed false-positive rates.
freesystems.substack.com - Hugging Face LLM Evaluation Guidebook Hugging Face (Fourrier)
Practical and theoretical notes on LLM evaluation from the Open LLM Leaderboard team.
github.com - Introducing SWE-bench Verified OpenAI
Case study: 93 engineers screened SWE-bench tasks for underspecified issues and unfair tests.
openai.com - Investigating unintended model actions in our evaluations and internal use Anthropic
Anthropic report on four behavior types where Claude acted on real websites during evals: injection exploits, real form submissions, fee-gate bypass, URL-shortener evasion.
anthropic.com - Life After Benchmark Saturation Princeton (Narayanan, Kapoor et al.)
Paper arguing saturated benchmarks still measure validity, generalization, efficiency, reliability and human uplift. CORE-Bench Hard is the case study.
arxiv.org - Many Benchmarks Scores Would Appear Much Higher If You Let The AIs Use Adequate Labor METR (Joel Becker, Nate Rush)
Blog post arguing reported benchmark scores understate AI ability when token, time or budget limits stop agents before performance plateaus.
joelbkr.substack.com - Measuring AI Ability to Complete Long Tasks METR
METR time-horizon method: fit success vs human task length. 50% horizon doubling ~7 months. TH 1.1 update Jan 2026.
arxiv.org - Measuring what Matters: Construct Validity in LLM Benchmarks Oxford Internet Institute (Bean et al.)
Systematic review of 445 benchmarks. 8 recommendations, starting with defining the phenomenon and justifying construct validity.
arxiv.org - Signal and Noise Ai2
Measures benchmark signal-to-noise. Picks metrics and subtasks that make small-scale decisions reliable.
arxiv.org - Successful language model evals Jason Wei
Jason Wei's checklist of what made evals like MMLU and GSM8K succeed.
jasonwei.net - Sunday evals (FoldLab laundry-folding failure analysis) BluPe
Failure analysis of Sunday Robotics' Memo folding laundry for three hours: 52 attempts labeled with 15 failure types, plus clips and an annotation tool.
blupe.io - The Benchmark Lottery Google (Dehghani et al.)
Shows benchmark choice changes which method looks best.
arxiv.org - The Leaderboard Illusion Cohere Labs et al.
Shows how private testing and uneven sampling distort Chatbot Arena rankings.
arxiv.org - The Lifecycle of LLM-as-a-Judge (Netflix) Netflix
Paper describing Netflix's LLM judge for recommendation explanations as a four-phase lifecycle: birth, training, deployment, monitoring.
arxiv.org - tinyBenchmarks Polo et al.
Estimates full-benchmark scores from ~100 IRT-selected items.
arxiv.org - Why Language Models Hallucinate OpenAI
Argues binary-graded benchmarks reward guessing. Proposes scoring that credits abstention.
arxiv.org - Your AI Product Needs Evals / LLM Evals FAQ Hamel Husain
Error-analysis-first approach to application evals and LLM judges.
hamel.dev