Forecasting
We track 12 benchmarks in forecasting, or 14 if you count each version.
How hard the tests are
Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.
Expert
Specialists with years of training, often over hours of work
Another 1 has no level. Levels don’t apply to arenas, safety evaluations or composite indices.
By kind
Real-world events
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| ForecastBench Models give probabilities for 500 auto-generated questions per round about future events, drawn from prediction markets and time-series datasets. Forecasts are scored with difficulty-adjusted Brier scores once the questions resolve. | Expert | Settled by events | 44.7 | In use |
| Prophet Arena Models and agents forecast open Kalshi prediction-market events and are scored once the events resolve. The live board ranks a difficulty-adjusted Arena score and also reports calibration error and market return. | Expert | Settled by events | 86.0 | In use |
| Echo General AI Prediction Leaderboard Models forecast unresolved events drawn from prediction markets, synthesized news questions and expert-written questions. Forecasts made at aligned times are compared in pairwise battles, and an Elo fit ranks the models. | Expert | Settled by events |  1,018 | Not in cards |
| Human Behavior Bench Models predict how specified groups of U.S. adults answer closed survey questions, as a distribution over options. Scored as 1 minus total variation distance. | Expert | Checked automatically |  80.6 | Not in cards |
| BTF-3 Agents forecast already-resolved binary and numeric questions while researching a frozen snapshot of the web. Scoring pools Brier and ranked probability scores. | Expert | Settled by events | 0.12 | Not in cards |
| BackSearch Paid API that searches a frozen web archive as of a given date, for forecasting backtests and reproducible web-enabled evals. A tool, not a benchmark. | Settled by events | Not in cards | ||
| BallotBench Nine frontier agents forecast 863 contests in the 2026 US midterms each day until polls close. Proper scoring rules grade forecasts against certified results, beside Kalshi, Polymarket and expert ratings. | Expert | Settled by events | Not in cards | |
| Bench to the Future Agents forecast hundreds of already-resolved questions while researching an offline corpus of web pages saved before the resolution. Scored by Brier score. | Expert | Settled by events | Not in cards | |
| BTF-2 Agents forecast 1,417 binary questions asked in October 2025 and resolved in December 2025, researching offline in a frozen corpus of 15 million documents. Scored by Brier score. | Expert | Settled by events | Not in cards | |
| FutureEval (Metaculus) Models forecast open Metaculus questions with a fixed prompt and are scored as the questions resolve. Seasonal bot tournaments and Pro Forecaster baselines run alongside. | Expert | Settled by events | Not in cards | |
| FutureX Agents predict upcoming events collected daily from 195 websites in 11 domains, within a one-week window. Answers are scored once real results are crawled, with harder tiers weighted more. | Expert | Settled by events | Not in cards | |
| Mantic vs Polymarket backtest Point-in-time backtest of Mantic's AI forecasts against prices on 4,139 Polymarket markets at volume snapshots. Scored by Brier score and calibration. | Expert | Settled by events | Not in cards |
Social simulation
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| KellyBench Agents bet through a simulated 2023 to 24 English Premier League season with historical data. The score is long-term bankroll growth. | Expert | Settled by events | Not in cards | |
| SWM-Bench (Social World Models) Predict how Kalshi and Polymarket belief trajectories shift after news events. 12k+ points across politics, finance and crypto, chronologically split. | Expert | Checked automatically | Not in cards |