Recognition of model-written user turns (Anthropic)
Models write synthetic user turns for real Claude Code transcripts, then judge which version is real. Score is accuracy above chance.
The same model acts as generator and judge. Anthropic reports the probability of a correct choice minus 0.5.
- Level
- Expert Specialists with years of training, often over hours of work
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Prediction Probabilities for events that have not happened yet.
Claude Mythos Preview leads with 0.35
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude Mythos Preview | 0.35 | Apr 2026 | Model card |
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Labs report it only in system cards
We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark. Some system cards do, below.
One system card reports it
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 7 Apr 2026 | Claude Mythos Preview system card | 0.35 | Model ability to distinguish its own simulated user turns from real user turns, probability of a correct choice minus 0.5, value printed on chart (7.7, Figure 7.7.A) | Other |
More in Other
- Personal Superintelligence Index (Meta) In use Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO.
- Artificial Analysis Intelligence Index v5 In use Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness.
Gemini 4 Carbon 59.0 - Anthropic ECI (AECI) Only in system cards Index that combines a basket of capability benchmarks into one capability score to track trends across Anthropic models. Claude Opus 5.5 169.4
- Qwen in-house English automatic evaluation Only in system cards Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports.
Qwen2-72B 72.9% - LiveBench Dropped Models answer monthly refreshed questions across math, coding, reasoning, language, instruction following, and data analysis. Scored automatically against objective ground truth. Claude Fable 5.1 85.7%
- PhiBench Dropped Models answer original prompts that Microsoft's Phi team wrote to test a wide range of skills and reasoning.
Phi-4-Reasoning-plus 74.2%