Benchmark Almanac

iOSWorld

Claude Opus 4.6 51.9% leads since Jun 2026 Not in model cards

Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history.

133 tasks: 27 single-app, 60 multi-app and 46 memory and personalization. Vision-only and vision plus XML settings. Best setup reached 51.9% overall. Apple's iPrefBench builds on it.

In the authors’ words

A useful phone agent needs to be personally intelligent. It should reason over a user's identity, history, and preferences as they exist on the device, not just follow isolated instructions in an impersonal sandbox. Existing mobile agent benchmarks lack this kind of personalization. We introduce iOSWorld, the first interactive native iOS simulator benchmark built around a persistent user identity spanning 26 newly built iOS apps. These apps contain connected data such as transactions, messages, travel records, social relationships, and financial activity. iOSWorld includes 133 tasks across three increasingly difficult categories. Single-app tasks (27) test one app, multi-app tasks (60) span 2 to 8 apps, and memory and personalization tasks (46) require agents to infer patterns from personal data. We evaluate frontier and open-source computer-use models in both vision-only and privileged vision+XML settings. The best configuration reaches 52% overall but only 37% on multi-app tasks. Privileged vision+XML access improves frontier models by up to 26 percentage points, while smaller models do not benefit from added accessibility-tree input. We release iOSWorld as an open-source benchmark with all apps, seeded data, tasks, rubrics, and evaluation code.

Abstract of iOSWorld: A Benchmark for Personally Intelligent Phone Agents (arXiv:2606.09764, CC0)
Made by Carnegie Mellon University (Salakhutdinov lab) Released Jun 2026 Phones All tasks public Last model added Mar 2026 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
3
Models scored on its own leaderboard

Claude Opus 4.6 leads with 51.9% among models released by Mar 2026

Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best.

The newest model with a score here came out in Mar 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Claude Opus 4.6 51.9%
  2. 2 Claude Sonnet 4.6 46.6%
  3. 3 GPT-5.4 39.8%
#ModelScoreSettingDateSource
1 Claude Opus 4.6 51.9% vision+xml Feb 2026 Official leaderboard
2 Claude Sonnet 4.6 46.6% vision+xml Feb 2026 Official leaderboard
3 GPT-5.4 39.8% vision+xml Mar 2026 Official leaderboard

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard Not tested since Jun 2026

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Opus 4.6 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Opus 4.6

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Computer and browser use

All 56 in Computer and browser use
  • BrowseComp In use Hard-to-find facts that need persistent web browsing. Short verifiable answers graded for exact correctness. Claude Fable 5 93.3%
  • ScreenSpot-Pro In use GUI grounding on high-resolution professional-software screenshots. The model must locate the target UI element. GPT-6 Astra 92.7%
  • AndroidWorld In use Agents complete 116 tasks across 20 real Android apps in an emulator. Each task checks the device's system state to judge success. GPT-6 Astra 100%
  • OSWorld 2.1 In use Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. Claude Opus 5.5 81.8%
  • WebArena-Verified In use Agents complete 812 audited WebArena tasks on self-hosted websites. Deterministic evaluators check the agent's answer and captured network traces. Muse Spark 1.1 69%
  • MobileWorld In use Agents complete 201 tasks across 20 mobile apps, including tasks that need user clarification or MCP tools. Backend checks verify success. Kimi K3 74.4%