Benchmark Almanac

MobileGym

Gemini 3.1 Pro 58.8% leads since May 2026 Not in model cards

Mobile GUI agents use 28 simulated apps in a browser-hosted phone, across 416 parameterized task templates. Deterministic judges check the app state.

EMNLP 2026. Also an online RL training platform. A 9-agent leaderboard was published in April 2026.

In the authors’ words

We present MobileGym, a browser-hosted, lightweight, fully controllable environment for everyday mobile use, targeting interaction fidelity without replicating proprietary backends. It enables two capabilities previously out of reach for everyday apps: verifiable outcome signals through deterministic state-based judging over structured JSON state, and scalable online RL through low-cost parallel rollouts. The full environment state is captured, configured, forked, and compared as structured JSON, and a single server can host hundreds of parallel instances, with about 400 MB memory per instance and about 3 s cold start. A layered state model and a declarative task-definition framework keep state programmability and task creation practical at scale, and a single programmatic judging mechanism delivers both deterministic evaluation verdicts and dense RL rewards. The accompanying MobileGym-Bench provides 416 parameterized task templates, including 256 test and 160 train templates, over 28 apps, with deterministic judges and a structured AnswerSheet protocol that avoids free-text matching failures. In a Sim-to-Real case study, GRPO on Qwen3-VL-4B-Instruct gains +12.8 percentage points on the 256-task test set, and on a 59-task real-device signal subset, real-device execution retains 95.1% of the simulation-side training gain. Project page: https://mobilegym.github.io.

Abstract of MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research (arXiv:2605.26114, CC0)
Made by MobileGym authors Released May 2026 Phones All tasks public Last model added Mar 2026 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
9
Entries on its own leaderboard, 4 models we track

Gemini 3.1 Pro leads with 58.8% among models released by Mar 2026

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Mar 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Gemini 3.1 Pro 58.8%
  2. 2 Seed2.0 Pro 52%
  3. 3 Qwen 3.6 Plus 45.7%
  4. 4 AutoGLM-Phone-9B 20%
  5. 5 UI-Venus-1.5-8B 15.4%
  6. 6 GUI-Owl-1.5-8B-Think 15.1%
  7. 7 UI-TARS-1.5 13.8%
  8. 8 Step-GUI-4B 12.9%
  9. 9 Qwen3-VL-4B 9.40%
#EntryScoreSettingDateSource
1 Gemini 3.1 Pro 58.8% Feb 2026 Official leaderboard
2 Seed2.0 Pro 52% Feb 2026 Official leaderboard
3 Qwen 3.6 Plus 45.7% Mar 2026 Official leaderboard
4 AutoGLM-Phone-9B 20% Official leaderboard
5 UI-Venus-1.5-8B 15.4% Official leaderboard
6 GUI-Owl-1.5-8B-Think 15.1% Official leaderboard
7 UI-TARS-1.5 13.8% Apr 2025 Official leaderboard
8 Step-GUI-4B 12.9% Official leaderboard
9 Qwen3-VL-4B 9.40% Official leaderboard

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind
Own leaderboard Not tested since May 2026

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Gemini 3.1 Pro Seed2.0 Pro UI-TARS-1.5 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Gemini 3.1 Pro Seed2.0 Pro

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Computer and browser use

All 56 in Computer and browser use
  • BrowseComp In use Hard-to-find facts that need persistent web browsing. Short verifiable answers graded for exact correctness. Claude Fable 5 93.3%
  • ScreenSpot-Pro In use GUI grounding on high-resolution professional-software screenshots. The model must locate the target UI element. GPT-6 Astra 92.7%
  • AndroidWorld In use Agents complete 116 tasks across 20 real Android apps in an emulator. Each task checks the device's system state to judge success. GPT-6 Astra 100%
  • OSWorld 2.1 In use Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. Claude Opus 5.5 81.8%
  • WebArena-Verified In use Agents complete 812 audited WebArena tasks on self-hosted websites. Deterministic evaluators check the agent's answer and captured network traces. Muse Spark 1.1 69%
  • MobileWorld In use Agents complete 201 tasks across 20 mobile apps, including tasks that need user clarification or MCP tools. Backend checks verify success. Kimi K3 74.4%